View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0002501 | SymmetricDS | Improvement | public | 2016-02-22 14:31 | 2016-03-12 00:24 |
| Reporter | elong | Assigned To | elong | ||
| Priority | normal | ||||
| Status | closed | Resolution | fixed | ||
| Product Version | 3.7.0 | ||||
| Target Version | 3.7.30 | Fixed in Version | 3.7.30 | ||
| Summary | 0002501: File sync performance on large file system | ||||
| Description | The file sync has performance and memory problems when used with a large file system, such as 1 million files. To find new, modified, and deleted files, it loads the current list of files and compares it to the last list of files. The directory path and filename loaded into memory for all files can use more memory than available. The entire list is also inserted/updated into the sym_file_snapshot table with a single transaction, which can require more resources than available. Another performance problem is the reading of the contents of each file to calculate a CRC32 hash. The CRC is used to detect a file conflict if it does not match on the target. Since some systems might be syncing in one direction, this should be a parameter. Less of a problem is the loading of each file into sym_incoming_file on the target side, which is used to prevent "ping back" of the file to the source. However, since some systems might be syncing in one direction, this should be a parameter also. | ||||
| Additional Information | Proposed solutions: file.sync.fast.scan - Parameter to enable a new method of finding changes, which scans for last modified time changes. For a directory that is modified, we must fall back to a slower method of retrieving the sym_file_snaphot and doing the in-memory comparison, but it is constrained to one directory. Use the data loader commit size to update sym_file_snapshot. file.sync.use.crc - Parameter to disable the reading of all file content to generate CRC checksums. file.sync.prevent.ping.back - Parameter to disable the insert into sym_file_incoming on a one-direction sync scenario. | ||||
| Tags | No tags attached. | ||||
|
SymmetricDS: 3.7 15fc4102 2016-02-22 12:58:26 Details Diff |
0002501: File sync performance on large file system |
Affected Issues 0002501 |
|
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ParameterConstants.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/file/DirectorySnapshot.java | Diff File | ||
| add - symmetric-core/src/main/java/org/jumpmind/symmetric/file/FileTriggerFileModifiedListener.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/file/FileTriggerTracker.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/model/FileSnapshot.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/model/ProcessInfoKey.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/ClusterConstants.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ClusterService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ClusterServiceSqlMap.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/FileSyncService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/FileSyncServiceSqlMap.java | Diff File | ||
| mod - symmetric-core/src/main/resources/symmetric-default.properties | Diff File | ||
| mod - symmetric-core/src/test/java/org/jumpmind/symmetric/file/FileTriggerTrackerTest.java | Diff File | ||
|
SymmetricDS: 3.7 986e8c21 2016-02-22 15:04:49 Details Diff |
0002501: File sync performance on large file system |
Affected Issues 0002501 |
|
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ClusterServiceSqlMap.java | Diff File | ||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2016-02-22 14:31 | elong | New Issue | |
| 2016-02-22 14:31 | elong | Status | new => assigned |
| 2016-02-22 14:31 | elong | Assigned To | => elong |
| 2016-02-22 20:00 | admin | Changeset attached | => SymmetricDS 3.7 15fc4102 |
| 2016-02-22 21:00 | admin | Changeset attached | => SymmetricDS 3.7 986e8c21 |
| 2016-02-23 00:29 | elong | Status | assigned => resolved |
| 2016-02-23 00:29 | elong | Fixed in Version | => 3.7.30 |
| 2016-02-23 00:29 | elong | Resolution | open => fixed |
| 2016-03-12 00:24 | elong | Status | resolved => closed |