View Issue Details

IDProjectCategoryView StatusLast Update
0002501SymmetricDSImprovementpublic2016-03-11 19:24
ReporterelongAssigned Toelong 
PrioritynormalSeveritymajorReproducibilityN/A
Status closedResolutionfixed 
Product Version3.7.0 
Target Version3.7.30Fixed in Version3.7.30 
Summary0002501: File sync performance on large file system
DescriptionThe file sync has performance and memory problems when used with a large file system, such as 1 million files.

To find new, modified, and deleted files, it loads the current list of files and compares it to the last list of files. The directory path and filename loaded into memory for all files can use more memory than available. The entire list is also inserted/updated into the sym_file_snapshot table with a single transaction, which can require more resources than available.

Another performance problem is the reading of the contents of each file to calculate a CRC32 hash. The CRC is used to detect a file conflict if it does not match on the target. Since some systems might be syncing in one direction, this should be a parameter.

Less of a problem is the loading of each file into sym_incoming_file on the target side, which is used to prevent "ping back" of the file to the source. However, since some systems might be syncing in one direction, this should be a parameter also.
Additional InformationProposed solutions:

file.sync.fast.scan - Parameter to enable a new method of finding changes, which scans for last modified time changes. For a directory that is modified, we must fall back to a slower method of retrieving the sym_file_snaphot and doing the in-memory comparison, but it is constrained to one directory. Use the data loader commit size to update sym_file_snapshot.

file.sync.use.crc - Parameter to disable the reading of all file content to generate CRC checksums.

file.sync.prevent.ping.back - Parameter to disable the insert into sym_file_incoming on a one-direction sync scenario.

TagsNo tags attached.

Activities

There are no notes attached to this issue.

Related Changesets

SymmetricDS: 3.7 15fc4102

2016-02-22 12:58:26

admin

Details Diff
0002501: File sync performance on large file system
0002501
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ParameterConstants.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/file/DirectorySnapshot.java Diff File
add - symmetric-core/src/main/java/org/jumpmind/symmetric/file/FileTriggerFileModifiedListener.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/file/FileTriggerTracker.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/model/FileSnapshot.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/model/ProcessInfoKey.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/ClusterConstants.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ClusterService.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ClusterServiceSqlMap.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/FileSyncService.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/FileSyncServiceSqlMap.java Diff File
mod - symmetric-core/src/main/resources/symmetric-default.properties Diff File
mod - symmetric-core/src/test/java/org/jumpmind/symmetric/file/FileTriggerTrackerTest.java Diff File

SymmetricDS: 3.7 986e8c21

2016-02-22 15:04:49

admin

Details Diff
0002501: File sync performance on large file system
0002501
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ClusterServiceSqlMap.java Diff File

Issue History

Date Modified Username Field Change
2016-02-22 09:31 elong New Issue
2016-02-22 09:31 elong Status new => assigned
2016-02-22 09:31 elong Assigned To => elong
2016-02-22 15:00 admin Changeset attached => SymmetricDS 3.7 15fc4102
2016-02-22 16:00 admin Changeset attached => SymmetricDS 3.7 986e8c21
2016-02-22 19:29 elong Status assigned => resolved
2016-02-22 19:29 elong Fixed in Version => 3.7.30
2016-02-22 19:29 elong Resolution open => fixed
2016-03-11 19:24 elong Status resolved => closed