View Issue Details

IDProjectCategoryView StatusLast Update
0000997SymmetricDSImprovementpublic2014-02-02 20:53
ReporterrottenAssigned Toabrougher 
PrioritylowSeverityminorReproducibilityN/A
Status closedResolutionfixed 
Product Version3.2.2 
Target Version3.2.3Fixed in Version3.2.3 
Summary0000997: dbimport and symmetricds dataloader throttle
DescriptionAt this time, when we run dbimport, it loads data as fast as it possibly can. We have a handy parameter to commit every X rows. What would be nice is a throttle which could slow down how fast it does a load.

My suggestion is it would be a 'sleep between transactions" time value (with zero being the default). Commit 1,000 rows, then Sleep a little bit before doing another 1,000 rows.

Recently we loaded a 12M row table into our production postgresql database. We jumped from our usual 1,000 writes per minute to 75,000 writes per minute. I think the bottleneck was the NFS volume we were feeding the CSV out of because there was no degradation in the production traffic or performance during the dbimport. (We were really nervous though.)

The next time we do something like that we need to figure out how to chunk the load, or slow it down. A work around is to break the CSV up into pieces and then write a wrapper script which sleeps between each dbimport. The work around isn't all that hard to implement, but it also seemed like it would be pretty easy to add this feature too. Hence a low priority minor improvement request.

Thanks for your attention to this!
TagsNo tags attached.

Activities

chenson

2013-01-18 16:21

administrator   ~0000185

This seems reasonable and easy to implement. Do you have suggestions for the command line option names?

rotten

2013-01-18 22:31

reporter   ~0000189

--coffee_break = time in ms to sleep between commits (transactions)
--batch_delay

I don't have a strong preference.

chenson

2013-01-23 08:50

administrator   ~0000194

Used --interval to be consistent with dbfill.

Related Changesets

SymmetricDS: master 59ac37f2

2013-01-22 22:36:25

abrougher

Details Diff
0000997: dbimport throttle

Added --interval option to specify a timeout between commits.

0000997
mod - symmetric-client/src/main/java/org/jumpmind/symmetric/DbImportCommand.java Diff File
mod - symmetric-client/src/main/resources/symmetric-messages.properties Diff File
mod - symmetric-io/src/main/java/org/jumpmind/symmetric/io/data/DbImport.java Diff File
mod - symmetric-io/src/main/java/org/jumpmind/symmetric/io/data/writer/DatabaseWriter.java Diff File
mod - symmetric-io/src/main/java/org/jumpmind/symmetric/io/data/writer/DatabaseWriterSettings.java Diff File

SymmetricDS: master cf504e53

2013-01-23 08:48:48

chenson

Details Diff
0000997: dbimport throttle
0000997
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ParameterConstants.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/load/DefaultDataLoaderFactory.java Diff File
mod - symmetric-core/src/main/resources/symmetric-default.properties Diff File
mod - symmetric-io/src/main/java/org/jumpmind/symmetric/io/data/writer/DatabaseWriter.java Diff File
mod - symmetric-io/src/main/java/org/jumpmind/symmetric/io/data/writer/DatabaseWriterSettings.java Diff File

Issue History

Date Modified Username Field Change
2013-01-18 14:12 rotten New Issue
2013-01-18 16:21 chenson Note Added: 0000185
2013-01-18 22:31 rotten Note Added: 0000189
2013-01-23 08:50 chenson Note Added: 0000194
2013-01-23 08:50 chenson Assigned To => abrougher
2013-01-23 08:50 chenson Status new => assigned
2013-01-23 08:50 chenson Fixed in Version => 3.2.3
2013-01-23 08:50 chenson Target Version => 3.2.3
2013-01-23 08:50 chenson Status assigned => resolved
2013-01-23 08:50 chenson Resolution open => fixed
2013-01-23 08:51 chenson Summary dbimport throttle => dbimport and symmetricds dataloader throttle
2013-01-23 20:35 chenson Status resolved => closed
2014-02-02 20:53 abrougher Changeset attached => SymmetricDS trunk r6773
2014-02-02 20:53 Changeset attached => SymmetricDS trunk r6775
2015-07-30 21:49 chenson Changeset attached => SymmetricDS master cf504e53
2015-07-30 21:49 abrougher Changeset attached => SymmetricDS master 59ac37f2