View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000997 | SymmetricDS | Improvement | public | 2013-01-18 19:12 | 2014-02-03 01:53 |
| Reporter | rotten | Assigned To | abrougher | ||
| Priority | low | ||||
| Status | closed | Resolution | fixed | ||
| Product Version | 3.2.2 | ||||
| Target Version | 3.2.3 | Fixed in Version | 3.2.3 | ||
| Summary | 0000997: dbimport and symmetricds dataloader throttle | ||||
| Description | At this time, when we run dbimport, it loads data as fast as it possibly can. We have a handy parameter to commit every X rows. What would be nice is a throttle which could slow down how fast it does a load. My suggestion is it would be a 'sleep between transactions" time value (with zero being the default). Commit 1,000 rows, then Sleep a little bit before doing another 1,000 rows. Recently we loaded a 12M row table into our production postgresql database. We jumped from our usual 1,000 writes per minute to 75,000 writes per minute. I think the bottleneck was the NFS volume we were feeding the CSV out of because there was no degradation in the production traffic or performance during the dbimport. (We were really nervous though.) The next time we do something like that we need to figure out how to chunk the load, or slow it down. A work around is to break the CSV up into pieces and then write a wrapper script which sleeps between each dbimport. The work around isn't all that hard to implement, but it also seemed like it would be pretty easy to add this feature too. Hence a low priority minor improvement request. Thanks for your attention to this! | ||||
| Tags | No tags attached. | ||||
|
|
This seems reasonable and easy to implement. Do you have suggestions for the command line option names? |
|
|
--coffee_break = time in ms to sleep between commits (transactions) --batch_delay I don't have a strong preference. |
|
|
Used --interval to be consistent with dbfill. |
|
SymmetricDS: master 59ac37f2 2013-01-22 22:36:25 Details Diff |
0000997: dbimport throttle Added --interval option to specify a timeout between commits. |
Affected Issues 0000997 |
|
| mod - symmetric-client/src/main/java/org/jumpmind/symmetric/DbImportCommand.java | Diff File | ||
| mod - symmetric-client/src/main/resources/symmetric-messages.properties | Diff File | ||
| mod - symmetric-io/src/main/java/org/jumpmind/symmetric/io/data/DbImport.java | Diff File | ||
| mod - symmetric-io/src/main/java/org/jumpmind/symmetric/io/data/writer/DatabaseWriter.java | Diff File | ||
| mod - symmetric-io/src/main/java/org/jumpmind/symmetric/io/data/writer/DatabaseWriterSettings.java | Diff File | ||
|
SymmetricDS: master cf504e53 2013-01-23 08:48:48 Details Diff |
0000997: dbimport throttle |
Affected Issues 0000997 |
|
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ParameterConstants.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/load/DefaultDataLoaderFactory.java | Diff File | ||
| mod - symmetric-core/src/main/resources/symmetric-default.properties | Diff File | ||
| mod - symmetric-io/src/main/java/org/jumpmind/symmetric/io/data/writer/DatabaseWriter.java | Diff File | ||
| mod - symmetric-io/src/main/java/org/jumpmind/symmetric/io/data/writer/DatabaseWriterSettings.java | Diff File | ||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2013-01-18 19:12 | rotten | New Issue | |
| 2013-01-18 21:21 | chenson | Note Added: 0000185 | |
| 2013-01-19 03:31 | rotten | Note Added: 0000189 | |
| 2013-01-23 13:50 | chenson | Note Added: 0000194 | |
| 2013-01-23 13:50 | chenson | Assigned To | => abrougher |
| 2013-01-23 13:50 | chenson | Status | new => assigned |
| 2013-01-23 13:50 | chenson | Fixed in Version | => 3.2.3 |
| 2013-01-23 13:50 | chenson | Target Version | => 3.2.3 |
| 2013-01-23 13:50 | chenson | Status | assigned => resolved |
| 2013-01-23 13:50 | chenson | Resolution | open => fixed |
| 2013-01-23 13:51 | chenson | Summary | dbimport throttle => dbimport and symmetricds dataloader throttle |
| 2013-01-24 01:35 | chenson | Status | resolved => closed |
| 2014-02-03 01:53 | abrougher | Changeset attached | => SymmetricDS trunk r6773 |
| 2014-02-03 01:53 | Changeset attached | => SymmetricDS trunk r6775 | |
| 2015-07-31 01:49 | chenson | Changeset attached | => SymmetricDS master cf504e53 |
| 2015-07-31 01:49 | abrougher | Changeset attached | => SymmetricDS master 59ac37f2 |