View Issue Details

IDProjectCategoryView StatusLast Update
0003356SymmetricDSBugpublic2018-03-28 22:16
Reporterwoehrl01 Assigned Tommichalek  
Prioritynormal 
Status closedResolutionfixed 
Product Version3.9.2 
Target Version3.9.6Fixed in Version3.9.6 
Summary0003356: Add initial.load.transport.max.bytes.to.sync parameter. transport.max.bytes.to.sync not respected in 3.9 during loads/reload
DescriptionAfter the upgrade to 3.9, the setting transport.max.bytes.to.sync isn't respected anymore. It know sends all the data until the max_batch_to_send limit for the channel is reached.

Output:
[central-0] - PullUriHandler - 807875 data and 1000 batches sent during pull request from mobil:node2:2
[central-0] - PullUriHandler - 181286 data and 347 batches sent during pull request from mobil:node2:2
TagsNo tags attached.

Activities

woehrl01

2018-01-08 08:48

reporter   ~0001097

I provided a PR which fixes this behaviour: https://github.com/JumpMind/symmetric-ds/pull/62

kpatierno

2018-01-10 16:04

viewer   ~0001101

Hey Woehrl01

I have been reviewing your support ticket and have attempted to reproduce the error that you encountered but was unable to get the logging error messages that you did.

To test I did the following steps:
-Install and configured a new instance of SymmetricDS Pro 3.8.32
-Select a default configuration using the Demo setup
-Change the value for parameter transport.max.bytes.to.sync to 1000
-Change the ITEM channel's value for MAX_BATCH_TO_SEND to 2000
-DB Fill the ITEM table on CORP with 2000 rows
-Check log for output statistics to ensure that transport.max.bytes.to.sync was updated and is being referenced
**INFO [corp-000] [DataExtractorService] [corp-000-dataextractor-5] Reached the total byte threshold after 1 of 5 batches were extracted for node '002' (extracted 1273 bytes, the max is 1000). The remaining batches will be extracted on a subsequent sync.**
-Stop SymmetricDS Pro and upgrade to ver 3.9.2, restart
-DB Fill the ITEM table on CORP with 2000 rows
-Check log for output statistics to ensure that transport.max.bytes.to.sync was updated and is being referenced
**INFO [corp-000] [DataExtractorService] [corp-000-dataextractor-13] Reached the total byte threshold after 1 of 3 batches were extracted for node '001' (extracted 452992 bytes, the max is 1000). The remaining batches will be extracted on a subsequent sync.**

My log output appears to be the same after upgrade. Can you send me additional information on your setup? Do you have stream.to.file.enabled = true or any other parameters that are not set to their default values?

woehrl01

2018-01-10 19:05

reporter   ~0001102

Thank you for your feedback kpatierno!
Here are additional infos to reproduce:

Configuration:
stream.to.file.enabled=true
initial.load.use.extract.job.enabled=true
initial.load.extract.and.send.when.staged=true

I get that output if I do an initial load

rwfuller

2018-01-15 15:29

reporter   ~0001105

This issue is similar to what was seen here, https://sourceforge.net/p/symmetricds/discussion/739236/thread/22260ca7/, and appears to have been introduced in this issue, http://www.symmetricds.org/issues/view.php?id=2577

mmichalek

2018-03-07 19:03

administrator   ~0001144

Hi Lukas,

Can you provide a more detail around the scenario where you see this problem? For example, are you doing a load/initial load/table reload request or anything like that? We have tested normal push/pull functionality with the SymmetricDS "demo" setup as Kenny (kpatierno) described above. When we test, we see the byte threshold get hit and respected, per the logic in the current DataExtractorService:885:

                    if (status.byteExtractCount >= maxBytesToSync && status.batchExtractCount < activeBatches.size()) {
                        log.info(
                                "Reached the total byte threshold after {} of {} batches were extracted for node '{}' (extracted {} bytes, the max is {}). "
                                        + "The remaining batches will be extracted on a subsequent sync.",
                                new Object[] { status.batchExtractCount, activeBatches.size(), targetNode.getNodeId(), status.byteExtractCount, maxBytesToSync });
                        status.shouldExtractSkip = true;
                    }

woehrl01

2018-03-07 19:49

reporter   ~0001145

Hi Mark,

sure, sorry for missing that information. It happens during an initial load, in a push/pull scenario. (Client is pushing, Server waits for pull).
I'm using the initial load extract job. If I do a simple trigger based sync the batches are respected.

(probably not relevant: The server is sql server and the client is sqlite. And I'm starting symds with the /bin/sym.bat file)

Which kind of additional informations would be helpful for you?

woehrl01

2018-03-10 23:31

reporter   ~0001147

I just thought about this issue a bit. I think the problem in the current implementation (without my PR) arises if the sending of the data is much slower than the extraction of the database.

In that case all the extraction could be already done, before the sending of data limits the transmission,which than transmits all the extracted data. Even so it would be more than the limit.

mmichalek

2018-03-14 15:48

administrator   ~0001150

Lukas, thank you for that additional information. I am reviewing again with these factors in mind.

mmichalek

2018-03-15 16:32

administrator   ~0001152

Lukas, we have reproduced the issue now. We discussed it with the development team, and we are thinking of adding a second parameter for the reload/initial load case. For a lot of SymmetricDS users, initial loads are a much different case than normal data sync, and they might want different thresholds for those cases. Perhaps the new parameter would be intial.load.transport.max.bytes.to.sync, and would have a higher default value than the existing transport.max.bytes.to.sync. Let us know if you have any concerns with that approach.

woehrl01

2018-03-15 16:47

reporter   ~0001153

Hi Mark, sounds like a great idea, I currently have changed that value to a much higher value (10Mb) especially for the initial load case. I think a second value could give a better fine tuning option.

As I created a pr do you want me to change it, or do you want to provide your own solution? I don't mind either :)

mmichalek

2018-03-21 16:56

administrator   ~0001155

Lukas, we were also thinking of increasing the default to at least 10 MB, so I am glad to hear you were thinking the same thing. If you are able to adjust your PR to use the new parameter, that would be great and I will review and accept it. If you don't get to get to it we will try and implement these changes in the next week or so. Thanks for your continued support of the project.

woehrl01

2018-03-21 20:20

reporter   ~0001156

Mark, I'll update my PR to use a new parameter with a default value of 10 MB. I think I can update it in the next day(s).

mmichalek

2018-03-21 22:01

administrator   ~0001157

Awesome, thank you Lukas.

woehrl01

2018-03-22 13:04

reporter   ~0001158

Mark, I updated the pull request. Let me know if it fit your needs.

mmichalek

2018-03-22 14:25

administrator   ~0001159

Thanks Lukas, that's perfect - it's merged and will be part of the build today or tomorrow. We really appreciate it. Please be aware we also bumped the transport.max.bytes.to.sync to 100MB and initial.load.transport.max.bytes.to.sync to 500 MB now, by default. See https://www.symmetricds.org/issues/view.php?id=3502

Related Changesets

SymmetricDS: 3.9 1b07622b

2018-01-08 03:47:33

woehrl01


Committer: GitHub Details Diff
0003356: transport.max.bytes.to.sync not respected Affected Issues
0003356
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataExtractorService.java Diff File

Issue History

Date Modified Username Field Change
2018-01-05 14:34 woehrl01 New Issue
2018-01-08 08:48 woehrl01 Note Added: 0001097
2018-01-10 16:04 kpatierno Note Added: 0001101
2018-01-10 19:05 woehrl01 Note Added: 0001102
2018-01-15 15:29 rwfuller Note Added: 0001105
2018-03-07 19:03 mmichalek Note Added: 0001144
2018-03-07 19:49 woehrl01 Note Added: 0001145
2018-03-10 23:31 woehrl01 Note Added: 0001147
2018-03-14 15:48 mmichalek Note Added: 0001150
2018-03-14 18:12 mmichalek Summary transport.max.bytes.to.sync not respected in 3.9 => transport.max.bytes.to.sync not respected in 3.9 during loads/reloads
2018-03-15 16:32 mmichalek Note Added: 0001152
2018-03-15 16:47 woehrl01 Note Added: 0001153
2018-03-21 16:56 mmichalek Note Added: 0001155
2018-03-21 20:20 woehrl01 Note Added: 0001156
2018-03-21 22:01 mmichalek Note Added: 0001157
2018-03-22 13:04 woehrl01 Note Added: 0001158
2018-03-22 14:14 mmichalek Assigned To => mmichalek
2018-03-22 14:14 mmichalek Status new => resolved
2018-03-22 14:14 mmichalek Resolution open => fixed
2018-03-22 14:14 mmichalek Fixed in Version => 3.9.6
2018-03-22 14:14 mmichalek Target Version => 3.9.6
2018-03-22 14:14 mmichalek Summary transport.max.bytes.to.sync not respected in 3.9 during loads/reloads => Add initial.load.transport.max.bytes.to.sync parameter. transport.max.bytes.to.sync not respected in 3.9 during loads/reload
2018-03-22 14:25 mmichalek Note Added: 0001159
2018-03-22 15:00 woehrl01 Changeset attached => SymmetricDS 3.9 1b07622b
2018-03-28 22:16 mmichalek Status resolved => closed