View Issue Details

IDProjectCategoryView StatusLast Update
0004045SymmetricDSBugpublic2019-08-23 16:46
Reporterhanes Assigned Toelong  
Prioritynormal 
Status closedResolutionfixed 
Product Version3.10.3 
Target Version3.10.4Fixed in Version3.10.4 
Summary0004045: Load Data request with multiple nodes and "create table" option fails to create foreign key batches for all but one node
DescriptionQueue up a Load Data request for multiple nodes with Create Table on. Use tables that have foreign keys. Only one of the nodes receives the "finalize" batches to set the foreign key constraints.
Steps To ReproduceSelect multiple nodes, click Load Data, select option to create tables
TagsNo tags attached.

Activities

hanes

2019-07-23 13:57

developer   ~0001563

Simple two table example. ITEM and ITEM_PRICE, in a parent / child relationship. Do a Load Data of that table, with Create Table on. (It will fail if you do a load with just the ITEM table and Create Table, too)

[corp-000] - RouterService - Found 1 table reload requests to process.
[corp-000] - DataService - Queueing up an initial load to node 001
[corp-000] - DataService - Before sending load 2 to target node store:001:001 create table events were sent for 2 tables
[corp-000] - DataService - Table reload request(s) for load id 2 have been processed.
[corp-000] - PurgeService - Purged all 2 incoming batch for node 001
[corp-000] - ConfigurationChangedDataRouter - About to refresh the cache of node security because new configuration came through the data router
[corp-000] - RouterService - Routed 4 data events in 5 ms
[corp-000] - DataExtractorService - Batch 001-13 is not ready for delivery. It is currently scheduled for extraction.
[corp-000] - PullUriHandler - 7 data and 5 batches sent during pull request from store:001:001
[store-001] - ConfigurationChangedDatabaseWriterFilter - About to refresh the cache of node security because new configuration came through the data loader
[store-001] - ConfigurationChangedDatabaseWriterFilter - About to refresh the cache of nodes because new configuration came through the data loader
[store-001] - DefaultDatabaseWriter - About to create table using the following definition: <?xml version="1.0"?>
<!DOCTYPE database SYSTEM "http://db.apache.org/torque/dtd/database">
<database name="dataextractor">
    <table name="ITEM">
        <column name="ID" primaryKey="true" required="true" type="VARCHAR" size="50">
            <platform-column name="h2" type="VARCHAR" size="50"/>
        </column>
        <column name="DESCRIPTION" type="VARCHAR" size="255">
            <platform-column name="h2" type="VARCHAR" size="255"/>
        </column>
    </table>
</database>
[store-001] - H2DatabasePlatform - Running alter sql:
CREATE TABLE "STORE-001"."PUBLIC"."ITEM"(
    "ID" VARCHAR(50) NOT NULL,
    "DESCRIPTION" VARCHAR(255) NULL,
    PRIMARY KEY ("ID")
);

[store-001] - DefaultDatabaseWriter - About to create table using the following definition: <?xml version="1.0"?>
<!DOCTYPE database SYSTEM "http://db.apache.org/torque/dtd/database">
<database name="dataextractor">
    <table name="ITEM_PRICE">
        <column name="ITEM_ID" primaryKey="true" required="true" type="VARCHAR" size="50">
            <platform-column name="h2" type="VARCHAR" size="50"/>
        </column>
        <column name="STORE_ID" primaryKey="true" required="true" type="VARCHAR" size="5">
            <platform-column name="h2" type="VARCHAR" size="5"/>
        </column>
        <column name="PRICE" required="true" type="DECIMAL" size="65535,32767">
            <platform-column name="h2" type="DECIMAL" size="65535" decimalDigits="32767"/>
        </column>
    </table>
</database>
[store-001] - H2DatabasePlatform - Running alter sql:
CREATE TABLE "STORE-001"."PUBLIC"."ITEM_PRICE"(
    "ITEM_ID" VARCHAR(50) NOT NULL,
    "STORE_ID" VARCHAR(5) NOT NULL,
    "PRICE" DECIMAL(65535,32767) NOT NULL,
    PRIMARY KEY ("ITEM_ID", "STORE_ID")
);

[store-001] - PullService - Pull data received from corp:000:000 on queue default. 7 rows and 5 batches were processed. (sym_table_reload_request, sym_node_security, sym_node, item, item_price)
[corp-000] - DataExtractorService - Batch 001-13 is not ready for delivery. It is currently scheduled for extraction.
[corp-000] - NodeCommunicationService - extract will use 20 threads
[corp-000] - DataExtractorService - Starting request 2 to extract table ITEM into batches 13 through 13 for node 001.
[corp-000] - DataExtractorService - Failed on request 2 to extract table ITEM into batches 13 through 13 for node 001
[corp-000] - NodeCommunicationService - Failed to execute EXTRACT for node 001 and channel default
java.lang.NullPointerException
    at org.jumpmind.symmetric.service.impl.DataExtractorService.checkSendDeferredConstraints(DataExtractorService.java:2266)
    at org.jumpmind.symmetric.service.impl.DataExtractorService.execute(DataExtractorService.java:2085)
    at org.jumpmind.symmetric.service.impl.NodeCommunicationService$1.run(NodeCommunicationService.java:515)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
[store-001] - ConfigurationChangedDataRouter - About to refresh the cache of node security because new configuration came through the data router
[store-001] - RouterService - Routed 4 data events in 19 ms
[corp-000] - RouterService - The 'default' channel is in common batch mode
[corp-000] - RouterService - The 'default' channel for the 'corp' node group has only default routers assigned to it. Change data won't be selected during routing
[corp-000] - RouterService - Routed 1 data events in 5 ms
[corp-000] - DataExtractorService - Starting request 2 to extract table ITEM into batches 13 through 13 for node 001.
[corp-000] - DataExtractorService - Resetting status of request 2 to extract table ITEM into batches 13 through 13 for node 001
[corp-000] - DataExtractorService - Failed on request 2 to extract table ITEM into batches 13 through 13 for node 001
[corp-000] - NodeCommunicationService - Failed to execute EXTRACT for node 001 and channel default
java.lang.NullPointerException
    at org.jumpmind.symmetric.service.impl.DataExtractorService.checkSendDeferredConstraints(DataExtractorService.java:2266)
    at org.jumpmind.symmetric.service.impl.DataExtractorService.execute(DataExtractorService.java:2085)
    at org.jumpmind.symmetric.service.impl.NodeCommunicationService$1.run(NodeCommunicationService.java:515)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
[corp-000] - RouterService - Routed 1 data events in 6 ms

elong

2019-07-23 14:06

developer   ~0001564

I just pushed a check for the null pointer. If the child requests are null, then it means you didn't get a parent-child relationship between two load requests. I think you have to submit the load requests at the exact same time or else they will be treated separately. Maybe we could talk about allowing a little wiggle room on the time.

Related Changesets

SymmetricDS: 3.10 079e5c5a

2019-07-22 10:41:25

admin

Details Diff
0004045: Load Data request with multiple nodes and "create table" option
fails to create foreign key batches for all but one node
Affected Issues
0004045
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataExtractorService.java Diff File

SymmetricDS: 3.10 d48fae75

2019-07-23 10:03:25

admin

Details Diff
0004045: Load Data request with multiple nodes and "create table" option
fails to create foreign key batches for all but one node
Affected Issues
0004045
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataExtractorService.java Diff File

Issue History

Date Modified Username Field Change
2019-07-17 13:22 hanes New Issue
2019-07-17 13:23 hanes Status new => confirmed
2019-07-22 14:40 elong Assigned To => elong
2019-07-22 14:40 elong Status confirmed => assigned
2019-07-22 14:43 elong Status assigned => resolved
2019-07-22 14:43 elong Resolution open => fixed
2019-07-22 14:43 elong Fixed in Version => 3.10.4
2019-07-22 14:43 elong Target Version => 3.10.4
2019-07-22 14:43 elong Summary Load Data request with mutiple nodes and "create table" option fails to create foreign key batches for all but one node => Load Data request with multiple nodes and "create table" option fails to create foreign key batches for all but one node
2019-07-22 14:43 elong Steps to Reproduce Updated View Revisions
2019-07-22 15:00 admin Changeset attached => SymmetricDS 3.10 079e5c5a
2019-07-23 13:57 hanes Status resolved => feedback
2019-07-23 13:57 hanes Resolution fixed => reopened
2019-07-23 13:57 hanes Note Added: 0001563
2019-07-23 14:06 elong Note Added: 0001564
2019-07-23 14:07 elong Status feedback => resolved
2019-07-23 14:07 elong Resolution reopened => fixed
2019-07-23 15:00 admin Changeset attached => SymmetricDS 3.10 d48fae75
2019-08-23 16:46 admin Status resolved => closed