View Issue Details

IDProjectCategoryView StatusLast Update
0004114SymmetricDSBugpublic2019-10-16 03:43
ReporterZBY Assigned To 
Prioritynormal 
Status newResolutionopen 
Product Version3.10.4 
Summary0004114: Several problems on setting HBase as a load only node
DescriptionEnvironment: Ubuntu18.04.03, hbase-2.0.0, hadoop-2.9.2, and SymmetricDS 3.10.4.

I have two nodes. one node is called "master" and the other is called "spare". The master node connects to MySQL. The spare node set the Hbase as the load only node, and MySQL is used for SymmetricDS runtime usage.

Here are the problems:

1. I add the the phoenix-5.0.0-HBase-2.0-client.jar to the /lib folder of SymmetricDS, and when I start the SymmetricDS, it gives the following error information:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/zby/symmetric1/lib/phoenix-5.0.0-HBase-2.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/zby/symmetric1/web/WEB-INF/lib/slf4j-log4j12-1.7.26.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: The following loggers will not work because they were created
SLF4J: during the default configuration phase of the underlying logging system.
SLF4J: See also http://www.slf4j.org/codes.html#substituteLogger
SLF4J: org.jumpmind.symmetric.AbstractSymmetricEngine
...

I solve this problem by deleting the /org/slf4j folder of the phoenix-5.0.0-HBase-2.0-client.jar.

2. I use the initial load to create table in HBase because I don't make sure how tables in Mysql are transformed.
And, the nodes give the following error information:

[master-000] - AcknowledgeService - The outgoing batch 001-11 failed: [42K01,1007] ERROR 1007 (42K01): A non primary key column may only be declared as not null on tables with immutable rows. columnName=PERSONS.Name

[spare-001] - HbasePlatform - Running alter sql:
CREATE TABLE "PERSONS"(
    "Id" INTEGER NOT NULL,
    "Name" VARCHAR(25) NOT NULL,
    CONSTRAINT NAME PRIMARY KEY ("Id")
);

[spare-001] - JdbcSqlTemplate - ERROR 1007 (42K01): A non primary key column may only be declared as not null on tables with immutable rows. columnName=PERSONS.Name. Failed to execute: CREATE TABLE "PERSONS"(
    "Id" INTEGER NOT NULL,
    "Name" VARCHAR(25) NOT NULL,
    CONSTRAINT NAME PRIMARY KEY ("Id")
)
[spare-001] - DefaultDatabaseWriter - Failed to alter table using the following xml: <?xml version="1.0"?>
<!DOCTYPE database SYSTEM "http://db.apache.org/torque/dtd/database">
<database name="dataextractor">
    <table name="persons">
        <column name="Id" primaryKey="true" required="true" type="INTEGER" size="10">
            <platform-column name="mysql" type="INT" size="10"/>
        </column>
        <column name="Name" required="true" type="VARCHAR" size="25">
            <platform-column name="mysql" type="VARCHAR" size="25"/>
        </column>
    </table>
</database>
org.jumpmind.db.sql.UniqueKeyException: java.sql.SQLException: ERROR 1007 (42K01): A non primary key column may only be declared as not null on tables with immutable rows. columnName=PERSONS.Name
    at org.jumpmind.db.sql.AbstractSqlTemplate.translate(AbstractSqlTemplate.java:296)
    ...

I create a new table where no column is declared as not null to avoid the issue. But i think it is unreasonable because I see the examples in "http://phoenix.apache.org/language/index.html#create_table" allow non primary key columns to be declared as not null.

3. When I update a row in MySQL, it gives the following error information:
[master-000] - AcknowledgeService - The outgoing batch 001-19 failed: [42P00,602] ERROR 602 (42P00): Syntax error. Missing "STATISTICS" at line 1, column 8.

[spare-001] - DefaultDatabaseWriter - Failed to process update event in batch 000-19 on channel 'persons_channel'.
Failed sql was: update "PERSONS" set "Name" = ? where "Id" = ?
Failed sql state and code: 42P00 (602)
Failed pk data was: "2"
Failed row data was: "2",""
Failed old data was: "2","b"

org.jumpmind.db.sql.SqlException: Exception while preparing sql [update "PERSONS" set "Name" = ? where "Id" = ?]
    at org.jumpmind.db.sql.JdbcSqlTransaction.prepare(JdbcSqlTransaction.java:439)
    at org.jumpmind.symmetric.io.data.writer.DefaultDatabaseWriter.prepare(DefaultDatabaseWriter.java:902)
    at org.jumpmind.symmetric.io.data.writer.DefaultDatabaseWriter.update(DefaultDatabaseWriter.java:492)
    at org.jumpmind.symmetric.io.data.writer.AbstractDatabaseWriter.write(AbstractDatabaseWriter.java:189)
    ...

I make several tests, and all updates are failed.

4. When I run the SymmetricDS on windows, and it connect to Hbase which is runned on Ubuntu, it will gives the following information:
PS D:\MY_TEST\Hbase\symmetric1\engines> ../bin/symadmin --engine master-000 create-sym-tables
Log output will be written to D:\MY_TEST\Hbase\symmetric1/logs/symmetric.log
[] - AbstractCommandLauncher - Option: name=engine, value={master-000}
[] - Shell - Did not find winutils.exe: {}
java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
        at org.apache.hadoop.util.Shell.fileNotFoundException(Shell.java:534)
        at org.apache.hadoop.util.Shell.getHadoopHomeDir(Shell.java:555)
        at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:578)
        at org.apache.hadoop.util.Shell.<clinit>(Shell.java:675)
        at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:78)
        at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1625)

I don't konw whether it is because my windows environment or phoenix or SymmetricDS.
Steps To Reproduce1. Create two nodes. one node is called "master" and the other is called "spare". The master node connects to MySQL. The spare node set the HBase as the load only node, and MySQL is used for SymmetricDS runtime usage.

2. Add the the phoenix-5.0.0-HBase-2.0-client.jar to the /lib folder of SymmetricDS, and start the SymmetricDS. Problem 1 should occur.

3. To avoid Problem 1, the /org/slf4j folder of the phoenix-5.0.0-HBase-2.0-client.jar need to be deleted.

Create a table where a non primary key column is declared as not null in MySQL like the following statements:
CREATE TABLE Persons
(
Id int NOT NULL,
Name varchar(25) NOT NULL,
PRIMARY KEY (Id)
);

Add "initial.load.create.first=true" to the "master" node's property file. Start the SymmetricDS and send an initial load like the following statement:
sudo ../bin/symadmin -engine master-000 reload-node 001

Problem 2 should occur.

4. Create a table where no non primary key column is declared as not null in MySQL to avoid Problem 2.
Start the SymmetricDS, send an initial load to create the corresponding table in HBase.
Then, update a row in the table of MySQL, Problem 3 should occurred.
(I did the step 4 after deleting the old SymmetricDS folders. However, I'm not sure if this is necessary.)

5.I don't install Hadoop or HBase on windows and just install the SymmetricDS with the phoenix-5.0.0-HBase-2.0-client.jar.
when starting the SymmetricDS, Problem 4 should occur.
Additional InformationBecause the error information are too long, I only show the first few lines.

As a supplement, I provide my node's property file where the private information is covered by "******".
Tagsinitial/partial load, logging, statistics

Activities

There are no notes attached to this issue.

Issue History

Date Modified Username Field Change
2019-10-16 03:43 ZBY New Issue
2019-10-16 03:43 ZBY Tag Attached: initial/partial load
2019-10-16 03:43 ZBY Tag Attached: logging
2019-10-16 03:43 ZBY Tag Attached: statistics