An incremental load doesn’t pick up the length of the column

  • 246Views
  • Last Post 04 September 2017
0
votes
vincent esterly posted this 25 August 2017

Salesforce data Replication to Amazon Redshift.

I've created a standard incremental replication package. The initial load failed. The reason is the [Name] varchar(80) column. Redshift's varchar size is in bytes, not characters. But the [Name] column in Salesforce can contain not only single-byte characters. Therefore I got the error: "String length exceeds DDL length”.

I edited column length for the [Name] column (from 80 to 320) in the incremental replication package. And the initial load works properly. I checked the DDL for this table and the [Name] column is varchar(320) as expected.

But the following incremental load failed (once a day).  I queried STL_LOAD_ERRORS to find out information about the error. And I see the same error: "String length exceeds DDL length”. The query is “COPY [table name] plus guid …”, colname is “name”, type is “varchar” and col_length is 80. But I expected varchar(320). So a temporary table is created to load data. But the incremental load doesn’t pick up the change of the column length. And the default value varchar 80 is used instead of varchar(320).

 

Are there any ways to work around it?

Order By: Standard | Newest | Votes
0
votes
Mariia Zaharova posted this 28 August 2017

I edited column length for the [Name] column (from 80 to 320) in the incremental replication package. And the initial load works properly. I checked the DDL for this table and the [Name] column is varchar(320) as expected.

You edited the column length in the replication package, not directly in the database? Is this correct?

Looking forward to your reply.

0
votes
vincent esterly posted this 28 August 2017

Yes, I edited the column length in the replication package (from 80 to 320). I didn't execute any DDL commands in Redshift.

Steps to reproduce.
1. Empty Redshift DB.
2. Create 
incremental replication package ('Incremental Updates' and 'Create tablescheck boxes are selected).
3. Edit 
column length for the certain column.
4. Run. 
The initial load is OK.
5. T
he following incremental load failed (once a day).

 

0
votes
Mariia Zaharova posted this 29 August 2017

Thank you for the reply. We are investigating the issue.

We will contact you as soon as possible.

0
votes
Mariia Zaharova posted this 31 August 2017

We have fixed the issue and updated Skyvia.

Please try your scenario again and notify us about the results.

0
votes
vincent esterly posted this 04 September 2017

The incremental load was run 4 times after your modification. And there are no errors on the Redshift side. Thank you for solving the problem so quickly.

 

Nevertheless, the detailed log contains a number of failed rows. I checked these objects and there is no difference between Salesforce and Redshift for now. Most probably there are some issues with the Run History pane. Anyway, I will create a separate ticket if there is a data inconsistency between Salesforce and Redshift. 

Close