• Stripe PaymentIntents API replication failures

Stripe PaymentIntents API replication failures

  • 93Views
  • Last Post 4 weeks ago
0
votes
Joseph Naujokas posted this 27 August 2021

All attempts at replicating the Stripe PaymentIntents table are failing in replication package (ID = 138211).

There are two kinds of failures:

1) Timeout errors connecting to the Stripe service - For example, run ID 66036558

The run log reports the following error: "Object error / Server did not respond within the specified timeout interval."

When I perform a manual GET operation on api.stripe.com/v1/payment_intents , even with limit=100 (the max),  I consistently get a response under 10 seconds.

Question for this failure: What is your timeout threshold for this API?  Can it be adjusted?

2) Failed rows - For example, run ID 66015945

The run log does not indicate any specific failure, it simply indicates the number of rows that failed.

Question for this failure: What is the cause of the failed rows?  And where would we find information about these failures in the future?

 

Thanks in advance.

 

Order By: Standard | Newest | Votes
0
votes
Olena Romanchuk posted this 4 weeks ago

Dear Joseph,


1. Please try the following:

Go to Connections in your workspace, open your Redshift connection, click the Advanced settings and increase the timeout values.

Please don't forget to save the connection. Run the package and inform us about the result.


2. By default Replication works in Bulk mode. It means that Skyvia writes data into multiple temporary CSV files, uploads them to Amazon S3 and then tells Redshift to import data from these CSV files. These actions are performed simultaneously, providing the best performance. After the CSV files are imported, they are deleted. However, when data is imported this way, it’s not possible to obtain a per-record error log. 


However, the package run log shows that there were the following errors occurred during the run:

  • BalanceTransactions | Value was either too large or too small for an Int32.: [Value: '-2182099740', Table: "BalanceTransactions", Column: "Amount", Id: '']

  • permission denied for relation paymentintents



Best regards,


Olena

Customer Support Engineer

Skyvia

 

0
votes
Joseph Naujokas posted this 4 weeks ago

I will try adjusting the Timeout value for Redshift to avoid future timeout errors.

Regarding "permission denied for relation paymentintents" - I confirmed that the Skyvia user did not have permissions on the Redshift "paymentintents" table.  I explicitly assigned all priveleges to the Skyvia user, and that change appears to have resolved the error.  Unfortunately, the target table only contains the successful rows - the package did not recover the failed rows. 

Does this mean I need to reset the Last Sync Time of the package in order to recover the failed rows for that table? 

If so, will the Last Sync Time result in a re-sync of ALL tables in this package?

 

0
votes
Joseph Naujokas posted this 4 weeks ago

Next question: Is it possible for you to manually trigger a re-sync of JUST the PaymentIntents table?

0
votes
Joseph Naujokas posted this 4 weeks ago

Also, regarding your recommendation to change the Redshift timeout value:

* The Command timeout value was 300 seconds (5 minutes) - I did NOT change that value

* The Connection timeout was 30 seconds - I changed that value to 60 seconds, but we are still seeing timeout errors

Questions:

* Did you make your suggestion to change the Redshift timeout based on evidence in the system logs indicating that Redshift was the source of the timeout error?  If that's the case, a) Why would the PaymentIntents load consistently throw a timeout error when other loads with much larger rowcounts do not? And b) What timeout value would you recommend, based on the log evidence?

* If Redshift is NOT the origin of the timeout error, and in fact the origin is the connection to Stripe...  a) Again, why would the PaymentIntents API consistently throw the timeout when other APIs with larger data sets consistently succeed?  And b) Is there any way you can adjust the Stripe timeout value?

 

0
votes
Olena Romanchuk posted this 4 weeks ago

Dear Joseph,

If you reset the LastSyncTime parameter and run the package, all the objects and data will be rewritten.

As per the error logs, the missing records belong to PaymentIntents and BalanceTransactions only. It makes sense to create a separate Replication Package and perform the full replication to rewrite the missing tables only.

As per the logs the timeout error was returned by Redshift server.

The time for reading the objects depends on a lot of factors, not only row count, but also column names length, number of columns, the relations between the objects and so on.

If you still get the timeout errors, you can try to increase the connection timeout value, for example to 3600 seconds. If the error persists even with this value, then further investigation is needed.

Best regards,

 

Olena

Customer Support Engineer

 

Skyvia

New Ticket
Categories
Popular Tags
Close