• Export from HubSpot to SFTP very slow with filter conditions

Export from HubSpot to SFTP very slow with filter conditions

  • 181Views
  • Last Post 22 November 2021
0
votes
Margaret Ngai posted this 15 October 2021

We are trying to export records from the HubSpot EmailEvents table once a day.

When we specific just 1 condition - CreatedAt equals YESTERDAY, the export runs within 1 minute and exported over 2800 records. However, we only want records that matches certain EmailCampaignId. Once we added some OR conditions with EmailCampaignId, the export ran for over 3h then ended with error A task was canceled. [Code: RC_03]

Is this something not doable with Skyvia?

sftp hubspot

Attached Files

Order By: Standard | Newest | Votes
0
votes
Nataliia Nikulina posted this 15 October 2021

Hello Margaret,

 

Thank you for contacting Skyvia Support team.

 

We are sorry for the inconvenience with this.

The "A task was canceled" error is an internal Skyvia issue and our developers are working on eliminating it completely.

We do our best to prevent this error to occur in the future.

 

Please let us know if everything works fine now.

 

 

Best regards,

Nataliia,

Customer Support Engineer

 

 

0
votes
Margaret Ngai posted this 20 October 2021

I tried running the export again. First time it ran successfully in just under 4 hours. Second & third time, it errored out (once at the 25 min mark and once just after 3 hours). The error message is a long string of HTML. It looks like a HubSpot API / Cloudflare error. Can you please advise?

<!DOCTYPE html> <!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]--> <!--[if IE 7]> <html class="no-js ie7 oldie" lang="en-US"> <![endif]--> <!--[if IE 8]> <html class="no-js ie8 oldie" lang="en-US"> <![endif]--> <!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]--> <head> <title>Origin DNS error | api.hubapi.com | Cloudflare</title> <meta charset="UTF-8" /> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" /> <meta name="robots" content="noindex, nofollow" /> <meta name="viewport" content="width=device-width,initial-scale=1" /> <link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/main.css" type="text/css" media="screen,projection" /> <script defer src="https://api.radar.cloudflare.com/beacon.js"></script> </head> <body> <div id="cf-wrapper"> <div class="cf-alert cf-alert-error cf-cookie-error hidden" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div> <div id="cf-error-details" class="p-0"> <header class="mx-auto pt-10 lgt-6 lgx-8 w-240 lg:w-full mb-15 antialiased"> <h1 class="inline-block md:block mr-2 md:mb-2 font-light text-60 md:text-3xl text-black-dark leading-tight"> <span data-translate="error">Error</span> <span>1016</span> </h1> <span class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed">Ray ID: 6a138b95267c70e2 &bull;</span> <span class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed">2021-10-20 16:150 UTC</span> <h2 class="text-gray-600 leading-1.3 text-3xl lg:text-2xl font-light">Origin DNS error</h2> </header> <section class="w-240 lg:w-full mx-auto mb-8 lgx-8"> <div id="what-happened-section" class="w-1/2 md:w-full"> <h2 class="text-3xl leading-tight font-normal mb-4 text-black-dark antialiased" data-translate="what_happened">What happened?</h2> <p>You've requested a page on a website (api.hubapi.com) that is on the <a data-orig-proto="https" data-orig-ref="www.cloudflare.com/5xx-error-landing/" target="_blank">Cloudflare</a> network. Cloudflare is currently unable to resolve your requested domain (api.hubapi.com). </div> <div id="resolution-copy-section" class="w-1/2 mt-6 text-15 leading-normal"> <h2 class="text-3xl leading-tight font-normal mb-4 text-black-dark antialiased" data-translate="what_can_i_do">What can I do?</h2> <p><strong>If you are a visitor of this website:</strong><br />Please try again in a few minutes.</p><p><strong>If you are the owner of this website:</strong><br />Check your DNS settings. If you are using a CNAME origin record, make sure it is valid and resolvable. <a rel="noopener noreferrer" href="https://support.cloudflare.com/hc/en-us/articles/234979888-Error-1016-Origin-DNS-error">Additional troubleshooting information here.</a></p> </div> </section> <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 smy-4 smx-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300"> <p class="text-13"> <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">6a138b95267c70e2</strong></span> <span class="cf-footer-separator sm:hidden">&bull;</span> <span class="cf-footer-item sm:block sm:mb-1"><span>Your IP</span>: 40.118.246.204</span> <span class="cf-footer-separator sm:hidden">&bull;</span> <span class="cf-footer-item sm:block sm:mb-1"><span>Performance &amp; security by</span> <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing" id="brand_link" target="_blank">Cloudflare</a></span> </p> </div><!-- /.error-footer --> </div><!-- /#cf-error-details --> </div><!-- /#cf-wrapper --> <script type="text/javascript"> window._cf_translation = {}; </script> </body> </html>

0
votes
Nataliia Nikulina posted this 21 October 2021

Hello Margaret,

 

Thank you for contacting Skyvia Support team.

 

The following issue does not occur on Skyvia's end and it looks like this has been fixed. 

As far as I see, the last launch was successful. Could you please confirm if everything is working fine now?

 

We look forward to hearing from you.

 

Best regards,

Nataliia,

Customer Support Engineer 

0
votes
Margaret Ngai posted this 22 October 2021

It is still not running. The latest error from this morning's run (Oct 22) is "Internal Error. Please contact Support for more details.". The error does not show on the website - I got that in my error notification email.

As you can see in the screenshot, it failed half the time over the last 7 runs. We have this scheduled to run each morning to monitor over the weekend. Please let us know how we can make this work consistently.

0
votes
Nataliia Nikulina posted this 26 October 2021

Hello Margaret,

 

Thank you for the update.

 

As we previously mentioned, we continue to work on our internal Skyvia issues and our developers are working on eliminating them completely.

We do our best to prevent this error to occur in the future. We will let you know when the issue is fixed.

Please feel free to contact us in case of any questions or issues. 

 

Best regards,

Nataliia,

Customer Support Engineer 

0
votes
Margaret Ngai posted this 28 October 2021

Hi Nataliia,

This process ran for 5 days with no error, and last night's run generated another internal error. How are we able to use Skyvia for any production work if the process can't even be stable for a week? What are these errors and when can we expect this process to run consistently?

Margaret

0
votes
Nataliia Nikulina posted this 02 November 2021

Hello Margaret,

 

Sorry for the inconvenience with this. We have passed your request to our development team.

We will contact you when we have any updates.

Feel free to contact us in case of any questions or issues.

 

Best regards,

Nataliia,

Customer Support Engineer

0
votes
Margaret Ngai posted this 06 November 2021

Hi Nataliia,

This week, we saw a new error Exception of type 'System.OutOfMemoryException' was thrown.

This happened a couple of times already. Is there any update?

Margaret

0
votes
Nataliia Nikulina posted this 08 November 2021

Hello Margaret,

 

Thank you for contacting Skyvia Support team.

 

We have investigated the performance issue in your package. The error occurs

because filtering is performed via the cache. This is related to HubSpot API limitations.

 

The filtering functionality has been improved in a HubSpot connector.

 

The following filters were added in a HubSpot - by EmailCampaignId, Recipient, and Type.

 

Note that this filter is used with an equal sign and without an OR operator. So to enable direct filtering (not through cache) you can modify the package to have two filters - CreatedAt and EmailCampaignId with one Id only in the package.

 

Please note: the changes will be available only after the app updates.

We will let you know when they are available.

 

Best Regards,

Nataliia,

Customer Support Engineer

0
votes
Margaret Ngai posted this 12 November 2021

Hi Nataliia,

I'm not sure I understand the solution.

  • What do you mean by the changes will be available after the app updates? You mean the next version of Skyvia updates?
  • You said the package can only ontain 1 EmailCampaignId? The package we are running currently selects 52 EmailCampaignIds. How is this new function supposed to help?

A related question - We usually find running the package again works. Is there a way to re-run the package automatically if there is an error? Or can time of day be an impact with system resources (as the package is scheduled to run at 2am ET, and we usually re-run the package manually if needed during office hour ET)?

Margaret

0
votes
Nataliia Nikulina posted this 17 November 2021

Hello Margaret,

 

Thank you for the update.

 

Please be informed that the app was updated and the changes are available now.

Unfortunately, it will not be possible to select all 52 campaigns in one package as the filter is used with an equal sign and without an OR operator. So to enable direct filtering (not through cache) you can modify the package to have two filters - CreatedAt and EmailCampaignId with one Id only in the package.

Feel free to make the tests on your end.

Currently failed package does not relaunch automatically once failed.

Please note, when Skyvia is under a peak load, a package run may be delayed for a few seconds or even minutes.

We can advise adjusting the schedule so that packages do not run, for example, at 10:00, 12:00, 11.30, 12.30 etc.

 

Please let us know if we may help you more.

 

Best regards,

Nataliia,

Customer Support Engineer

 

0
votes
Margaret Ngai posted this 17 November 2021

Hi Nataliia,

We won't be able to run one package for each ID. First we are only allowed to run 50 scheduled pacakges with our account. Also, that will be a significant change to the process currently setup to import these files. So the new feature is basically not useable for us.

As for the load, this package currently take 3.5 hours to 6 hours to run. We are flexible on the start time so we can definitely adjust that to maximize the chance of success. It is currently schedule to run at 1am Eastern Time (UTC-05:00). Can you advise what would be a non peak time to schedule this package? I'm not sure what time zone you are referring to in your examples, and whether you mean to avoid starting on the hour & half past tthe hour in general, or to avoid those specific times in a particular time zone.

Margaret

0
votes
Nataliia Nikulina posted this 22 November 2021

Hello Margaret,

 

Thank you for the update. 

 

We are sorry to hear that a new feature is not useful for you.

Please note that the server load is approximately the same during the day.

As we mentioned, it is not recommended to launch the packages at x : 00, x : 30 of each hour

as this is the most loaded time of the day. 

 

Let us know if we may help you more.

 

Best regards,

Nataliia,

Customer Support Engineer

New Ticket
Categories
Popular Tags
Close