Problem with File Mask

  • 64Views
  • Last Post 3 weeks ago
0
votes
Webster Jorgensen posted this 15 September 2017

In an sftp import I files that are named like this:

Company_Name_Activities_2017-09-15T1452.csv

Based on the documentation I think I should be able to use this fiile mask:

name_{yyyy-MM-ddTHHmm}.csv

However, when I do this it says no such file exists.

Order By: Standard | Newest | Votes
0
votes
Mariia Zaharova posted this 18 September 2017

In your case, you should use the following file mask: Company_Name_Activities_{yyyy-MM-ddTHHmm}.csv

However, please note, it's not recommended to use seconds or fractions of seconds, and even minutes in the mask, since it's not guaranteed that package will run in exactly the specified time. When Skyvia is under peak load, package run may be delayed for a few seconds. And even if the difference of the package run time and the date/time in the file name is minor, the package run will fail.

We recommend you to use file mask like this one: Company_Name_Activities_{yyyy-MM-dd}.csv

0
votes
Webster Jorgensen posted this 18 September 2017

I did use Company_Name_Activities_{yyyy-MM-ddTHHmm}.csv.  I should have been more precise in my example. I wrote name_ just as a placeholder.  I just tried it again and am still getting the error "No such file"

the file is in a subfolder.  I selected the subfolder in the CSV Folder field so I assume I don't add that in the file mask.  Or do i need to input /subfolder//Company_Name_Activities_{yyyy-MM-ddTHHmm}.csv?

 

0
votes
Webster Jorgensen posted this 18 September 2017

Regarding the format, the program that is exporting the data adds the hour and minute.  I don't think I can change that, unless I can put in some kind of wildcard for that part of the filename. 

0
votes
Mariia Zaharova posted this 19 September 2017

As we have said above, it's not recommended to use seconds or fractions of seconds, and even minutes in the mask. The HHmm is reason for the error you are getting. For example, when you click Apply, current time is used and is compared with the file mask, i.e. if click Apply today at 19:45, the following file be searched - Company_Name_Activities_2017-09-19T1945.csv. If there is no file with this name you will get the "No such file" error.

That's why, we recommend you to use files without HHmm in the name and file mask like this one: Company_Name_Activities_{yyyy-MM-dd}.csv

 

0
votes
Webster Jorgensen posted this 19 September 2017

I can talk to BlueConic (our customer data platform) about making more date formats for their exports. However, this is an export/import job that I need to run hourly.  If I take the minute and hour off, Skyvia won't know which of that day's files are the most recent. I would at least need the hour on there for this to work. 

0
votes
Mariia Zaharova posted this 20 September 2017

You can try the following solution:

1) Ask your customer data platform to make export with hours, e.g.: 

Company_Name_Activities_2017-09-20T11.csv

2) Use the following file mask:

Note: when adding new task and clicking Apply for the file mask, make sure that there is a file with the name that corresponds to the current time, i.e. click Apply today at 19:45, the following file should be available - Company_Name_Activities_2017-09-20T19.csv. In fact, you can add such a file manually in order to add the file and create import task successfully. Also, use your time zone here.

 

3) Set schedule, for example, like this:

Make sure, that the corresponding file exists before the package run.

 

Please tell us if this information helps.

0
votes
Dmitry Belakhov posted this 3 weeks ago

Another solution that we are now implementing, because our file exports always contain an unpredictable hash in the file name, is a script that renames the most recently exported file with a generic filename, without the hash and the timestamp.

In our case, we have data files exported as CSVs to Amazon S3. There is a standard functionality in S3 to create a notification event for a created file. These notifications are passed to a Lambda function/script run in the same account, which copies the created files with predefined filenames into a different directory, for use by Skyvia.

Close