In an sftp import I files that are named like this:
Company_Name_Activities_2017-09-15T1452.csv
Based on the documentation I think I should be able to use this fiile mask:
name_{yyyy-MM-ddTHHmm}.csv
However, when I do this it says no such file exists.
Thank you for your suggestion, we will definitely consider it and post the results here as soon as any available.
Thanks Dmitry, but I've never written a lambda script before. I know very little programming.
I have an issue with this method of retrieving file names:
"For example, when you click Apply, current time is used and is compared with the file mask, i.e. if click Apply today at 19:45, the following file be searched - Company_Name_Activities_2017-09-19T1945.csv. If there is no file with this name you will get the "No such file" error."
Instead of using the current time to search for the file name, why not have a checkbox for 'most recent' and grab the file with the most recent date in the file name?
Most of the vendors and partners we buy data from use minutes and seconds in their file names. This update would make Skyvia far more useful for us.
Another solution that we are now implementing, because our file exports always contain an unpredictable hash in the file name, is a script that renames the most recently exported file with a generic filename, without the hash and the timestamp.
In our case, we have data files exported as CSVs to Amazon S3. There is a standard functionality in S3 to create a notification event for a created file. These notifications are passed to a Lambda function/script run in the same account, which copies the created files with predefined filenames into a different directory, for use by Skyvia.
You can try the following solution:
1) Ask your customer data platform to make export with hours, e.g.:
Company_Name_Activities_2017-09-20T11.csv
2) Use the following file mask:
Note: when adding new task and clicking Apply for the file mask, make sure that there is a file with the name that corresponds to the current time, i.e. click Apply today at 19:45, the following file should be available - Company_Name_Activities_2017-09-20T19.csv. In fact, you can add such a file manually in order to add the file and create import task successfully. Also, use your time zone here.
3) Set schedule, for example, like this:
Make sure, that the corresponding file exists before the package run.
Please tell us if this information helps.
I can talk to BlueConic (our customer data platform) about making more date formats for their exports. However, this is an export/import job that I need to run hourly. If I take the minute and hour off, Skyvia won't know which of that day's files are the most recent. I would at least need the hour on there for this to work.
As we have said above, it's not recommended to use seconds or fractions of seconds, and even minutes in the mask. The HHmm is reason for the error you are getting. For example, when you click Apply, current time is used and is compared with the file mask, i.e. if click Apply today at 19:45, the following file be searched - Company_Name_Activities_2017-09-19T1945.csv. If there is no file with this name you will get the "No such file" error.
That's why, we recommend you to use files without HHmm in the name and file mask like this one: Company_Name_Activities_{yyyy-MM-dd}.csv
Regarding the format, the program that is exporting the data adds the hour and minute. I don't think I can change that, unless I can put in some kind of wildcard for that part of the filename.
I did use Company_Name_Activities_{yyyy-MM-ddTHHmm}.csv. I should have been more precise in my example. I wrote name_ just as a placeholder. I just tried it again and am still getting the error "No such file"
the file is in a subfolder. I selected the subfolder in the CSV Folder field so I assume I don't add that in the file mask. Or do i need to input /subfolder//Company_Name_Activities_{yyyy-MM-ddTHHmm}.csv?
In your case, you should use the following file mask: Company_Name_Activities_{yyyy-MM-ddTHHmm}.csv
However, please note, it's not recommended to use seconds or fractions of seconds, and even minutes in the mask, since it's not guaranteed that package will run in exactly the specified time. When Skyvia is under peak load, package run may be delayed for a few seconds. And even if the difference of the package run time and the date/time in the file name is minor, the package run will fail.
We recommend you to use file mask like this one: Company_Name_Activities_{yyyy-MM-dd}.csv
