For large files we can run a supervised off-cycle DW workflow. This is to prevent a major slow-down of the next regular daily refresh cycle. Offcycle DW runs to process are necessary if the transaction volume exceeds 2x your normal daily load volume. Otherwise, the normal pipeline should be able to handle it.
By "supervised" we mean that Support need to coordinate a slot internally, where we can ensure computing resources are available and will complete the processing before the next automated cycle starts. For these reasons this needs to be done during regular working hours.
This is the process to follow:
- Do not place large off-cycle files in the regular SFTP processing directory, since these can inadvertently be pulled into the regular refresh cycle if there was not enough time for us to intervene. Place them in a staging directory and notify Support by raising a ticket
-Please ensure that the file's column naming conventions match the production file naming conventions exactly. Failure to comply may result in errors or, at the least, data not loaded in all of the columns..
- To your Support ticket, please add the following details:
- File names and location in the FTP (note - do not place files in the production SFTP processing directory)
- Verified record count for each file (this helps us assess whether we can run multiple cycles on a given day).
- A brief explanation of why these files are being loaded
Since our support team needs to coordinate slotting internally, please understand that these files might not be processed the same day. Support will communicate their best estimate for when the files will be ingested and then validate when they were ingested.
Caveats
- Large data ingestion runs may cause errors or failures with the pipeline,thus requiring intervention
- In the event of data skews or data integrity problems caused by the offcycle ingestion, additional Professional Services team members may need to be involved
- Offcycle processing may still impact the next daily cycle, either by delaying it, or causing it to run longer than usual.