For large files we can run a supervised off-cycle DW workflow. This is to prevent a major slow-down of the next regular daily refresh cycle. Offcycle DW runs to process are only necessary if the transaction volume exceeds 2x your normal daily load volume. Otherwise, the normal pipeline should be able to handle it.
By "supervised" we mean that Support need to coordinate a slot with the IT-Ops team, where they can ensure computing resources are available and will complete the processing before the next automated cycle starts. For these reasons this needs to be done during regular working hours.
This is the process to follow:
- Do not place large off-cycle files in the regular SFTP processing directory, since these can inadvertently be pulled into the regular refresh cycle if there was not enough time for us to intervene. Place them in a staging directory and notify Support by raising a ticket
- To your Support ticket, please add the following details:
- File names and location in the FTP (note - do not place files in the production SFTP processing directory)
- Verified record count for each file (this helps us assess whether we can run multiple cycles on a given day).
- A brief explanation of why these files are being loaded
Since our support team needs to coordinate slotting with IT-Ops, please understand that these files might not be processed the same day. Support will communicate their best estimate for when the files will be ingested and then validate when they were ingested.