Client Backfill Guide

What is a "backfill"?

A backfill is the method by which we furnish existing catalogue data for you during your onboarding to Catalogue Events. It follows the identical process as our standard, day-to-day events and is delivered using the same protocol – through POST requests to your dedicated endpoint. Please note that this endpoint remains the same, regardless of whether it is for backfilling or regular events; you cannot designate separate endpoints for both types of events. As a result, both backfilled and regular events will be delivered to you simultaneously.

What is the duration of a backfill?

The backfill process operates at a rate of up to 400 requests per second (RPS). For a complete catalogue backfill of 30 million releases, it takes around 21 hours at this rate. It is advised you aim for your systems to be able to handle this rate effectively and be able to scale its support as day-to-day events will be sent at a significantly lower rate per second (single digits) once you have been onboarded. It is essential that you promptly acknowledge receipt of the data at or near this rate as requests will time out which will result in events permanently failing. If you are unable to handle the backfill rate, you may be able to receive, acknowledge, and then store the events somewhere while you process these at a lower rate but this is at your discretion.