Incremental sync

The shape

1. Take an event watermark.
2. Backfill by paging the list endpoints.
3. Drain the change feed from the watermark, for ever.
4. Use updated_after only to recover, never as your main loop.

Rule 1: take the watermark before the backfill

def cold_start(session, store):
    # 1. Watermark FIRST. One page, limit=1, keep next_after.
    head = session.get(f"{BASE}/events", params={"limit": 1}, timeout=30).json()
    watermark = head["next_after"]

    # 2. Backfill. Slow, and that is fine — it happens once.
    backfill_jobs(session, store)
    backfill_applicants(session, store)
    backfill_applications(session, store)

    # 3. Only now commit the watermark, so a crash mid-backfill restarts the
    #    whole cold start rather than resuming from a point we never reached.
    store.save_sequence(watermark)

Rule 2: the sequence number is the state

Not a timestamp. Events are written by concurrent transactions and can become visible slightly out of wall-clock order, so “everything since 10:04” can miss an event stamped 10:03 that committed at 10:05.

sequence is totally ordered and has no such window.

Rule 3: save the watermark after the work, not before

def drain(session, store):
    after = store.load_sequence()

    while True:
        params = {"limit": 100}
        if after is not None:
            params["after"] = after
        body = session.get(f"{BASE}/events", params=params, timeout=30).json()

        for event in body["data"]:
            # At-least-once delivery: the same event id can arrive more than
            # once, so every handler must be idempotent and this guard is not
            # an optimisation.
            if store.already_processed(event["id"]):
                continue
            dispatch(session, store, event)
            store.mark_processed(event["id"])

        if body["data"]:
            # AFTER the page is durably handled. Saving it first turns a crash
            # halfway through a page into permanently skipped events.
            after = body["next_after"]
            store.save_sequence(after)

        if not body["has_more"]:
            return

Save-then-work loses events on a crash. Work-then-save replays them, and replaying is free if your handlers are idempotent — which they must be anyway, because delivery is at-least-once even without crashes.

Rule 4: re-read the resource, do not trust the payload

Event payloads are thin: identifiers and a few scalars. Read the resource by handle when you handle the event.

previous_attributes is the exception worth using directly: it tells you which fields actually changed, so you can skip a re-read when nothing you mirror is in it.

MIRRORED_JOB_FIELDS = {"status", "title", "department", "locations", "accept_applications"}


def dispatch_job_update(session, store, event):
    changed = set(event["data"].get("previous_attributes") or {})
    # An edit that touched only fields we do not mirror needs no re-read. On a
    # busy workspace this is most of them.
    if changed and not (changed & MIRRORED_JOB_FIELDS):
        return
    handle = event["data"]["object"]["id"]
    store.upsert_job(session.get(f"{BASE}/jobs/{handle}", timeout=30).json())

When to use updated_after

Most list endpoints accept ?updated_after=. It is for recovery, not for your steady state:

• Your consumer was down for days and you would rather re-read than replay a long feed.
• You suspect drift and want to reconcile a window.
• You added a field to your schema and need to backfill it.

Webhooks, and why you still keep the feed

Webhooks push the same events with the same envelope, so one handler serves both. Take them as latency optimisation, not as a replacement.

Deliveries fail, endpoints get disabled after repeated failures, and a delivery that exhausts its attempts is abandoned. The ordered log is the source of truth and the only way to recover — so keep the drain loop, run it on a schedule as a backstop, and keep persisting the watermark from it.

Pacing

• limit=100. One request instead of four.
• One worker per credential. Parallel workers share the quota and make 429s look random.
• Honour Retry-After. On a personal-data quota it can be hours, and it is authoritative.
• Watch personal_data_reads_used_today during a backfill and stop deliberately. Walking into the wall mid-run leaves you guessing what got written.
• Log the request_id with your own run record.