API rate limits and pagination: why automations fail at scale and how to fix them

A workflow that syncs ten test contacts perfectly can fall over the first time it meets a real import, a busy sale weekend or a full-history sync. The two usual causes are rate limits and pagination.

Rate limits: how fast you’re allowed to ask

APIs cap how many requests you can make in a period, to protect their service. Limits are usually per app or per account, measured per second, minute or day. Shopify, for example, documents different limit methods for its APIs and publishes the details on its API limits page.

When you go over, the API typically responds with HTTP 429 Too Many Requests. It may include a Retry-After header telling you how long to wait.

Signs you’re hitting limits

• Errors mentioning 429, “rate limit”, “too many requests” or “throttled”
• Workflows that fail only on busy days or big imports
• Some records processed, the rest missing

Fixes

1. Batch and wait. Process records in groups with a pause between them. In n8n, the Loop Over Items node splits items into batches, and the HTTP Request node has batching options. Make and Zapier have similar delay and batching tools.
2. Retry with a delay, respecting Retry-After when the API sends it. Retry on fail with a wait between attempts handles brief spikes.
3. Use bulk endpoints when available: one request that creates or updates many records.
4. Don’t fetch what you don’t need. Look up a record once and reuse it, instead of calling the API for every step.
5. Spread schedules. Don’t run five heavy workflows at the top of every hour.

Pagination: getting all the results

APIs rarely return thousands of records at once. They return a page (for example 50 or 250 items) plus a way to get the next one:

The silent failure

If your workflow only reads the first page, it doesn’t error. It just processes 250 of your 3,000 customers and reports success. Many pre-built steps handle pagination with a “return all” option. When you call an API directly, configure pagination (n8n’s HTTP Request node has a pagination setting) and check the count at the end.

Large syncs: a different approach

For the first full import, or a nightly sync of thousands of records:

• Use the vendor’s import tool or bulk API for the one-time load, then automations for new changes.
• Sync changes only: filter by “updated since” the last successful run.
• Use webhooks for ongoing changes, so you don’t poll everything. See webhooks vs APIs.
• Run off-peak and in chunks.

Make failures visible

• Log the number of records expected vs processed in each run.
• Alert when a run fails or is incomplete. See error handling in n8n.
• Keep failed records somewhere to retry, rather than losing them.

Estimate before you build

For each workflow, work out requests per run:

2,000 orders a day × 3 API calls each (look up customer, create order, update stock) = 6,000 calls a day.

Compare with the API’s published limits, and remember other tools may share the same limit. The planning sheet does this for each API.

Frequently asked questions

Does a paid plan raise the limits?

Sometimes. Some vendors offer higher limits on higher plans. Check their documentation rather than assuming.

Our automation platform says “success” but data is missing. Why?

Most often: pagination (only the first page processed), or a filter that silently dropped records. Compare counts.

Can we just add a long wait between every request?

It works, but slows everything down. Batching with short waits, plus retries for the occasional 429, is usually better.

If you’d like a workflow that handles real volumes reliably, that’s part of our tech stack service.