Most teams that break a running collection pipeline during a rotation migration don’t break it by choosing the wrong provider or misconfiguring a pool. They break it by cutting over everything on a single afternoon, discovering a problem at 2 a.m., and having no clean way back to the setup that worked yesterday. The failure isn’t technical competence. It’s sequencing. A migration that keeps jobs alive is one that moves in stages, with each stage reversible on its own.
This guide walks through that phased approach: inventory, shadow testing, a graded rollout starting with the least important job, and a rollback plan attached to every step. Do them in order and you can abandon the whole effort at any point without losing a night’s data.
Take stock of what you have running today vs. what rotation will replace
Before you touch anything, write down what is actually running. Not what the architecture diagram says — what the cron table, the queue, and the logs say. For each job, record the target sites, the request volume, the current outbound address or static proxy it uses, any session or login state it depends on, and how you’d know within the hour if it stopped producing usable data.
That last column is the one people skip, and it’s the one that matters most. A rotation system changes the shape of your traffic, so the way a job fails changes too. A static setup tends to fail loudly and all at once when an address gets blocked. A rotation setup fails quietly and partially — some requests succeed, some come back with soft blocks or altered pages. If you don’t already have a per-job definition of “healthy,” define it now, while the old system is still the reference you can trust.
Sort the resulting list by stakes. The job whose output feeds a customer dashboard is not the job you experiment on first.
Run the new system in shadow mode alongside the old one before trusting it
Shadow mode means the rotation system does the work for real, but nobody depends on its output yet. Point a duplicate of a job at the new setup, let it run against the same targets on the same schedule, and store its results in a separate location. The old static job keeps running and keeps feeding production. You are now collecting two answers to the same question and comparing them.
Compare more than “did it return 200.” Check that the parsed fields match, that page structure is the same, that you’re not getting a lightweight decoy page instead of the real one. Watch how the rotation setup behaves across a full daily cycle, including the hours when target sites are most defensive. A shadow run that looks perfect for two hours and diverges overnight has told you exactly what you needed to know — and it cost you nothing, because production never saw it.
Migrate a low-stakes job first vs. flipping everything at once
Once one shadow job matches production reliably, promote it — but promote the least important job you have, not the biggest. The goal of the first live cutover is to catch the problems that only appear when something real depends on the output: alerting gaps, downstream jobs that assumed a certain latency, quota math that was fine in testing and wrong at full volume.
Let that one job live on rotation for several days before touching the next. Then move up your stakes-sorted list one job at a time. Flipping the entire fleet at once feels efficient and removes your ability to tell which change caused which symptom. A staged rollout keeps every failure attributable to a single move you can undo.
Keep a rollback path ready as each job moves across
Every job that crosses over should keep its old static configuration parked and runnable, not deleted, until the new setup has proven itself over a meaningful stretch. Treat rollback as a first-class feature: a documented, tested switch that a tired engineer can throw in one step, not a scramble to reconstruct last week’s config from memory.
Test the environment your jobs actually run in, too. If your collection runs on Headless Chrome instances managed by a provider based in Austin, confirm those instances route through the new rotation layer cleanly before you retire the static path — browser-based jobs surface session and fingerprint issues that raw HTTP requests never will. Keep the fallback for a week past the point you think you need it.
The moment to distrust is the moment everything looks fine and someone suggests deleting the old configs to tidy up. That cleanup is how a recoverable migration turns into a two-day outage.
