Screaming Frog, off the laptop and into the cloud.
Crawldock runs scheduled Screaming Frog crawls on ephemeral Google Cloud VMs and delivers the results to BigQuery, Cloud Storage and Sheets — with your machine switched off. It deploys into your own GCP project in about half an hour.
Every technical SEO team runs the same ritual. Someone kicks off a crawl of a large site on their laptop, and then that laptop is gone for the day — fans up, memory eaten, can’t sleep, can’t close the lid. Sometimes the crawl dies at hour six and nobody notices until the morning. And when it does finish, the output is a desktop file on one person’s machine: not queryable, not comparable to last month, not reportable without someone re-exporting it by hand.
Screaming Frog itself isn’t the problem — it’s the best tool there is for this, which is precisely why nobody wants to replace it. The problem is where it runs and where the data lands.
Crawldock is a self-hosted platform for running Screaming Frog crawls on Google Cloud. You define one-off crawls and scheduled projects in a local web UI, each run gets its own cloud VM sized to the job, and the results are delivered automatically to Cloud Storage, BigQuery and Google Sheets. The schedule lives in the cloud, so crawls fire whether or not the UI is running.
It deploys into a blank Google Cloud project with Terraform — make setup then make deploy — and the whole thing is designed around one number: a stranger with a GCP project and a Screaming Frog licence should reach a completed crawl, with data in BigQuery, in under thirty minutes following only the README. Every scope decision gets audited against that.
v0.1 is working. Since then it’s picked up in-app comparison reporting over the BigQuery data, per-run resource monitoring that right-sizes future crawls from measured CPU and memory rather than guesswork, self-service billed actuals, a static egress-IP pool for allow-listing, and crawl enrichment through the Screaming Frog API integrations. It isn’t released yet — I’m still working on it.
One run, one VM. When a crawl starts — on demand or on a schedule — the control plane sizes it, provisions a Cloud Batch VM for that run alone, and tears it down when the crawl finishes. Screaming Frog runs headless on it with a virtual display so JavaScript rendering still works. Nothing stays warm between runs, which is what keeps an unused installation close to free.
Define one-off crawls and scheduled projects
Sizes the run, writes the job, owns the activity log
Schedules fire in the cloud, with or without a UI running
Screaming Frog headless, sized S/M/L/XL per crawl
Provisioned for the run, torn down after it
GCS — the crawl file and every export
BigQuery — queryable tables and views
Google Sheets — native export · webhooks on completion
At Indago I built “Cloud Screamer” — a 24/7 automated site-auditing agent that did a version of this for our own clients. It worked, it was genuinely valuable, and it was completely locked inside one agency. Crawldock is that idea rebuilt properly and in the open: same problem, no proprietary lock-in, Apache 2.0, and designed for someone else’s Google Cloud project rather than mine.
Single-user is a licensing posture, not a product limitation. Screaming Frog licences are per individual person. Screaming Frog explicitly permits one licensed user to run multiple instances, including in the cloud — which is exactly what this does. But that permission doesn't stretch to sharing a licence between people, so the product is single-user by design and nothing in the docs or the interface may suggest otherwise. It's the constraint that shapes everything else: no accounts, no roles, no tenancy. Each person runs their own deployment with their own licence.
The binary is never redistributed. Screaming Frog enters the system in exactly one place: a runner image built inside your own Google Cloud project at deploy time, which downloads the official version-pinned Linux package, verifies its checksum, and pushes only to your own private registry. The publicly published base image contains everything except Screaming Frog. It's a two-stage build purely so that no image anyone can pull ever contains someone else's software.
No config builder. Ever. Crawl configuration is upload-only — you export a .seospiderconfig from the desktop app you already use and upload it. Rebuilding Screaming Frog's settings surface in a web UI would be months of work, permanently out of date the moment they ship a release, and worse than the thing it copies. The scope is closed on purpose: a fixed feature list, everything else either deferred or explicitly out of scope, so the project stays small enough to actually finish.
A failed crawl must never look like a successful one. A crawl that hit a robots.txt block, ran out of memory, or quietly stopped at 500 URLs will still produce a file and still load into BigQuery. That's the dangerous failure — the data looks fine and the conclusion is wrong. So runs carry a failure taxonomy and crawl-health diagnostics that flag thin, blocked or mis-sized crawls, and every run reaches a terminal state without anyone watching it.
Idle has to cost approximately nothing. The whole thing is scale-to-zero by default: no always-on database, no idle VM, no standing compute. A deployed installation nobody is using should generate a rounding error on a GCP bill, because the alternative is that people try it, forget it, get billed, and never come back. Runs carry per-run labels so the actual cost of every crawl comes back from the billing export, and there's a pre-run estimate before you commit to a big one.