Error message: Too many collections
If you ever get the “Too Many Collections” error trying to create a new bulk job or crawl job, either via the API or the UI in your account’s dashboard, you’ve reached the limit our system supports: 1000 jobs.
This is why there is a timeout period after which inactive crawls will be removed – to make room for newer crawls.
If you have the need for more than 1000 active crawljobs, you have two options:
- Upgrade to a Plus or higher plan and we'll remove this limit from your account or
- Delete some jobs and try again
We recommend you keep an eye on your active crawls and delete them as they are no longer needed. If the data has become stale or you have downloaded it, you are encouraged to remove the crawl. Likewise, if you have already downloaded the data but would like to keep the metadata (crawl settings) around, you can download these settings by using the Crawlbot API. Such a backed up JSON file is enough to restore your crawljob’s settings even after years of inactivity on your account.