Crawler
About ClimoBot
ClimoBot is the crawler behind Climo. It reads a small number of public pages on behalf of people who track their market.
How to recognise it
Every request carries this user agent:
ClimoBot/1.0 (+https://tryclimo.com/bot)
How it behaves
- It only requests public pages over HTTP or HTTPS. It never logs in, submits forms or runs checkouts.
- It reads and respects robots.txt, which it caches for up to 24 hours per domain.
- It makes at most one request per second per domain, and usually far fewer: most pages are checked once a day or once a week.
- It backs off when it receives 403 or 429 responses, and stops checking pages that keep failing.
- It downloads at most 2 MB per page and only HTML or text.
How to opt out
Add this to your robots.txt and ClimoBot will stop fetching your site within 24 hours:
User-agent: ClimoBotDisallow: /
You can also write to our contact form and we will block your domain on our side.