Frequently Asked Questions

Could you explain the various editions, in a nutshell?

There are currently three flavors.

InterroBot is the free base application, available on every platform. It includes classic crawling, search, and all the reports. No purchase is required to get started, and the mobile versions no longer have a trial or paywall.

InterroBot Plus is a $10, one-time upgrade for the app store editions. It unlocks several additional crawler options:

request header overrides (User-Agent, Authorization, X-Auth-Token)
crawl exclusions by directory or regex

Plus is purchased inside the app, with billing handled by your store account. On the App Store, one universal purchase covers iPhone, iPad, and Mac.

InterroBot Pro is the full desktop product for Windows, macOS, and Linux. It includes everything in Plus, along with JavaScript crawling. A $100, single-user, perpetual license covers all three platforms and includes one year of support and upgrades.

There are still some platform differences to be mindful of. The CLI is included with the direct desktop downloads, even when used for free, but is unavailable in store packages. JavaScript crawling requires Pro and the full desktop builds. The Flathub version includes the free base application, without the CLI, Plus, or Pro features.

See purchasing options and downloads for the details.

What happened to Community Edition?

The free version is now simply called InterroBot. You no longer need to wait for a Desktop Edition trial to expire to use it for free. The direct desktop downloads still include a Pro trial, and you can continue using the base application after it ends.

It's a great tool to get started with website analysis without any upfront cost. You can upgrade anytime if you need the additional crawler options in Plus or JavaScript crawling in Pro.

I already paid for InterroBot. What changes for me?

You won't lose any features you paid for. Google Play and App Store customers keep the features they had in 2.13, and I'm offering those paid users a free upgrade path to Pro.

To claim your upgrade, use the in-app contact form, select "Billing questions or concerns," and send your Google Play or App Store purchase ID. Once I match it to the purchase records, I'll generate a Pro license for your email, totally free.

See the 2.14.0 release notes for more on the changes.

How to deal with an excess of site-internal norobots?

If you are encountering norobots while crawling the project domain with InterroBot, check whether they come from robots.txt or from directives in the pages themselves.

For robots.txt restrictions, a workaround is to modify your site's robots.txt to allow InterroBot:

User-agent: interrobot
Allow: /

Rule order does not determine priority. More specific disallow paths in rules that apply to InterroBot may need specific allow overrides. Details on robots.txt are available at Google Search Central.

For rel nofollow and META norobots directives, Plus and Pro include crawler options to override that behavior.

What are the limits of the crawler?

Part of using your own hardware, be it a desktop computer or a mobile device, is that there are inherent hardware limitations. Performance and indexing capabilities depend on your CPU, memory, storage, and network connection. A quality solid-state drive (SSD) is key if you're thinking big.

A mid-tier laptop should be able to handle 100,000 total pages across all projects. With a high-performance desktop, that can be pushed much further.

On mobile, your bottleneck will almost certainly be storage, with large sites consuming proportionally large amounts of disk. A mid-to-upper end phone should be capable of handling 10,000 pages without taking an inordinate amount of storage for an app. You can go further, just watch the storage.

I found a bug?

Great! Let me know. There's a contact form in the client application, under the Options page (the gear icon).

InterroBot is a web crawler and developer tool for Windows, macOS, Linux, iOS, and Android.
Want to learn more? Check out our help section or download the latest build.