Frequently Asked Questions

Home » InterroBot Help

How to deal with an excess of site-internal norobots?

If you are encountering norobots while crawling the project domain with InterroBot... a workaround is to modify your site's robots.txt to globally allow InterroBot, internal to your site. As robots.txt is read top to bottom, it is best to add at the bottom of the file to reduce potential for other rules overriding the allow.

User-agent: interrobot
Allow: /

When is it okay to use the Personal Edition?

Personal Edition is intended for non-commercial use. The software trial will revert to Personal Edition once the trial has ended.

What's the difference between the editions?

Both InterroBot and the Microsoft Store Edition are commercially licensed. These products include one year of email support, and one year of licensed upgrades. At the end of the one year term, the version you have can be used as for long as it is works, but support will terminate.

Personal Edition is a non-commercial product, and is the default mode for the InterroBot program after the trial is expired. The functional difference of Personal Edition is that a handful of advanced features are shut off (outside of the trial period). These include:

  • JavaScript crawling
  • request header overrides (User-Agent, Authorization, X-Auth-Token)
  • user-defined norobots to ignore specific paths

What's are the limits of the crawler?

Part of using a desktop environment is that there are inherent hardware limitations. Performance and indexing capabilities will be proportional to the CPU and network capabilities of your computer. A quality solid-state drive (SSD) is key if you're thinking big.

A mid-tier laptop should be able to handle 100,000 total pages (across all projects). Add a high-performance desktop environment, that can be pushed much further.

Why is there no version 1.x?

Version 1.0 would have been the evolution of the 0.9x branch, which was built on the Windows UWP (native) platform. In 2020, it became apparent that UWP was no longer a priority at Microsoft, and development was shifted to CEF (Chrome Embedded Framework). This tech is known colloquially as an "electron app," though InterroBot does not use electron.

I found a bug?

Great! Let me know. There's a contact form in the client application, it's under the Options page (the gear icon).

InterroBot is a web crawler and devtool for Windows 10/11.
Want to learn more? Check out our help section or download the latest build.