Frequently Asked Questions
How to deal with an excess of site-internal norobots?
If you are encountering a lot of norobots while crawling with InterroBot... a workaround is to modify your site's robots.txt to globally allow InterroBot, internal to your site. As robots.txt is read top to bottom, it is best to add at the bottom of the file to reduce potential for other rules overriding the allow.
User-agent: interrobot
Allow: /
For sites beyond your control, there is no currently no way to ignore the robots.txt preferences, or to change the user-agent.
When is it okay to use the Personal Edition?
Personal Edition is intended for non-commercial use, commercial use within the context of a 45 day trial, or commercial use when licensed by the Windows Store (side-loaded alternative).
What's the difference between the two editions?
Currently, there is no functional difference. The Microsoft Store Edition is commercially licensed, and includes one year of email support, and one year of licensed upgrades. At the end of the one year term, the version you have can be used as for long as it is works, but support will terminate. At some point features may be restricted to the paid tier.
What's are the limits of the crawler?
There are no hard limits, only limits of the host machine. Most sites with less than 100,000 pages should be achievable. Storage is a consideration, however. The more pages, the more space you will need on your disk. A large site will utilize gigabytes of storage.
Why is there no version 1.x?
Version 1.0 would have been the evolution of the 0.9x branch, which was built on the Windows UWP (native) platform. Sometime around 2020, it became apparent that UWP, the platform, was no longer a priority at Microsoft, and development was shifted to CEF (Chrome Embedded Framework). This tech is known colloquially as an "electron app," though InterroBot does not use electron.
I found a bug?
Great! Let me know. There's a contact form in the client application, it's under the Options page (the gear icon).