Technical Limitations

Home » InterroBot Help

Crawl Limits

Part of using a desktop environment is that there are inherent hardware limitations. Performance and indexing capabilities will be proportional to the CPU and network capabilities of your computer. A quality solid-state drive (SSD) is key if you're thinking big.

A mid-tier laptop should be able to handle, maybe 100,000 pages. Add a high-performance desktop environment, that can be pushed much further.

JavaScript Support

InterroBot crawler was designed with server-side generated HTML in mind. This is how 90% of the web is constructed—WordPress, Drupal, Django, Rails, they all render server-side out of the box.

InterroBot crawler does not load JavaScript. This means that SPA-type JavaScript frameworks are not a good match for the app. The JavaScript-loaded content and links will not be indexed, and such sites will not benefit from InterroBot.

The trade-off is crawler efficiency. Using HTML DOM is quite a bit faster, and less memory intensive than loading a page with JavaScript.

JavaScript rendering is on a maybe list of features for the future. Let me (the developer) know if this would be useful. There is a contact form, in-app, under Options.