Version 2.10.0 Released
With InterroBot 2.10.0, the addition of the Custom List project option fundamentally changes how you can structure InterroBot projects. You can continue to use classic site and web-directory level crawling, of course, but now you can also set up projects with arbitrary lists of URLs, including wildcards (example.com/about-us/*). This is useful for a variety of setups, including seamlessly stitching together arbitrary web pages into a unified search for AI.
New crawler options have been added that will be helpful to anyone trying to reduce their crawl size to essentials. With External Pages and Assets, you can independently handle them as ignore, store as no-robots, or GET via HTTP. This way, no additional HTTP requests are made and minimal-to-no database storage is used for data you don't care about. These options can be manually enabled in crawler settings.
Additionally, the underlying runtime has been updated, providing marginal but predictable gains in memory usage across the board.
I hope you enjoy InterroBot 2.10. Pick up your copy today on the web or at your platform's app store.
InterroBot is a web crawler and developer tool for Windows, macOS, Linux, and Android.
Want to learn more? Check out our help section or
download the latest build.