API and Plugin Development

Home » Help »

InterroBot offers API and plugin development capabilities, allowing you to leverage InterroBot's best-in-class crawl filtering toward your own ends. This guide covers the basics of working with the InterroBot API, creating plugins, and accessing crawl data programmatically.

InterroBot API Overview

The API provides a web crawler and web scraping interface for developers, agents, and power users. You can use it to build custom reports, visualizations, and interactive apps from raw crawl data or query it directly from the command line.

Technical Sandbox

If you're just getting started, InterroBot provides a technical API sandbox in-app. You can get there from the Options page by navigating to the "Crawler Data API."

The sandbox is the best place to suss out plugin capabilities and data access. The sandbox provides a programmable web crawler interface where you can experiment with different queries and see live results.

The box above the JSON response is the query you can use to generate a predictable result from the form arguments, or as a quick-start from which to build your own queries.

Navigating to the API interface via Options, in-app.

Navigating to the API interface via Options, in-app.

API Usage

JavaScript API Access

JavaScript can access the data through the interrobot-plugin library. With the library, you can build web crawler plugins that extend InterroBot's functionality to suit your specific needs. Please refer to the repository for details on setting up a development environment.

Core API Methods

The core API methods are GetProjects, GetResources, and GetCrawls. These form the backbone of InterroBot's data API—between them, you can transform your website data into tables, visualizations, or interactive UIs. The report methods (GetLinkCheckReport, GetScraperReport, GetSpellCheckReport, GetCannibalizationReport, GetExporterReport) build on top of these, and are available via the CLI as well as in plugins.

Technical documentation for the plugin library is hosted on GitHub.

Example API Requests

Retrieve a list of all projects, with the icon image data

let result;  
result = await Plugin.postApiRequest("GetProjects",  {
    fields: ["image"],
    projects: []
});

Retrieve a list of pages with header data.

result = await Plugin.postApiRequest("GetResources", {
    external: true,
    fields: ["headers"],
    offset: 0,
    query: "headers: image",
    project: <projectId>,
    type: "any",
});

Retrieve a list of crawls by project.

const crawls = await Plugin.postApiRequest("GetCrawls", {
    complete: "any",
    fields: [],
    project: <projectId>,
});

To get a better feel for what data and filters are available, see the sandbox, and to get an idea of what is possible, check out the plugins directory full of plugin examples, many of which are open source.

The API/Plugin Connection

The API and plugin development are closely related—plugins are the primary way to build custom reports and visualizations on top of crawl data. That said, the API isn't limited to the plugin context. The same core methods and report suite are accessible directly from the CLI, making them available for scripting, automation, and agent workflows without writing a plugin.

For data access outside of both contexts, most pages within InterroBot provide exportable data in standard formats, and you can export the entire SQLite database if you need everything at once.

Development

The interrobot-plugin GitHub repo is the most active reference for plugin development details. Contact me through the app if you have any questions.

InterroBot is a web crawler and developer tool for Windows, macOS, Linux, iOS, and Android.
Want to learn more? Check out our help section or download the latest build.