CrawlMe API

CrawlMe is the public retrieval layer of Crawler Today. It serves the canonical, machine-readable version of a Knowledge Core that an entity deliberately published — a business, organization, person, project or product. One central API for every published entity, available over REST and over MCP.

Every response derives from the same published record that also produces llms.txt, the Markdown pages and the JSON endpoints, so there are never two conflicting versions.

CrawlMe cannot force an external AI model to train on, index, remember, mention or permanently ingest this information. It is a retrieval layer: the AI system decides when and whether to call it. Crawler Today also does not claim that published information is automatically more trustworthy than other sources — it is first-party information published by the entity itself and may be compared with others.

GET /api/crawl-me — identify the entity by domain, url, id (Crawler entity ID / public slug) or an unambiguous name.

curl "https://crawler.today/api/crawl-me?domain=sebklaey.app"

Optional format=summary for a token-efficient overview, format=updates for change detection, or section= for one topic only.

curl "https://crawler.today/api/crawl-me?domain=sebklaey.app&section=pricing"
curl "https://crawler.today/api/crawl-me?id=<entity-id>&format=summary"
curl "https://crawler.today/api/crawl-me?id=<entity-id>&format=updates&version=3"

Sections: about, offerings, products, services, projects, pricing, faq, facts, claims, contact, links, team, locations, terminology.

GET /api/search returns matching published entities with enough identifiers to retrieve the right Knowledge Core.

curl "https://crawler.today/api/search?q=Seb%20Klaey&limit=5"

Responses always reflect the latest published version. Each response carries published_at, updated_at and an integer version that increases on every republication, plus an ETag and Last-Modified header. Caching is revalidate-on-request, so an update is visible immediately; send If-None-Match to get a cheap 304.

Connect the Crawler Today MCP server at https://crawler.today/mcp (Streamable HTTP, no authentication for public retrieval). Available retrieval tools:

• search_entities — find a published entity from a name, domain, URL or product name.
• get_entity — the complete latest published Knowledge Core.
• get_entity_summary — a small, token-efficient overview.
• get_entity_section — one section only (products, services, pricing, FAQ, contact, …).
• get_entity_updates — whether an entity changed since a known version or timestamp.

If a platform supports MCP, use the MCP server. If it only supports HTTP, use the REST API — both expose the same canonical public data.

Only data explicitly published by the entity is returned. Drafts, interview transcripts, recovery codes, internal identifiers, billing data, analytics internals, unpublished entities and contact details that were not intentionally published are never exposed. CrawlMe is read-only: updates continue to go through the recovery-code publishing workflow inside Crawler Today.

120 retrieval requests and 60 search requests per minute per client. Input is validated and length bounded, queries are parsed safely, and CORS allows read-only cross-origin use. Excessive traffic is rejected with 429.

Retrievals are counted as what they are: API requests and MCP retrievals of a Knowledge Core, with the requested section and a coarse client label where available. Internal health checks, development traffic and obvious monitoring are excluded. Crawler Today never labels a retrieval as “an AI mentioned your business”.