Skip to content

feat(search): add Outcrawl as a web search provider - #1802

Open
heydryft wants to merge 1 commit into
tashfeenahmed:mainfrom
heydryft:feat/outcrawl-search
Open

heydryft wants to merge 1 commit into
tashfeenahmed:mainfrom
heydryft:feat/outcrawl-search

Conversation

@heydryft

@heydryft heydryft commented Nov 2, 2026

Copy link
Copy Markdown

What it adds

Outcrawl (https://outcrawl.ai) as a keyed web search engine, built the same way as Brave (#1650) and Firecrawl: an outcrawl search platform whose key lives in the normal key pool, plus an adapter in services/search.ts. Outcrawl is a search and scrape API for AI agents. Its free tier is 1,000 credits a month, no card needed, which fits the catalog's "free to start without a credit card" rule.

  • server/src/providers/outcrawl.ts: OutcrawlProvider (search only). Validation reads the account's credit balance (GET https://api.outcrawl.ai/v1/credits with the bearer key), which spends nothing, like the Firecrawl and Linkup probes. It goes through the shared probe, so 401/403 is a bad key, other 4xx still proves the key, and 5xx or a timeout is inconclusive.
  • outcrawl adapter in services/search.ts: sends POST /v1/search with { query, limit } and maps each hit's title, url and snippet to { title, url, content } through keepHits. Outcrawl returns hits already ranked and with no score, so keepHits scores them by position. It has no news vertical, so a news search runs as a web search (same as Keenable). Error handling needs nothing special: 401 benches the key, 402 (out of credits) takes the long credit cooldown, 429 the rate-limit cooldown, all through the existing coolDown.
  • outcrawl appended to SEARCH_PLATFORMS (shared, server, client), the Platform union, the keys route, key-parser (OUTCRAWL_ prefix and auth.json), SEARCH_PLATFORM_INFO, the Add key picker ("Outcrawl (1,000 credits/month)", linking to https://outcrawl.ai/console/keys) and the platform color map. Every list keeps its order, with Outcrawl after Brave.

Defaults

Opt-in only. No existing engine, order or default changes. Like every engine, it only runs once the catalog has a searchEngines row for it, so the row, its priority and its Premium/Free timing are your call. A row like { platform: 'outcrawl', engineId: 'outcrawl-search', displayName: 'Outcrawl Search', quotaLabel: '1,000 credits/month', features: { news: false, answer: false, fullContent: false } } matches what the adapter does.

What is sent to Outcrawl

The search query, limit, and the API key in the Authorization header. Nothing else.

How I tested

  • New tests, mirroring the Brave and Firecrawl ones:
    • providers/search-providers.test.ts: Outcrawl added to the shared validation cases (no-cost probe URL and method, 401 invalid, other 4xx authenticated, 5xx and timeout inconclusive, chat refused) and to the registry check.
    • services/search.test.ts: request URL, bearer header and body, hit mapping (entity decoding, hits without a URL dropped, title falling back to the URL, rank scores), a 401 benching the key, and a 402 parking it for the credit cooldown.
  • npx vitest run on search.test.ts, search-providers.test.ts, catalog-sync.test.ts and keys.test.ts: 201 passed.
  • Full server suite (npm run test -w server): 382 files passed, 4818 tests passed, 6 skipped. One timing test failed in that run (compression.test.ts, "stays within the 100 KB synchronous performance budget"); it passes when run on its own (21 of 21) and does not touch search.
  • Client: npx vitest run 436 passed (56 files), tsc -b clean, check:i18n passed for 60 locales.
  • npm run lint clean; tsc --noEmit (server) clean.
  • Live, through the new code path: an in-memory DB with one Outcrawl key and an outcrawl-search engine row, then OutcrawlProvider.validateKey(key) returned true and runSearch('open source LLM API gateway', { engine: 'outcrawl-search', maxResults: 5 }) returned provider: outcrawl, 5 results from llmgateway.io (2), github.com (2) and litellm.ai, each with a title and snippet.

Disclosure: I work on Outcrawl.

This PR was prepared with AI assistance (omp with Claude): the code, tests and this description were written with it, and the tests and the live call above were run as part of preparing it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant