Platform Overview

Data Flow

1. Scrapers run on Apify and emit a lean v3 job contract.
2. Webhooks ingest Apify datasets into Bordfeed.
3. The pipeline normalizes jobs, reconciles status changes, and stores source metadata.
4. Company and job enrichment adds fields used by feeds, search, and exports.
5. Organizations consume the data through dashboard views, feeds, webhooks, email digests, and the public REST API.

Scraping Boundary

Scraping work belongs in Apify. Local development should interact with Apify remotely for actor starts, run polling, logs, datasets, and validation. That keeps local iteration fast while letting Apify handle browsers, proxies, network retries, and runtime-specific dependencies.

Monitoring

Bordfeed tracks source status, webhook runs, job status, and closure reconciliation. Admin source pages are the operational view for registered actors, recent runs, failures, and manual retries.

Recommended Checks

bun run scraper:check
bun run scraper:list
bun run test
bun run typecheck
bun run test:e2e

For authenticated browser smoke tests, set E2E_SUPERADMIN_EMAIL and E2E_SUPERADMIN_PASSWORD in the shell that runs Playwright.