Planning deliverable · 2026-08-28 · 1099:claude-app-vox1 · directive v53 Sources verified live this session (probe evidence inline). Plan only; PR1 code is delegated via the Cursor prompt in section F.
Generalize worker/scan.js into a source-adapter pipeline where SAM.gov becomes adapter #1 with byte-identical behavior, then add eVA (Virginia) and eMMA (Maryland) as adapters #2/#3 feeding the same opportunities table, change detection, attachment, extraction, and alert passes — shipped in the proven Grants 3-PR shape (PR1 zero-LLM ingest + schema, PR2 code-crosswalk matching + cheap-model enrichment, PR3 UI jurisdiction filter with free/Pro tiering). Both state sources are publicly viewable without auth, but neither offers a documented API for live solicitations, so both adapters are structured-scrape with mandatory breakage detection (parse-yield floor + per-source health rows) and hard failure isolation so a broken state adapter can never block the SAM pass. Set-asides map into the existing set_aside field with jurisdiction-prefixed values so SWaM-SDV (VA) and VSBE (MD) hit the same veteran matching that drives "Coming up."
| Source | Access method (verified) | Auth | Reliability / risk | Coverage | Docs fetchable |
|---|---|---|---|---|---|
VA — eVA VBO (mvendor.cgieva.com/Vendor/public/AllOpportunities.jsp) | Public HTML/XHR listing. Probe: plain curl → 403; browser User-Agent → 200 (17KB JS shell; data loads via XHR). UA-gated WAF, not IP-gated. | None to browse | Fragile scrape. WAF posture can change without notice; JS-rendered listing means the XHR endpoint must be captured in PR1 dev and pinned. Breakage detection mandatory. | State agencies + ~740 local bodies (IFB/RFP/QuickQuote). NIGP commodity codes. SWaM designations. | Attachment links on detail pages; fetch per-notice like SAM |
| VA — eVA Open Data (via data.virginia.gov CKAN) | CKAN API probed: package_search for eVA/solicitations → 0 relevant datasets. eVA's open-data page routes to historical PO/award exports only. | Portal login for bulk | Stable but wrong product: awards history, not live solicitations. | Historical POs/awards | n/a |
MD — eMMA public solicitations (emma.maryland.gov/page.aspx/en/rfp/request_browse_public) | Public ASP.NET (Ivalua) page. Probe: plain curl → 200, 86KB, no login. Server-rendered grid + Export control; paging via ASP.NET postback params. | None to browse | Medium-fragile scrape. Ivalua markup is stable release-to-release but robots.txt is Disallow: / — see Risk 1. | All MD state, county, schools, university public notices. UNSPSC categories. MBE / VSBE / SBR flags. | Attachment links on solicitation detail pages |
| MD — opendata.maryland.gov / BPW dashboard | Tyler/Socrata portal + Comptroller BPW dataset (CSV/XLSX export, quarterly refresh) | None | Stable | Approved awards only, quarterly lag — not live solicitations. Useful later for a state awards pass. | n/a |
Bottom line: for live solicitations there is exactly one viable feed per state, both scrape-class. Design accordingly.
Adapter interface (zero-dep, plain Node 20, lives in worker/adapters/):
adapter = {
source: 'sam.gov' | 'eva' | 'emma',
jurisdiction: 'federal' | 'state',
enabledEnv: 'EVA_SCAN_ENABLED' | ...,
fetchPass(ctx) -> { notices: NormalizedNotice[], raw_count, parse_yield }
}
NormalizedNotice = { source, jurisdiction, native_id, source_url, title, agency,
sub_agency, office, native_category[], set_aside[], notice_type, posted_date,
response_deadline, description, attachments: [{name,url}] }scan.js main loop iterates adapters; each pass wrapped in its own try/catch + timeout; one adapter's throw records a source_health failure row and continues. SAM adapter is a pure extraction of current code — no behavior change (PR1 acceptance: SAM pass output identical before/after).
Normalized schema (DDL sketch; full DDL in section F): extend opportunities rather than fork a table so change detection, attachments, extraction, and alerts reuse as-is:
source text not null default 'sam.gov', jurisdiction text not null default 'federal' check (jurisdiction in ('federal','state','local')), native_id text, source_url text, native_category text[](source, coalesce(native_id, notice_id)); SAM rows untouched.source_health (per-pass row: source, ran_at, fetched, upserted, changed, parse_yield, ok, error), code_crosswalk (system: nigp|unspsc, native_code, naics, confidence, method) — both added to the QC privacy-inventory lock list.Pipeline flow: adapter fetch → normalize → existing upsert-with-change-detection on TRACKED fields → attachment download to the existing bucket (state/<source>/<native_id>/…) → existing text extraction → existing alert pass now matching on naics ∪ crosswalk(native_category) and prefixed set_aside. Zero LLM calls in ingest — crosswalk in PR1 is static seed rows; enrichment of unmapped codes is PR2's cheap-model batch job, run offline, writing to code_crosswalk, never inline.
Set-aside mapping: store prefixed canonical values in the existing set_aside field: VA-SWAM-MICRO | VA-SWAM-SMALL | VA-SWAM-WOMEN | VA-SWAM-MINORITY | VA-SWAM-SDV, MD-MBE | MD-VSBE | MD-SBR. Veteran matching treats VA-SWAM-SDV and MD-VSBE as equivalent to federal SDVOSB.
PR1 — adapter framework + zero-LLM ingest (branch agent/cursor-fedm8/state-contracts) Scope: extract SAM into adapter #1 unchanged; add eVA + eMMA adapters behind env flags (default OFF); schema migration; source_health; static crosswalk seed (top ~40 NIGP + ~40 UNSPSC codes for the 12-NAICS pack, keyword-derived — no licensed NIGP dictionary redistribution). Files: worker/scan.js, worker/adapters/{sam.js,eva.js,emma.js,normalize.js}, worker/2026xxxx_state_sources.sql + mirror supabase/migrations/, qc/soc2-privacy-inventory.spec.cjs (add source_health, code_crosswalk). QC: adapter unit fixtures (saved HTML/XHR samples), SAM-unchanged regression, parse-yield floor test. Rollback: env flags OFF restores exact current behavior; migration is additive (new columns nullable/defaulted, new tables independent).
PR2 — matching + enrichment Scope: alert/profile matching consumes crosswalk; offline cheap-model job proposes mappings for unmapped native codes (writes code_crosswalk with method='llm', confidence); set-aside equivalence in match logic; "state opportunity" alert labeling. Rollback: crosswalk rows are data; matching change behind flag.
PR3 — UI + tiering Scope: jurisdiction filter on the existing search (not a separate tab — one search, per the goal), Federal/State badges on rows and alerts, free = state discovery + filters, Pro = state attachment/document analysis (same split as federal). Bundle rebuilt via scripts/make-bundle.sh + ?v= bump only. Rollback: filter flag off hides state rows in UI while ingest continues.
Each PR: gated lane — PR → verify on test.fedm8.com → chat approve → prod. One claim per PR.
Disallow: / while the page is public-by-design for vendors and MD DGS instructs the public to browse it. Ingesting is defensible but is a posture call: proceed with low-rate polling (1 pass / 12h, identified UA) or seek written OK from MD DGS first? Default if silent: proceed low-rate, identified UA FedM8Bot (contact@fedm8.com).jurisdiction='local' (free breadth for DMV vets) or state-only first? Default: ingest both, filter UI to state+local under one "Virginia/Maryland" label.(Self-contained; paste as-is.)
You are implementing PR1 of FedM8 state-contract ingest in the private repo jbnx/fedm8-scan. Branch: agent/cursor-fedm8/state-contracts. Open a PR to main; the repo is gated (PR → test.fedm8.com verify → chat approval → prod). Never auto-merge.
Repo conventions (absolute):
worker/scan.js is a long-running, zero-dependency Node 20 worker. No new npm deps. It talks to Postgres only via PostgREST over HTTPS using the secret key from env. No ORM, no client libs, no DDL at runtime.worker/*.sql mirrored byte-identically in supabase/migrations/. You write the files; a human applies them.qc/soc2-privacy-inventory.spec.cjs.assets/os-bundle.js by hand; no frontend changes in PR1.Env vars (add to .env.example names-only): STATE_SOURCES_ENABLED (default false — master kill), EVA_SCAN_ENABLED (false), EMMA_SCAN_ENABLED (false), STATE_SCAN_INTERVAL_MINUTES (720), STATE_HTTP_USER_AGENT (default FedM8Bot/1.0 (+https://fedm8.com; contact@fedm8.com)), STATE_HTTP_TIMEOUT_MS (30000), STATE_MAX_NOTICES_PER_PASS (2000).
Task 1 — adapter extraction (no behavior change). Create worker/adapters/sam.js exporting {source:'sam.gov', jurisdiction:'federal', fetchPass(ctx)} by moving the existing SAM fetch/normalize code; scan.js iterates an ADAPTERS array. Acceptance: with state flags off, a SAM pass produces identical upserts to current main (prove with the existing test fixtures).
Task 2 — normalized notice + isolation. worker/adapters/normalize.js defines NormalizedNotice as in the plan: {source, jurisdiction, native_id, source_url, title, agency, sub_agency, office, native_category[], set_aside[], notice_type, posted_date, response_deadline, description, attachments[]}. Main loop wraps each adapter in try/catch + STATE_HTTP_TIMEOUT_MS-bounded fetches; on throw, write a failed source_health row and continue to the next adapter. A state adapter must never delay or abort the SAM pass.
Task 3 — eVA adapter (worker/adapters/eva.js). Target the public VBO listing at https://mvendor.cgieva.com/Vendor/public/AllOpportunities.jsp. Known: plain UAs get 403; a browser-class or the identified STATE_HTTP_USER_AGENT gets 200; the listing hydrates via XHR — capture the XHR endpoint + params in DevTools and code against it (document the endpoint in a header comment). Parse: title, buying entity (→ agency), NIGP codes (→ native_category), SWaM designation strings (→ set_aside mapped to VA-SWAM-{MICRO|SMALL|WOMEN|MINORITY|SDV}), notice type, posted/close dates, detail URL (→ source_url), native solicitation id (→ native_id), attachment links from the detail page. Jurisdiction: state for Commonwealth entities, local otherwise if distinguishable, else state. Breakage detection: compute parse_yield = parsed/fetched; if fetched > 0 and yield < 0.5, mark the pass failed in source_health and upsert nothing.
Task 4 — eMMA adapter (worker/adapters/emma.js). Target https://emma.maryland.gov/page.aspx/en/rfp/request_browse_public (public, no login; Ivalua ASP.NET; paging via postback params — capture and pin them). Parse: solicitation code (→ native_id), title, issuing entity (→ agency), UNSPSC categories (→ native_category), MBE/VSBE/SBR flags (→ MD-MBE|MD-VSBE|MD-SBR), dates, detail URL, attachments from detail pages. Same parse-yield floor. Politeness for both adapters: sequential requests, ≥1s spacing, one pass per STATE_SCAN_INTERVAL_MINUTES.
Task 5 — migration. File worker/20260828_state_sources.sql + identical supabase/migrations/20260828_state_sources.sql:
alter table public.opportunities
add column if not exists source text not null default 'sam.gov',
add column if not exists jurisdiction text not null default 'federal',
add column if not exists native_id text,
add column if not exists source_url text,
add column if not exists native_category text[];
alter table public.opportunities
add constraint opportunities_jurisdiction_check
check (jurisdiction in ('federal','state','local')) not valid;
create unique index if not exists opportunities_source_native_uidx
on public.opportunities (source, coalesce(native_id, notice_id));
create index if not exists opportunities_jurisdiction_idx
on public.opportunities (jurisdiction) where jurisdiction <> 'federal';
create table if not exists public.source_health (
id bigint generated always as identity primary key,
source text not null, ran_at timestamptz not null default now(),
ok boolean not null, fetched int not null default 0,
parsed int not null default 0, upserted int not null default 0,
changed int not null default 0, parse_yield numeric,
error text, duration_ms int);
alter table public.source_health enable row level security;
create table if not exists public.code_crosswalk (
id bigint generated always as identity primary key,
system text not null check (system in ('nigp','unspsc')),
native_code text not null, naics text not null,
confidence numeric not null default 1.0,
method text not null default 'seed',
created_at timestamptz not null default now(),
unique (system, native_code, naics));
alter table public.code_crosswalk enable row level security;No grants to anon/authenticated in PR1 (worker uses the secret key). Seed ~40 NIGP and ~40 UNSPSC rows (method='seed') covering IT services, engineering, construction, facilities, admin/consulting — keyword-derived mappings to the NAICS pack (541511, 541512, 541519, 541611, 541330, 561210, 236220 et al). Do not embed or redistribute the NIGP dictionary.
Task 6 — plumbing + stubs. Upsert path sends the new columns; dedupe on (source, native_id) for state rows; attachments to state/<source>/<native_id>/ in the existing bucket; existing extraction/alert passes run unchanged over state rows. Add empty-interface stubs/TODOs that make PR2 (crosswalk-aware matching) and PR3 (jurisdiction filter) mechanical, clearly marked // PR2: / // PR3:.
Task 7 — QC. Add source_health and code_crosswalk to the privacy-inventory lock list. Add fixture-based unit specs per adapter (commit sanitized sample HTML/XHR JSON under qc/fixtures/state/), a SAM-unchanged regression spec, and a parse-yield-floor spec.
DO NOT: add npm deps to the worker · run DDL at runtime · call any LLM in ingest · touch assets/os-bundle.js or any frontend file · enable the state flags by default · fetch either portal in CI tests (fixtures only) · commit any credential or .env.
PR description must include: file list, the two migration paths, flag defaults (all OFF), and the note that a human applies migrations + env before test verification.
20260828_state_sources.sql to the live App DB and confirm mirror parity (worker/ vs supabase/migrations/).STATE_/EVA_/EMMA_* vars on the worker's App Host service (values above; flags ON only in test first).source_health rows, spot-check 5 upserted notices per source against the portals.source_health for 48h.