all docsdocs/plans/card-data-ingestion.md

Plan: Initial card data ingestion

Status: proposed · 2026-08-30 · builds on ../research/data-sources.md (full source detail + fit scores) and ../research/competitors.md.

Objective

Replace @frankie/mocks with verified, provenance-carrying data for the top ~100 cards (Phase 0), then stand up continuous monitoring for ~250 cards (Phase 1) — populating the same domain types the app already renders. No UI changes should be needed: that is the point of the repository seam (apps/web/src/lib/data).

Ground rules (from research)

  • No turnkey source exists. CFPB data lacks rewards/benefits; RewardsCC bans redistribution below enterprise tiers; affiliate feeds are restricted-use. The moat is our own pipeline over primary sources.
  • Issuer "Pricing & Terms" (Schumer box) pages are the core real-time source — public, standardized, factual (facts aren't copyrightable).
  • Keep affiliate-feed data segregated from the licensed database; scrape only logged-out public pages; retain raw evidence for every value.

Phase 0 — Seed the top 100 (target: ~2 weeks)

  1. Card universe list. Done — see ../research/initial-card-universe.md: 151 cards researched across the major issuers, 76 marked P0 for this phase, with the 2025–26 structural shifts (fee hikes, issuer moves, sunsets) that should become day-one card_events.
  2. Supabase schema. Create tables per docs/PRD.md §Data model (issuers, cards, card_terms versioned, signup_offers, card_rewards, card_benefits, transfer_partners, sources, data_evidence, card_events). Write the SupabaseDataSource adapter behind the existing interfaces.
  3. CFPB TCCP import. Semiannual XLSX (150+ issuers, APRs/fees) → import script → card_terms rows with sourceType: regulatory_filing, confidence: high. This is the free ground-truth baseline.
  4. Manual + LLM-assisted entry for rewards/benefits. For each card: fetch the issuer product page + pricing & terms page (store raw HTML/PDF in Supabase storage with timestamp), run LLM extraction to structured JSON matching @frankie/shared-types, human-review every field once, write data_evidence rows linking values → sources.
  5. Cross-validation. One month of RewardsCC BASIC ($29) as an oracle: diff our 100 cards against theirs; investigate mismatches. (Internal QA only — their terms forbid storing/redistributing below MEGA tier.)
  6. Cutover. NEXT_PUBLIC_DATA_SOURCE=supabase in staging; mock stays the dev/test fixture forever.

Exit criteria: 100 cards, ≥95% of headline fields (fee, APR range, FTF, bonus, category multipliers, marquee credits) evidence-linked and human-verified; app renders identically from Supabase.

Phase 1 — Monitor ~250 cards (months 1–3)

  1. Expand the universe to ~250 (long-tail co-brands, secured/subprime via FlexOffers-listed issuers).
  2. Change detection: changedetection.io (self-hosted) or Visualping on every tracked pricing/terms URL; on diff → refetch with Firecrawl/ Playwright → LLM re-extract → schema validation → compare with current values.
  3. Versioning: any changed field closes the old card_terms row, opens a new one, and emits a card_events row (old/new, direction, evidence) — the Changes feed becomes real.
  4. Anomaly gates: range checks (e.g. fee jumps >3× flag for review); human approves before publish. Extraction accuracy is a tracked metric.
  5. Offer signal: apply to Impact (Chase/Amex catalogs) + Bankrate partner feed for near-real-time bonus changes; use as alerting signal, store licensed data separately.

Phase 2+ (preview)

CFPB Credit Card Agreement Database (quarterly T&C archive since 2011) parsed for retroactive history; T&C red/green diffing (the prototype's best idea); multi-daily checks; API productization.

Effort & cost guess (Phase 0–1)

One engineer + LLM extraction ≈ tens of dollars of tokens per full sweep; RewardsCC $29/mo (validation); change-detection self-hosted ~$10/mo VPS; Supabase free tier suffices. The scarce resource is human review time — budget ~10 min/card for first verification.

Open questions (need founder input)

  1. Build the pipeline as scripts in this monorepo (apps/ingest or packages/pipeline) or a separate repo? (Lean: a packages/pipeline workspace of Node scripts + GitHub Actions cron.)
  2. Who reviews extractions day-to-day once monitoring runs?
  3. Affiliate applications require a consumer-facing property — apply now with the landing page, or wait?