all docsdocs/research/data-sources.md

Frankie — Credit Card Product Data Sourcing Research

Date: 2026-08-30 Status: Research complete; recommendations at end. All claims verified against live sources unless explicitly marked [unverified]. Fit score legend: 1 = not useful, 5 = core building block for Frankie.

Executive Summary

There is no single commercial source that provides everything Frankie needs (fees + APRs + signup bonuses + multipliers + benefits + transfer partners, with provenance and history). The market splits into four buckets, and the winning strategy combines them:

  1. Regulatory/public data (free, authoritative, slow): CFPB's TCCP survey gives semiannual, machine-readable APR/fee data for 150+ issuers with a public data dictionary — the ideal seed and ground-truth backbone, but it has no signup bonuses or earn rates and is 3–6 months stale. The CFPB Credit Card Agreement Database (quarterly, ~600 issuers) provides full T&C documents for provenance/versioning.
  2. Commercial card-data APIs (cheap-ish, broad, shallow provenance): RewardsCC is the only verified turnkey "card attributes" API (~4,000 cards, manually maintained, $29–$499/mo, caching restricted to top tiers). Useful for bootstrapping and cross-checking, not as a resellable foundation.
  3. Affiliate networks (free, fresh, marketing-truth): Impact.com's Credit Card Product Catalogs, the Bankrate Credit Cards partner feed (XML/RSS/API), CJ, and FlexOffers expose issuer-approved offer data (bonus, annual fee, APR ranges) that updates when offers change — the best near-real-time signal for public offers, gated on affiliate approval and usable only per network terms.
  4. Own scraping + LLM extraction (the moat): Issuers publish Reg Z–mandated "pricing & terms"/Schumer box pages for every card. Monitoring ~250–1,000 URLs with change detection + LLM structured extraction is entirely feasible with commodity tools (changedetection.io/Visualping + Firecrawl/Playwright) and is the only path to field-level provenance, timestamps, and history — which is Frankie's actual product.

Recommended stack (short version): Seed from CFPB TCCP + the open-source credit-card-bonuses-api dataset (internal bootstrapping only) + a RewardsCC subscription for cross-validation → build own Schumer-box/product-page monitoring pipeline for the top ~250 cards → join affiliate feeds (Impact, Bankrate, FlexOffers) for offer-change signal and monetization later → treat Comperemedia/Curinos as competitors-slash-acquirers, not sources.

1. Commercial APIs & Datasets

1.1 RewardsCC — "Rewards Credit Card API" — Fit: 4 (bootstrap), 2 (long-term foundation)

  • URL: https://rewardscc.com / docs: https://rewardscc.com/docs / RapidAPI: https://rapidapi.com/rewardsccapi/api/rewards-credit-card-api
  • What it provides: REST API; card-level detail: network, card type, URL, card image, annual fee, foreign transaction fees, earn/spend rates, credit range required, signup bonus, travel benefits (free nights, baggage, TSA/Global Entry credits), annual spend bonuses, category multipliers, general benefits.
  • Coverage: claims "99.5% of US credit cards"; FAQ says ~4,000 cards. All data manually entered by hand from issuer sites (their claim) — no third-party sourcing.
  • Freshness: not published; missing cards added "within 24 hours" on request. No stated SLA on updates to existing cards — treat freshness claims skeptically.
  • Pricing/terms: BASIC/PRO/ULTRA via RapidAPI ($29+); MEGA $199/mo (200k calls), SUPREME $499/mo (1M calls); ENTERPRISE by contact (info@rewardscc.com). Caching/storage only allowed on MEGA+; no redistribution, no CSV export to users, delete-on-cancel. This directly conflicts with Frankie reselling the data — usable only for validation/bootstrap or under a negotiated enterprise license.
  • Skeptical note: solo/small operation; manual entry means no provenance trail and unknown error rates. Do not build the product on it.

1.2 Bankrate Credit Cards partner feed (Red Ventures; also owns CreditCards.com) — Fit: 4

  • URL: https://bankratecreditcards.com/ ; feed docs at https://feeds.bankratecreditcards.com/ ("Bankrate Credit Cards Feeds API Version 1.13.1" per search index — the docs host did not resolve publicly when tested; likely partner-gated).
  • What it provides: "complete integration of our credit card content, rate report data directly via an XML, RSS, or API feed" for approved affiliate partners; offers from Amex, Chase, Citi, Capital One, Discover, Barclays, etc.
  • Freshness: offer-driven — affiliate feeds update when issuer offers change (that's the network's business), so effectively near-real-time for public offers.
  • Access: free affiliate enrollment, but feed access is for "qualified partners"; expect approval friction and terms restricting use to displaying offers (not building a licensed database). Red Ventures also owns CreditCards.com — one partnership likely covers both brands' offer inventory. [partially unverified — exact feed fields unknown without partner login]

1.3 Impact.com Credit Card Product Catalogs — Fit: 4

  • URL: https://help.impact.com/partner/what-would-you-like-to-learn-about/platform-features/marketing-content/product-marketplace-and-catalogs/credit-card-product-catalogs-for-partners
  • What it provides: brands (issuers run their affiliate programs here — notably Chase and Amex historically) upload credit card product catalog feeds; partners get direct CSV download (UTF-8), FTP, or API access. Field list lives in Impact's "Product Feed Format Detail Spreadsheet" (partner-visible).
  • Freshness: "most up-to-date versions provided by the brand"; update notifications available. Offer terms in these feeds are issuer-approved — high accuracy for bonus/fee marketing terms.
  • Access: requires an approved partner relationship with each brand. Finance verticals have strict approval (traffic/compliance review); a pre-revenue B2B data company may struggle to get approved — a consumer-facing property helps.

1.4 CJ (Commission Junction) — Fit: 3

1.5 FlexOffers — Fit: 3

1.6 Sovrn Commerce — Fit: 1

1.7 CardRatings (QuinStreet) — Fit: 2

  • URL: https://www.cardratings.com/affiliate.html
  • What it provides: sub-affiliate program (embeddable comparison tables, widgets, CardMatch integration), $10–$100+/action payouts. No verified public data API — it's a monetization channel, not a data source. Widgets could theoretically be parsed but that violates the spirit of the partnership.

1.8 Even Financial / Engine by MoneyLion — Fit: 3 (different axis)

1.9 Issuer/network developer APIs — Fit: 1–2

  • Capital One Credit Offers API: https://github.com/capitalone/CreditOffers-API-reference-app — prequalified offer listings + a card products endpoint for affiliates. Archived July 2019, no longer maintained. Dead end unless revived via direct partnership.
  • Chase Credit Card Offers: https://developer.chase.com/solutions/chase-credit-card-offers/ — exists, but page content is gated/thin; appears to be a prescreened-offers partner product, not open product data. [unverified beyond existence]
  • Amex targeted offers API, Visa Offers Platform, Mastercard APIs: card-linked/prescreen infrastructure, not product-terms data. Not relevant to the database.

1.10 Consumer apps with big card databases (CardPointers, Kudos, MaxRewards) — Fit: 2

  • CardPointers (https://cardpointers.com/): claims 5,000+ cards from 900+ banks — the largest claimed card-attribute DB in the consumer space. No public API or licensing program found; conceivably licensable via direct outreach. [licensing unverified]
  • Kudos (https://www.joinkudos.com/): consumer recommendation app; data partners are Plaid/Spinwheel (account connectivity, not product terms). No licensing found.
  • These are future competitors for the consumer engine; their existence proves the DB is buildable by small teams.

1.11 Enterprise competitive-intelligence incumbents — Fit: 1 as source; important as market comps

1.12 Data marketplaces (Datarade etc.)

2. Public / Primary Sources

2.1 CFPB — Terms of Credit Card Plans (TCCP) Survey — Fit: 5 (as seed + ground truth)

2.2 CFPB — Credit Card Agreement Database — Fit: 4 (provenance/history layer)

  • URL: https://www.consumerfinance.gov/credit-cards/agreements/ ; archive: https://www.consumerfinance.gov/credit-cards/agreements/archive/
  • What: CARD Act–mandated quarterly collection of consumer credit card agreements from 600+ issuers: general T&Cs, pricing, fee schedules (not account-specific). Searchable by issuer + bulk quarterly archives (agreements are documents — PDF et al.; verify current packaging on the archive page). Gap: no 2015 collection.
  • Use for Frankie: the raw legal documents behind every card → LLM extraction gives fee/APR provenance with quarterly versioning back to 2011. Heavy parsing work (thousands of PDFs/quarter) but a unique historical asset almost nobody has structured.

2.3 Issuer product pages + Reg Z "Pricing & Terms" (Schumer box) disclosures — Fit: 5 (the core real-time source)

  • Every issuer must publish a standardized Schumer-box disclosure for each card application (Reg Z / Fair Credit and Charge Card Disclosure Act 1988). See https://en.wikipedia.org/wiki/Schumer_box and issuer conventions: Amex "Rates and Fees", BofA "Terms and Conditions", Capital One "View important rates and disclosures" (https://www.nerdwallet.com/article/credit-cards/how-to-read-schumer-box).
  • Machine-readability: HTML tables/PDFs, standardized content (mandated fields) but not standardized markup — ideal for template-per-issuer scraping + LLM extraction with schema validation. Product marketing pages (bonuses, multipliers, benefits) sit adjacent and are scrapeable in the same pass.
  • Freshness: real-time — the page is the offer. Monitoring these directly is the only true real-time primary source.
  • Licensing: public, unauthenticated pages; facts (rates/fees) are not copyrightable; but see §4 on ToS.

2.4 Federal Reserve / other federal data — Fit: 2 (context, not product data)

3. Scraping / Monitoring Infrastructure

3.1 Change detection

  • changedetection.io — open source, self-hostable, per-URL diffing with CSS/XPath filters + browser rendering; near-zero cost at 250–1,000 URLs. Best fit for owned infrastructure. (https://www.g2.com/products/changedetection-io/pricing — hosted tier also exists.)
  • Visualping (https://visualping.io/): SaaS, API + webhooks on every plan incl. free (5 monitors); Personal from $10/mo; AI-summarized changes. Good for quick start / non-engineering monitoring, gets pricey at scale.
  • Pattern: change-detect → trigger re-scrape+extraction only on diff. Keeps LLM costs proportional to actual market change, and every diff event is a provenance timestamp.

3.2 Scraping + extraction

  • Firecrawl (https://www.firecrawl.dev/): scrape→LLM-ready markdown; /extract endpoint does schema-driven structured extraction (LLM under the hood). Strong fit for Schumer-box pages: define a JSON schema (APRs, fees, bonus, multipliers) and validate outputs.
  • Apify (https://apify.com/): 40k+ actors, proxy/browser infra; no maintained credit-card-specific actor found — you'd write your own actors. Useful for anti-bot-hardened issuer sites (Akamai etc. on Chase/Amex — expect to need residential proxies and Playwright).
  • Practical stack: Playwright + proxy pool (or Firecrawl/Apify as managed layer) → per-issuer parsers for the ~10 big issuers (covers most of top 250 cards) → LLM extraction w/ JSON-schema validation + human review queue for low-confidence diffs.

3.3 Existing open-source projects (verified on GitHub)

  • andenacitelli/credit-card-bonuses-api — https://github.com/andenacitelli/credit-card-bonuses-api — signup-bonus data as JSON/CSV/YAML + OpenAPI spec; 562 commits, synced daily from a private monorepo (read-only mirror, powers offeroptimist.com). License: MIT + Commons Clause — non-commercial only; "do not resell as a substantial part of a commercial product." Great bootstrap/cross-check; cannot be resold. Bonus-focused: no fees/APR depth.
  • azxie/credit-card-scraper — https://github.com/azxie/credit-card-scraper — scrapes bonuses from Finder. Small/stale.
  • kaburkett/API-CreditCardRewardsScraper — https://github.com/kaburkett/API-CreditCardRewardsScraper — Python/Selenium rewards scraper. Proof-of-concept grade.
  • eric99ying/CreditCardScraper — https://github.com/eric99ying/CreditCardScraper — scrapes T&C + aggregator sites for APR/fees/bonuses. Proof-of-concept grade.
  • JinKim7/cc-scraper — https://github.com/JinKim7/cc-scraper — npm pkg for points/categories/cash-back calendars.
  • rsachoc/credit-card-bonuses — https://github.com/rsachoc/credit-card-bonuses — CSV of bonuses, updated "every few months".
  • USCreditCardGuide (org) — https://github.com/USCreditCardGuide — 5 repos of reference data (e.g., issuer phone numbers CSV); the site itself is a churning-community card encyclopedia; no structured full-terms dataset published.
  • Takeaway: nothing here is production-grade, but the bonuses-api proves a 1–2 person team can keep ~top-cards bonus data fresh daily. Community sources (r/churning wikis, Doctor of Credit) are excellent change alerting signals (humans notice offer changes fast) even if unlicensed as data.

4. Legal / Licensing Considerations (brief; not legal advice)

  • Scraping public pages: hiQ v. LinkedIn (9th Cir.) and Meta v. Bright Data (2024) support that scraping publicly accessible, logged-out pages isn't a CFAA violation — but hiQ still lost on breach of contract for scraping after accepting ToS, and settled with an injunction (https://www.morganlewis.com/blogs/sourcingatmorganlewis/2022/12/linkedin-v-hiq-landmark-data-scraping-suit-provides-guidance-to-data-scrapers-and-web-operators). Practical posture: scrape only logged-out public pages, never create accounts, respect robots.txt where feasible, keep volumes trivial (a few hundred pages/day is negligible), and don't scrape behind consent walls.
  • Copyright/database rights: facts (APRs, fees, bonus amounts) are not copyrightable (Feist); the US has no EU-style sui generis database right. Issuer marketing prose and card art are copyrighted — extract facts, don't republish text/images wholesale. Reg Z disclosures are mandated factual disclosures — the strongest possible position.
  • Affiliate feed terms: network/issuer program terms typically restrict feed data to promoting offers on approved properties — building a licensed B2B database from feed data likely breaches them. Use feeds as signal and cross-check; keep the licensed database built from primary sources. Getting terminated from Impact/CJ is a business risk, not a legal catastrophe — but issuer relationships matter long-term, so stay clean.
  • Commercial API terms: RewardsCC prohibits redistribution and (below MEGA) caching, with delete-on-cancel. Do not commingle its data into the persistent store without an enterprise agreement; use it in a segregated validation pipeline.
  • Provenance as defense: field-level provenance (URL + timestamp + snapshot hash) is not just the product feature — it's also the audit trail proving data came from public primary sources.

5. Recommended Phased Approach

Phase 0 (weeks 0–2): Schema + seed

  • Design the card-product schema versioned from day one (product identity ≠ offer identity; offers are time-bounded entities).
  • Ingest TCCP latest XLSX (Jul–Dec 2025) → seeds ~150 issuers' fee/APR data with regulatory provenance. Backfill historical TCCP for instant "APR history since 1990" content.
  • Bootstrap bonuses/multipliers for top cards from credit-card-bonuses-api (internal reference only — Commons Clause) + manual entry for the top ~100 cards. Manual entry is fine: ~100 cards × 30 fields is days, not months.

Phase 1 (months 1–3): Monitoring the top ~250 cards

  • Enumerate the top ~250 US consumer cards (≈15 issuers). For each: product page URL + pricing/terms (Schumer box) URL.
  • Stand up changedetection.io (self-hosted) on all URLs, daily checks; diffs trigger Firecrawl/Playwright scrape → LLM extraction against JSON schema → human review queue. Every accepted change writes a new version with provenance (URL, timestamp, raw snapshot).
  • Subscribe to RewardsCC MEGA ($199/mo) as an automated cross-validation oracle; discrepancies route to review. Keep its data segregated.
  • Add community signal listeners (Doctor of Credit, r/churning new-offer threads) as alerting only.

Phase 2 (months 3–6): Depth + history

  • Parse the CFPB Agreement Database quarterly archives for covered issuers → LLM-extract fee schedules → attach as document-level provenance and build the historical spine (2011→present).
  • Ingest each new TCCP release as a semiannual reconciliation audit of the whole DB.
  • Apply to affiliate programs (Impact, Bankrate/Red Ventures, FlexOffers) once a consumer-facing property exists; use catalogs as a third accuracy signal and future monetization.

Phase 3 (months 6–12): Real-time + long tail

  • Raise check frequency on volatile fields (bonuses) to multiple times daily for the top 100 cards; add A/B-offer detection (issuers vary bonuses by landing page/cookie — crawl clean sessions from multiple proxies).
  • Extend coverage down the long tail via FlexOffers (subprime/secured/fintech) + TCCP's small-issuer sample.
  • Expose Frankie's own versioned API; consider Even/Engine integration for the consumer recommendation engine's application rails.

What we would NOT do

  • Don't license from Comperemedia/Curinos (enterprise-priced, non-resellable, and they're the competition).
  • Don't build on RewardsCC as the foundation (no provenance, redistribution ban).
  • Don't scrape aggregators (NerdWallet/WalletHub/Bankrate pages) for data — legally messier than primary sources, and their data is derivative anyway. NerdWallet/WalletHub show no evidence of a data-licensing business [verified absence-of-evidence, not proof].

Appendix: Fit-score summary

SourceTypeFitWhy
Issuer pricing/terms pages (own pipeline)Primary scrape5Only real-time primary source; provenance = product
CFPB TCCP surveyPublic dataset5Free, semiannual, 150+ issuers, APR/fee ground truth
CFPB Agreement DatabasePublic documents4Quarterly T&Cs, 600+ issuers, historical spine
Impact.com card catalogsAffiliate feed4Issuer-approved offer data, CSV/FTP/API; approval-gated
Bankrate CC partner feedAffiliate feed4XML/RSS/API offers from major issuers; partner-gated
RewardsCC APICommercial API4→2Best turnkey breadth (~4k cards); no redistribution
FlexOffersAffiliate feed3Long-tail (secured/subprime/fintech) coverage
Even / Engine by MoneyLionMarketplace API3Future consumer-engine rails, not DB source
CJAffiliate feed3Product feeds; thinner major-issuer card presence
credit-card-bonuses-api (GitHub)Open source3Daily bonus data; non-commercial license
CardRatings (QuinStreet)Sub-affiliate2Monetization channel only; no data API found
CardPointers / KudosConsumer apps2Big DBs, no licensing program found
Fed G.19 / SEC filingsPublic data2Market context, not product terms
Comperemedia / CurinosEnterprise CI1Competitors/validators, not sources
Sovrn, Visa/MC/issuer dev APIsMisc1Not product-terms data (Cap One API archived 2019)