How a grade is made.

Changes in 0.2.1

No changes to the grading method or to any grade. Output-schema improvements for machine consumers (grades.json schema_version 2): machine-readable status ("graded" | "insufficient_data") per carrier-line with a constant field set; per-member below_credibility_floor flags on every aggregated underwriting company; member_companies is always an array; exposure_base_kind and exposure_base_year on every signal (the CA exposure-mix approximation is now machine-readable); coverage entries normalized to common exposure fields; the grade scale embedded under grade_scale; carriers keyed by slug with NAIC group codes. Carrier map v0.3.0 adds curated consumer-brand aliases (with provenance notes) and negative aliases for popular names that must not resolve to any rostered group.

Changes from 0.1.0: roster expanded from 12 to the top 25 US P&C groups (NAIC 2024 market share, all lines); New York DFS automobile complaint rankings added as a third source (auto line only); recency weights rebalanced accordingly; a premium-denominated credibility floor added for the NY signal.

Independence disclaimers

HonestPolicy Grades are calculated independently and without the endorsement of the NAIC (National Association of Insurance Commissioners).

HonestPolicy Grades are calculated independently and without the endorsement of the Texas Department of Insurance; the underlying complaint counts, policy counts, and complaint indexes are public records published by TDI on the Texas Open Data Portal.

HonestPolicy Grades are calculated independently and without the endorsement of the California Department of Insurance; the underlying justified-complaint counts, ratios, and exposure counts are public records published by CDI in its annual Consumer Complaint Study.

HonestPolicy Grades are calculated independently and without the endorsement of the New York State Department of Financial Services; the underlying upheld-complaint counts, premiums, and rankings are public records published by DFS on Open Data NY.

Data inputs

Deliberately excluded: AM Best and Demotech financial-strength ratings and J.D. Power study data (proprietary, license-restricted — not used in any form); NAIC Consumer Information Source complaint indexes (no clean programmatic access without a click-through agreement, so the PoC skips NAIC data entirely rather than scrape it).

Investigated and found unusable — Florida: neither the Florida Office of Insurance Regulation nor the DFS Division of Consumer Services publishes machine-readable company-level complaint statistics. FLOIR's public data tools carry residential market share (policy counts) but no complaints; DFS company-level complaint figures have only surfaced via records requests (e.g., press reporting, May 2025). New York was added instead.

Step 1 — Carrier groups

National insurance brands underwrite through many legal entities. The roster of graded carriers is the 25 largest US property/casualty groups by countrywide direct premium written, all lines combined, per the NAIC 2024 Market Share Reports for Property/Casualty Groups and Companies by State and Countrywide (June 2025) — used as editorial context only; no NAIC complaint data is retrieved. Group membership follows that report's company-to-group index. Several top-25 groups are commercial/specialty writers (Zurich, CNA, Fairfax, W.R. Berkley, American Financial, AXA XL, and largely AIG); they are kept on the roster and reported honestly as insufficient data rather than dropped, so the roster remains a mechanical, auditable list.

carrier_map.json maps each group to its member companies as they appear in each source: by NAIC company ID in the Texas and New York data and by printed company name in the California study. Every non-obvious membership carries a note (e.g., Safeco → Liberty Mutual, Foremost → Farmers, Esurance/National General → Allstate, Main Street America → American Family, Harleysville → Nationwide). Ambiguous entities are marked uncertain and are excluded from computation but disclosed in output — e.g., Consumers County Mutual (sells Travelers-branded policies while Travelers' own disclosures state it “is not a Travelers Company”), State National (Markel-owned but a fronting carrier for unaffiliated programs), and Adirondack Insurance Exchange (a NatGen-managed reciprocal that NAIC assigns to Allstate but that Allstate does not own or brand).

If a roster carrier writes no business in the covered states (e.g., Auto-Owners), its grade is reported as insufficient data, never guessed.

Step 2 — Relative complaint index per signal

Every (state, line, year) combination yields one signal, expressed on a common scale:

relative_index = (group complaints / market complaints)
              ÷ (group exposure base / market exposure base)

• Texas: exposure base = policies in force; complaints = TDI “confirmed complaints.” The market is every company reporting that line-year. This is TDI's own complaint-index formula applied at the carrier-group level.
• California: exposure base = approximate earned exposures; complaints = CDI “justified complaints.” The “market” is the ~50 largest writers CDI studies. Approximation: CDI publishes exposures only for the latest study year, so prior-year indices reuse the latest-year exposure mix for both group and market. This is disclosed per signal in the output.
• New York (auto only): exposure base = premiums written in NY ($M); complaints = DFS “upheld complaints” (complaints where DFS agreed the insurer made an inappropriate decision). The market is every automobile insurer DFS ranks that filing year. This is DFS's own ranking formula (upheld complaints per premium dollar) applied at the carrier-group level. Premium is a money-denominated exposure base, so NY indices are not exactly comparable to policy-count bases — a group with higher-than-average premium per policy is flattered slightly, and vice versa. NY carries the lowest weights partly for this reason.

relative_index = 1.00 means the group's share of complaints equals its share of the market's exposure; below 1.00 is better than average.

Credibility floor: a signal only counts if the group's exposure base is at least 10,000 policies/exposures (TX, CA) or $25 million of premiums written (NY) in that state-line-year.

Step 3 — Composite index

Signals are combined with a weighted geometric mean (indices are ratios, so geometric averaging is the natural choice). Zero-complaint indices are clamped to 0.05 before taking logs.

Base weights (renormalized over the signals a carrier actually has):

A letter grade requires at least two credible signals spanning at least two distinct years; otherwise the carrier-line is reported as insufficient data.

Step 4 — Score and letter grade

score = 100 / (1 + composite_index)        # 0-100, higher is better

A carrier's overall grade is the letter for the (unweighted) geometric mean of its per-line composite indices; the overall score is the mean of its line scores.

Provenance

Every signal in data/grades.json records: source name, publisher, URL, retrieval timestamp, year, line, the member companies aggregated (with their raw complaint and exposure figures), and the market totals used as the denominator. The published pages link each number back to this record.

Known limitations (v0.2.x)

1. Three states ≠ the nation. TX, CA, and NY are three of the four largest markets, but a national grade should add more state complaint datasets (and NAIC indexes if licensed access is arranged). Florida — the missing member of the top four — publishes no machine-readable complaint data (see Data inputs).
2. CA prior-year indices reuse latest-year exposure weights (see Step 2).
3. The CA “market average” is the average of the top-50 writers, not all writers.
4. Complaint counts are small for some carrier-lines; a future version should add credibility-weighted shrinkage toward 1.00 rather than a hard floor. This bites hardest for niche books that pass the exposure floor with near-zero complaints (e.g., Cincinnati auto, Markel's collector-car program): their composites are driven by the 0.05 zero-complaint clamp and should be read as “very few complaints observed,” not as a precise index.
5. Group membership is curated by hand and versioned; corporate M&A can silently change it. The pipeline flags every mapped entity that stops matching source data.
6. The NY exposure base is premium dollars, not policy counts (see Step 2), and NY covers the auto line only.
7. The TX and NY “Automobile” populations include commercial auto writers; for the personal-lines groups the effect is negligible, but grades for groups whose auto book is mostly commercial (e.g., Tokio Marine via Philadelphia Indemnity) mix commercial and personal complaint experience. Member-company tables on each carrier page disclose exactly which books are aggregated.