Definition:Backtesting
🔍 Backtesting is a validation technique used across the insurance industry to assess how well a model, assumption set, or risk metric would have performed against actual historical outcomes. Insurers, reinsurers, and rating agencies apply backtesting to internal models for capital requirements, reserving methodologies, catastrophe models, pricing algorithms, and increasingly to machine learning tools used in underwriting and claims operations. The fundamental question backtesting answers is whether the model's predictions — be they loss distributions, reserve estimates, or risk scores — align with what actually transpired when the future became the past.
⚙️ The process typically involves feeding historical input data into the model as though it were being run prospectively, then comparing the model's output to known outcomes for the same period. In a value-at-risk context — central to Solvency II internal model approval in Europe and to enterprise risk management frameworks globally — an insurer might check whether actual losses exceeded the model's 99.5th percentile estimate no more often than statistically expected over a given lookback window. For catastrophe models, backtesting can involve replaying historical events such as Hurricane Andrew or Typhoon Jebi through the model to see if modeled losses approximate actual insured losses. Actuaries performing reserve backtests compare projected ultimate losses at successive evaluation dates to actual emergence, identifying systematic bias. Regulatory frameworks differ in how formally they mandate backtesting: EIOPA requires it as part of the internal model validation standards under Solvency II, while in the United States the NAIC's own risk and solvency assessment ( ORSA) guidance encourages but does not prescribe a uniform backtesting protocol.
💡 Effective backtesting serves as a critical check against overconfidence in models that may appear sophisticated but fail to capture real-world dynamics. A pricing model that consistently underestimates loss ratios in a particular line of business, for example, will produce backtesting failures that should prompt recalibration before the underpricing causes material financial harm. In the insurtech space, where firms deploy data-intensive predictive models at speed, backtesting disciplines are essential to demonstrating model governance to regulators and investor stakeholders alike. The technique also underpins confidence in capital models: if an insurer cannot show that its internal model would have produced reasonable results historically, supervisors are unlikely to approve its use for calculating regulatory capital.
Related concepts: