Calibration Trong Credit Scoring: Khi AUC Đẹp Nhưng PD Không Đáng Tin

Điểm cần nhớ

• AUC/Gini/KS cho biết mô hình xếp hạng rủi ro có tốt không. Chúng không chứng minh rằng PD 8% thật sự tương ứng với tỷ lệ bad 8%.
• Khi PD (xác suất vỡ nợ) đi vào pricing, hạn mức hoặc ECL, calibration trở thành vấn đề quản trị mô hình, không phải bước làm đẹp kết quả.
• Không hiệu chỉnh xác suất trên tập validation đã từng dùng để chọn feature, chọn tham số hoặc dừng training sớm.
• Với boosting model, hãy bắt đầu bằng Platt scaling, một cách hiệu chỉnh xác suất đơn giản và ít overfit hơn khi dữ liệu chưa quá lớn.
• Luôn kiểm tra trên test/OOT cuối cùng bằng reliability diagram, tức biểu đồ so sánh PD dự báo với tỷ lệ bad thực tế.

Sơ đồ dưới đây tóm tắt ba pattern calibration hay gặp. Khi đọc hình, hãy nhìn vị trí đường cong so với đường chéo: nằm dưới nghĩa là model đang overconfident, nằm trên nghĩa là model đang underconfident.

Hình 1. Reliability diagram giúp kiểm tra thang PD có đáng tin không; AUC cao không đảm bảo đường calibration nằm gần đường chéo.

Ranking is not probability

AUC answers who is riskier than whom. Pricing, limits, provision and ECL need a different answer: how much risk is there?

A credit model can rank borrowers well and still give the wrong absolute probability. Once PD enters PD x LGD x EAD, probability scale errors become money errors, not just chart diagnostics.

A model is calibrated when, among all cases with predicted probability p, the fraction of positives is approximately p. Formally: P(Y=1 | f(X)=p) = p.

Discrimination metrics (AUC, Gini, KS) answer who is riskier than whom. Calibration metrics (Brier, ECE, reliability plots) answer whether the probability scale is honest.

When calibration matters

Skip calibration for pure ranking: model selection, approve/reject thresholds, SHAP.

Require calibration when PD enters formulas: risk-based pricing, credit limits, IFRS 9 ECL (PD × LGD × EAD), portfolio PD aggregation, or cross-model PD comparison. At that point calibration is governance, not cosmetic cleanup.

Practical methods and metrics

• Platt scaling (method='sigmoid', cv='prefit'): default for gradient boosting.
• Isotonic: more flexible; needs a large calibration set (many positives).
• Brier Score: primary scalar KPI for probability quality.
• ECE / MCE: bin-wise gaps; useful for monitoring and regulatory views.
• Reliability diagram: diagnostic; overconfident curves sit below the diagonal.

Reliability diagram — worked example

Static diagrams in the VI section cover: quantile binning, one point (0.10, 0.06), four-bin overconfident boosting, and raw vs Platt on the same test set. Use calibration_curve(..., strategy="quantile") for equal-frequency bins.

Implementation essentials I would check in review

1. Hold out a calibration set that was never used for training, early stopping, or feature selection.
2. Prefer out-of-time calibration when vintages allow.
3. Evaluate on a test set used only once at the end.
4. Deploy two outputs: raw_score for ranking; calibrated_pd for pricing and provisioning.
5. Monitor mean(PD) vs realized default rate after labels mature.

Tham khảo / References

• Platt, J. (1999). Probabilistic outputs for SVMs. Advances in large margin classifiers.
• Zadrozny, B., & Elkan, C. (2002). Transforming classifier scores into accurate probability estimates. KDD.
• Guo, C., et al. (2017). On calibration of modern neural networks. ICML.
• Niculescu-Mizil, A., & Caruana, R. (2005). Predicting good probabilities with supervised learning. ICML.
• scikit-learn: Probability calibration
• IFRS Foundation. IFRS 9 Financial Instruments (ECL framework).
• Basel Committee. International Convergence of Capital Measurement and Capital Standards (Basel II/III context for PD).
• European Parliament. Regulation (EU) No 575/2013 (CRR) — Article 179 (PD as long-run default rate, where IRB applies).