Risk modeling projectPersonal project · Public dataset2024
Credit-risk explorer
An interactive Shiny app for exploring default risk and understanding model predictions.
The problem
Default prediction is a cost-sensitive problem. A model that mostly recognizes the majority class can look accurate while failing at the decision it was built to support.
What I built
The project produced a complete trail from data cleaning and exploratory analysis through modeling, threshold selection, reporting, and an interactive prediction interface.
The result
Threshold selection improved the decision rule beyond an unquestioned 0.50 default.
In context
The UCI dataset is a benchmark sample, not evidence that the model is suitable for live underwriting.
Explore the technical details
Question & ownership
Can payment history and account behavior identify default risk without hiding minority-class errors behind a comfortable accuracy score?
I owned data preparation, exploratory analysis, stratified sampling, feature work, logistic and Random Forest experiments, threshold evaluation, reporting, and the Shiny prediction interface.
Data & validation
The public UCI dataset contains payment status, billing, repayment, credit-limit, and demographic variables for 30,000 clients. The target distribution is imbalanced and the inspected data contained no missing values.
- Rename and type the target consistently before analysis.
- Inspect target prevalence and preserve it through a stratified 70/30 split.
- Review numeric distributions, category encoding, correlation, and class imbalance.
- Use Weight of Evidence transformations where an interpretable risk relationship is useful.
Key decisions
Preserve the target distribution
A stratified split keeps the evaluation population comparable when the default class is less common.
A naive random split could shift prevalence and make model comparisons noisier.
Start with an interpretable benchmark
Credit decisions benefit from a model whose direction and feature effects can be explained before adding nonlinear complexity.
Random Forest was evaluated as a nonlinear comparison rather than assumed to be better.
Do not optimize headline accuracy alone
The no-information accuracy was higher than the tuned model's raw accuracy, so accuracy alone could not support a useful claim.
A single accuracy score would hide weaker behavior on the minority class.
Treat the threshold as a decision parameter
The classification threshold was evaluated explicitly instead of accepting the library default of 0.50.
The default threshold does not encode the relative cost of missed defaults and false alarms.
Evaluation
At the selected 0.42 threshold, the logistic model reached 82.4% accuracy and 76.0% balanced accuracy. The majority-class no-information rate was 86.9%, making balanced evidence essential to an honest interpretation.
- Threshold selection improved the decision rule beyond an unquestioned 0.50 default.
- Raw accuracy remained below the majority-class baseline, so it was not treated as proof of model quality.
- Balanced accuracy provided a more useful cross-class view, while still leaving room for cost-sensitive evaluation.
Limitations & next steps
- The UCI dataset is a benchmark sample, not evidence that the model is suitable for live underwriting.
- The evaluation does not yet include probability calibration, explicit business costs, or temporal validation.
- A production credit model would require fairness, stability, governance, and regulatory review.
- Evaluate PR-AUC, calibration, and cost-weighted threshold policies.
- Audit performance across relevant subgroups before any decision use.
- Validate on a later time period and monitor population drift.
Public dataset & source