Halide Yılmaz Tags
Metrics
3 posts
Metrics10 minAnnotator Agreement and Evaluator Reliability: Kappa, Alpha, and Validating an LLM Judge
Raw percentage agreement is not reliability. How Cohen kappa, Fleiss kappa and Krippendorff alpha correct for chance, how to read the thresholds, and how to validate an LLM judge the way you validate a human annotator.
Metrics9 minModel Calibration: What Does It Mean When Your Model Says "90 % Confident"?
An uncalibrated score makes threshold selection meaningless and silently invalidates every downstream cost-benefit decision. Reliability diagrams, expected calibration error, Platt scaling, isotonic and temperature scaling explained.
Metrics8 minClassification Metrics: When to Use Accuracy, Precision, Recall, F1, and ROC-AUC
Starting from the confusion matrix, we explain accuracy, precision, recall, F1, and ROC-AUC through everyday examples and clarify which metric is the right choice and when.