1;3409;0c Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers

Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers

18th International Conference on Machine Learning, 2001
Pages: 609-616

ICML

bibtex

Accurate, well-calibrated estimates of class membership probabilities are needed in many supervised learning applications, in particular when a cost-sensitive decision must be made about examples with example-dependent costs. This paper presents simple but successful methods for obtaining calibrated probability estimates from decision tree and naive Bayesian classifiers. Using the large and challenging KDD'98 contest dataset as a testbed, we report the results of a detailed experimental comparison of ten methods, according to four evaluation measures. We conclude that binning succeeds in significantly improving naive Bayesian probability estimates, while for improving decision tree probability estimates, we recommend smoothing by -estimation and a new variant of pruning that we call curtailment.