Usage
proscore(object, newdata = NULL, ...)
## Default S3 method:
proscore(
object,
newdata = NULL,
na.action = na.pass,
type = c("logs", "crps"),
aggregate = TRUE,
drop = FALSE,
...
)
Details
The function proscore provides a unified framework for scoring probabilistic forecasts (in-sample or out-of-sample). The following scores are currently available, using the following notation: \(Y\) is the predicted random variable with cumulative distribution function \(F(\cdot)\) and probability density function \(f(\cdot)\). The corresponding expectation and variance are denoted by \(E(Y)\) and \(V(Y)\). The actual observation is \(y\).
Log-score: Also known as logarithmic score. This is the negative log-likelihood where the negative sign has the effect that smaller values indicate a better fit.
-f(y)
Log-likelihood: Also known as log-density. Clearly, this is equivalent to the log-score above but using the conventional sign where bigger values indicate a better fit.
f(y)
Continuous ranked probability score (CRPS):
_{-}^{} ( F(x) - 1(x y) )^2 x
where \(1(\cdot)\) denotes the indicator function.
In case of a discrete rather than a continuous distribution, the ranked probability score (RPS) is defined analogously using the sum rather than the integral. In other words it is then the sum of the squared deviations between the predicted cumulative probabilities \(F(x)\) and the ideal step function for the actual observation \(y\).
Mean absolute error (MAE):
| y - E(Y) |
Mean squared error (MSE):
( y - E(Y) )^2
Dawid-Sebastiani score (DSS):
+ (V(Y))
Internally, the default proscore method first computes the fitted/predicted probability distribution object using prodist (corresponding to \(Y\) above) and then obtains the corresponding observation \(y\) using newresponse. Subsequently, the scores are evaluated using either the log_pdf method, crps method, or simply the mean. Finally, the resulting individual scores per observation can be returned as a full data frame, or aggregated (e.g., by using mean, sum, or summary, etc.).