# Textbook error 101 -- A low p-value for the full model does not mean that the model is a good predictor of the response

On page 606, of Lock et al “Statistics: Unlocking the Power of Data”, the authors state in item D “The p-value from the ANOVA table is 0.000 so the model as a whole is effective at predicting grade point average.” Ah no.

```
library(data.table)
library(mvtnorm)
rho <- 0.5
n <- 10^5
Sigma <- diag(2)
Sigma[1,2] <- Sigma[2,1] <- rho
X <- rmvnorm(n, mean=c(0,0), sigma=Sigma)
colnames(X) <- c("X1", "X2")
beta <- c(0.01, -0.02)
y <- X%*%beta + rnorm(n)
fit <- lm(y ~ X)
summary(fit)
```

```
##
## Call:
## lm(formula = y ~ X)
##
## Residuals:
## Min 1Q Median 3Q Max
## -4.4908 -0.6749 -0.0011 0.6736 4.2441
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -0.006924 0.003165 -2.188 0.0287 *
## XX1 0.016228 0.003642 4.456 8.35e-06 ***
## XX2 -0.019160 0.003641 -5.263 1.42e-07 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 1.001 on 99997 degrees of freedom
## Multiple R-squared: 0.0003215, Adjusted R-squared: 0.0003015
## F-statistic: 16.08 on 2 and 99997 DF, p-value: 1.04e-07
```

A p-value is not a measure of the predictive capacity of a model because the p-value is a function of 1) signal, 2) noise (unmodeled error), and 3) sample size while predictive capacity is a function of the signal:noise ratio. If the signal:noise ratio is tiny, the predictive capacity is small but the p-value can be tiny if the sample size is large.