is best, but feel free to get technical if absolutely needed. The more explanations I read, the better I will understand. Thanks!
In: Mathematics
One way to think about statistics is fitting a mathematical model to patterns observed in the data. The statistician must strike a balance between a model that is too simple and a model that is too complex. A too-simple model has no chance of actually capturing the data. A too-complex model can “overfit” the data by treating noise in the data as something real in the underlying model. An extreme example of overfit is a model that uses 10 parameters to fit 10 data points. This model can always fit the data perfectly just by saying “Everything that looks exactly like data point 1 has exactly the same value as data point 1. Everything that looks exactly like data point 2 has exactly the same value as data point 2…” Despite the “perfect” fit in this case, we haven’t learned anything about the underlying data process or, for example, what new data might look like.
The more complex a model is, the more “degrees of freedom” it uses. The more complex the data are, the more degrees of freedom they provide. A model using 10 parameters to fit 11 data points has only one degree of freedom left. A model using 10 parameters to fit 1000 data points has 990 degrees of freedom left. Statisticians often report the degrees of freedom remaining after they fit their model so others can decide whether they’re overfitting. In the case of some canonical statistical models (like linear regression), there are formal statistical tests for fit that use degrees of freedom as an input. The fewer degrees of freedom left after the model is fit, the stricter the test will be.
Latest Answers