What is machine learning?
Machine learning is a subfield of artificial intelligence, which is broadly defined as the capability of a machine to imitate intelligent human behavior.
ML workflow
The workflow below applies to any data analysis task.
Now, how does a ML-specific workflow look like?
1. Pre-processing: data split
We can split our data into training and test data sets:
Training set: used to develop feature sets, train our algorithms, tune hyperparameters, compare models, etc.
Test set: having chosen a final model, these data are used to estimate an unbiased assessment of the model’s performance, which we refer to as the generalization error.
1. Pre-processing: data split
The two most common ways of splitting data include simple random sampling and stratified sampling.
We want our train and test sets to be similar in data distribution, as seen on the left.
If they are not, we can use stratified sampling to ensure the same proportions of the predicted variable fall in both training and test sets.
This is important to avoid data shift!
Bias
Bias is the difference between the expected (or average) prediction of our model and the correct value which we are trying to predict.
It provides a sense of how well a model can conform to the underlying structure of the data.
Variance
Error due to variance is defined as the variability of a model prediction for a given data point.
Variance-bias trade-off
To minimize prediction error (on test set), need to create a model that balances the trade-off between variance and bias.
Notice how train set error always decrease as complexity increases. It is the test set error that presents the variance-bias trade-off (which is what counts for prediction).
Variance-bias trade-off
In other words, a model needs to be complex enough to capture the signal in the data, but not more complex to the point that it starts capturing noise.
What k value (hyperparameter) above do you think provides a good balance between bias and variance?
2. Training: data resampling
So, we need to optimize hyperparameters of a model so it balances the variance-bias trade-off, and is optimum for prediction.
Right now, this is how our data looks like:
We cannot touch the test set until the end. So how can we leverage the training set for hyperparameter optimizization? Using resampling methods.
2. Training: data resampling
Data resampling methods further split the training set into training and validation.
For each resample, models with given hyperparameter values are fit independently on the analysis set and evaluated on the assessment set.
2. Training: v-fold cross-validation
The most common cross-validation method is V-fold cross-validation. The data are randomly partitioned into V sets of roughly equal size (called the folds).
2. Training: v-fold cross-validation
For each resampling, 2 folds are used for training the model. and 1 fold is used to estimate model performance:
In practice, values of V are most often 5 or 10 (10 preferred because it is large enough for good results in most situations.).
2. Training: hyperp. optimiziation
There exist two general classes of hyperparameter optimization algorithms:
- Grid search
- Iterative search
2. Training: grid search
Predefined a set of parameter values to evaluate.
Inefficient since the number of grid points required to cover the parameter space can become unmanageable with the curse of dimensionality.
2. Training: iterative search
Sequentially discover new parameter combinations based on previous results.
In some cases, an initial set of results for one or more parameter combinations is required to start the optimization process.
3. Validation: predictive assessment
For continuous predicted variables, a predicted vs. observed plot and its metrics are commonly used:
3. Validation: predictive assessment
Models with greater agreement between predicted and observed (greater R2) and lower error metrics (e.g. RMSE, MAE) are better at predicting new observations.