What are parameters and Hyperparameters in Machine Learning?

As a data science student, I am still learning many new things in the field.

As a new student back when I was still taking baby steps into machine learning, I came across two terms, terms that looked really similar, yet so different; Parameters and Hyperparameters. These terms are often interchangeably used but there is a significant difference between both.

Let us first check out what Parameters are.

Parameters are configuration variables that are set internally in a model. These values are generated from learned data. They are required mandatorily yet not often set by us. Its value determines the skill of the model but it cannot be manually defined.

Let us take an example of an NLP task. Here, the word frequency, the sentence length, noun or verb distribution per sentence, the number of a specific characters, lexical diversity, etc are all examples of parameters.

Model parameters differ for each experiment and depend on the type of data and task at hand. These are estimated based on the data during model training.

Difference between Parameters and Hyperparameters

Next, let us see what hyperparameters are.

Hyperparameters are those configuration variables that are/can be changed to fine-tune the model in order to improve the accuracy of the model. Traditionally speaking, hyperparameters in a machine learning model are the parameters that need to be specified by the user, in order to run the algorithm.

Hyperparameters may or may not be learned from the data. We can consider it more as an aspect of the algorithm. They are usually fixed before training the model.

Different models have different hyperparameters and let us go through some of them:

• Ridge and Lasso: Alpha
• Logistic Regression: Penalty, c, Solver
• KNN: n_neighbors
• SVM: C, Kernel, Epsilon (For SVR)
• NB: alpha
• DT: criterion, max_depth, min_samples_split, min_sample_leaf, max_features
• RF and ET: n_estimators, max_depth, criterion, max_samples_split, min_samples_leaf, max features
• XGBoost: n_estimators, max_depth, learning_rate, subsample, colsample_bytree
• Voting: estimators, voting
• Bagging: base_estimator, n_estimators
• AdaBoost: base_estimator, n_estimators, learning_rate
• Deep learning: number of hidden layers, units per layer, loss, optimizer, Activation, learning_rate, dropout rate, epochs, batch_size, early_stop_patience
• K means: n_clusters
• Hierarchical clustering: n_clusters, distance threshold
• DBScan: eps, min_samples
• Gaussian mixture: n_components
• PCA: n_components
• LDA: n_components.

In this post, we have seen the difference between parameters and hyperparameters. In short, parameters are calculated from data automatically while hyperparameters need to be manually set to help estimate parameters.