Definition of Hyperparameter and it’s Various Types
Hyperparameters are external configuration settings that are not learned from the data but are set before the training process of a machine learning model begins. It is unlike model parameters, which are learned from the data. To determine the most effective hyperparameters for a machine learning model, practitioners often engage in a trial-and-error process.
Types of Hyperparameters:
The types of hyperparameters vary across different models and adapt based on the specific characteristics and requirements of the given dataset.
1. Learning Rate (Gradient Descent-based algorithms): Determines the step size during the optimization process.
2. Number of Trees (Random Forest, Gradient Boosting algorithms):
Specifies the number of decision trees to be used in ensemble, maximum depth of the trees, minimum number of samples required to perform a split node.
3. Regularization Parameters (Linear Models): Types of regularization (L1, L2) applied to prevent overfitting.
5. Kernel Parameters (Support Vector Machines): Specifies the type and parameters of the kernel function.
6. Number of Neighbors (K-Nearest Neighbors): Determines the number of neighbors (values of K) considered for classification or regression.
7. Hidden Layer Architecture (Neural Networks): Specifies the structure and number of hidden layers and neurons in a neural network.
8. Types of Activation Functions: Choosing the activation function (Relu, Sigmoid, Leaky Relu, tanh) depending on the requirements.
9. Number of Epochs: Selecting the number of epochs i.e. complete pass through the entire training dataset during the model training process.
10. Batch Size: the number of training examples processed in a single iteration.
11. Number of Clusters (K-Means Clustering): Specifies the desired number of clusters in the data.
Hyperparameter tuning involves systematically searching through different combinations of hyperparameter values to find the configuration that results in the best model performance. Techniques such as grid search, random search, and more advanced optimization methods are used for hyperparameter tuning.
