Optimizing Random Forest Models: A Deep Dive into Hyperparameter Tuning with Optuna

16 min readFeb 5, 2024

In this project, I’ll leverage Optuna for hyperparameter tuning optimization. Utilizing Seaborn’s life expectancy dataset, I aim to guide you through every stage of the Optuna model implementation. As we progress, I’ll explain each step to the best of my knowledge, ensuring a clear understanding of the optimization process.

To culminate this article, I’ll conduct a comprehensive comparison of the model performances, providing a detailed explanation of the key performance metrics, including Mean Absolute Error (MAE), Mean Squared Error (MSE), and R2 Score. This comparative analysis will offer valuable insights into the efficacy of each model and enable a nuanced understanding of their predictive capabilities.

What is Optuna?

Optuna is an open-source hyperparameter optimization framework designed for automating the process of tuning machine learning model hyperparameters. It provides a flexible and efficient platform for conducting hyperparameter search, which is essential for improving the performance of machine learning models.

Here are the key features and components of an Optuna model:

Study:

The central entity in Optuna is the “study”, which represents a single optimization run. A study is an iterative process where the hyperparameters are sampled, a model is trained, and the objective function is evaluated to find the optimal set of hyperparameters.

2. Objective Function:

The objective function is the function that Optuna aims to optimize. It takes a set of hyperparameters as input, and its output is the metric or score that is being optimized. Optuna seeks to minimize or maximize this objective function, depending on the specified optimization direction (e.g., minimize mean squared error).

3. Sampler:

The sampler is responsible for deciding the next set of hyperparameters to be evaluated in the study. Optuna provides various samplers, such as random search and Bayesian optimization, to explore the hyperparameter space efficiently.

4. Trials:

Each iteration in a study is called a “trial”. A trial represents a single evaluation of the objective function with a specific set of hyperparameters. The goal is to iteratively sample hyperparameters, evaluate the objective function, and update the search space to find the optimal configuration.

5. Parameter Suggestions:

Optuna provides methods to suggest hyperparameter values during each trial. For example:
suggest_int: Suggests integer values.
suggest_float: Suggests floating-point values.
suggest_categorical: Suggests values from a categorical set.

6. Direction:

The optimization direction specifies whether the objective function should be minimized or maximized. Common choices are ‘minimize’ or ‘maximize’.

7. Integration with Machine Learning Libraries:

Optuna can seamlessly integrate with popular machine learning libraries like scikit-learn, PyTorch, TensorFlow, and others. This allows for easy incorporation of hyperparameter tuning into machine learning workflows.

8. Visualization:

Optuna provides tools for visualizing the optimization process, including plots of hyperparameter distributions and optimization history plots. These visualizations help in understanding the search space exploration and convergence.

9. Parallel Execution:

Optuna supports parallel execution of trials, allowing for faster exploration of the hyperparameter space by evaluating multiple sets of hyperparameters concurrently.

Now, let’s transition to the preparation of our data before delving into the model development phase.

Python Library Imports

we first import the necessary libraries:

import pandas as pd
import seaborn as sns

from sklearn import metrics
from sklearn.model_selection import train_test_split, RandomizedSearchCV
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.ensemble import RandomForestRegressor

import optuna

Next, we employ the Seaborn library to load the life expectancy dataset and assign it to the variable ‘df’:

df = sns.load_dataset('healthexp')

To gain an initial understanding of our dataset, we utilize the `head()` function from the Pandas library, displaying the top 5 rows:

df.head()

Additionally, to provide a snapshot of the dataset’s dimensions, we leverage the `shape` method:

This combination of methods allows us to inspect the structure and size of the dataset efficiently.

Given the straightforward nature of our dataset, minimal data cleaning is necessary. Consequently, we can transition directly to the preparation of our model.

MODEL PREPARATION

In the initial phase of model preparation, our first step involves the creation of binary numbers for categorical columns. To achieve this, we leverage Pandas’ ‘get_dummies’ function:

df = pd.get_dummies(df)

Upon examination of our dataset, it becomes evident that the ‘get_dummies()’ function has generated additional columns for each categorical variable, assigning them binary labels of True or False. This transformation enhances the model’s ability to interpret and utilize categorical data effectively.

Moving forward, our next step involves the assignment of features to ‘X’ and the target variable to ‘y’. In this context, life expectancy serves as our independent variable and will be assigned to ‘X’.

Subsequently, we proceed to split the data into training and test sets, adjusting the ‘test_size’ parameter to 0.2, signifying that 20% of the entire dataset will be allocated to the test set. Additionally, we set ‘random_state’ to 54 for reproducibility:

X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.2 , random_state = 54)

In anticipation of running our models, a custom function named ‘modelresults’ has been implemented. This function is designed to print out three crucial metrics:

Mean Absolute Error
Mean Squared Error
The R2 Score.

The function enhances the interpretability and evaluation of model performance.

def modelresults(predictions):
    mae = mean_absolute_error(y_test, predictions)
    mse = mean_squared_error(y_test, predictions)
    r2 = r2_score(y_test, predictions)
    
    print('Mean absolute error on model is {:.4f}'.format(mae))
    print('')
    print('Mean squared error on model is {:.4f}'.format(mse))
    print('')
    print('The r2 score on model is {:.4f}'.format(r2))

Random Forest Regressor

To assess the effectiveness of our Optuna-tuned model in improving a Random Forest prediction, we first establish a baseline Random Forest Regressor. Setting the ‘random_state’ to 21 ensures reproducibility. We then train the Random Forest model and evaluate its performance using the ‘modelresults’ function:


rfr = RandomForestRegressor(random_state = 21)
rfr.fit(X_train, y_train)
y_pred_rfr_fit = rfr.predict(X_test)


y_pred_rfr_fit = rfr.predict(X_test)

Performance Metrics of Random Forest Model

The generated predictions, denoted as ‘y_pred_rfr_fit’, represent the model’s output on the test set. These results serve as our baseline for subsequent comparisons with Optuna-tuned models.

While the R2 score obtained from our initial Random Forest model is promising, for the sake of practice and optimization, we will now integrate the Optuna model. The goal is to enhance our results by fine-tuning the hyperparameters and evaluating the impact on model performance.

Optuna Study With 200 Trails

This is it the automated hyperparameter tuning model. Here’s the full function of my Optuna study, I will explain each line in details afterwards:

def objective(trial):
    n_estimators = trial.suggest_int('n_estimators', 100, 1000)
    max_depth = trial.suggest_int('max_depth', 10, 50)
    min_samples_split = trial.suggest_int('min_samples_split', 2, 32)
    min_samples_leaf = trial.suggest_int('min_samples_leaf', 1, 32)
    max_features = trial.suggest_categorical('max_features', ['sqrt', 'log2'])
    criterion = trial.suggest_categorical('criterion', ["squared_error", "absolute_error", "friedman_mse", "poisson"])

    model = RandomForestRegressor(
        n_estimators=n_estimators,
        max_depth=max_depth,
        min_samples_split=min_samples_split,
        min_samples_leaf=min_samples_leaf,
        max_features=max_features,
        criterion=criterion,
        random_state= 21
    )

    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)

    # metric  to optimize
    score = mean_squared_error(y_test, y_pred)
    
    return score

study = optuna.create_study(direction='minimize', sampler=optuna.samplers.RandomSampler(seed=42))
study.optimize(objective, n_trials=200)

# Print the best parameters found 
print("Best trial:")
trial = study.best_trial

print("Value: {:.4f}".format(trial.value))

print("Params: ")
for key, value in trial.params.items():
    print("    {}: {}".format(key, value))

Let’s break down our Optuna model’s `objective` function line by line:

def objective(trial):

1. Function Definition:

Defines the objective function named `objective` that takes a single argument `trial`. This function represents the metric to be optimized during the hyperparameter tuning process.

n_estimators = trial.suggest_int('n_estimators', 100, 1000)

2. Hyperparameter Suggestion — `n_estimators`:

Suggests an integer value for the hyperparameter ‘n_estimators’ in the range from 100 to 1000 (inclusive). This value will be used in the construction of the Random Forest model.

max_depth = trial.suggest_int('max_depth', 10, 50)

3. Hyperparameter Suggestion — `max_depth`:

Suggests an integer value for the hyperparameter ‘max_depth’ in the range from 10 to 50 (inclusive). This parameter represents the maximum depth of the individual trees in the Random Forest.

min_samples_split = trial.suggest_int('min_samples_split', 2, 32)

4. Hyperparameter Suggestion — `min_samples_split`:

Suggests an integer value for the hyperparameter ‘min_samples_split’ in the range from 2 to 32 (inclusive). It determines the minimum number of samples required to split an internal node in a tree.

min_samples_leaf = trial.suggest_int('min_samples_leaf', 1, 32)

5. Hyperparameter Suggestion — `min_samples_leaf`:

Suggests an integer value for the hyperparameter ‘min_samples_leaf’ in the range from 1 to 32 (inclusive). It sets the minimum number of samples required to be at a leaf node.

max_features = trial.suggest_categorical('max_features', ['sqrt', 'log2'])

6. Hyperparameter Suggestion — `max_features`:

Suggests a categorical value for the hyperparameter ‘max_features’, choosing either ‘sqrt’ or ‘log2’. This parameter controls the number of features to consider when looking for the best split.

criterion = trial.suggest_categorical('criterion', ["squared_error", "absolute_error", "friedman_mse", "poisson"])

7. Hyperparameter Suggestion — `criterion`:

Suggests a categorical value for the hyperparameter ‘criterion’, choosing one of [“squared_error”, “absolute_error”, “friedman_mse”, “poisson”]. This parameter specifies the function to measure the quality of a split.

model = RandomForestRegressor(
n_estimators=n_estimators,
max_depth=max_depth,
min_samples_split=min_samples_split,
min_samples_leaf=min_samples_leaf,
max_features=max_features,
criterion=criterion,
random_state= 21
)

8. Random Forest Model Initialization:

Initializes a `RandomForestRegressor` model with the hyperparameters suggested by Optuna, as well as a specified random state for reproducibility.

model.fit(X_train, y_train)
y_pred = model.predict(X_test)

9. Model Training and Prediction:

Fits the model to the training data (`X_train` and `y_train`) and predicts the target variable for the test data (`X_test`).

#metric to optimize
score = mean_squared_error(y_test, y_pred)

10. Metric Calculation:

Calculates the mean squared error between the true test labels (`y_test`) and the predicted values (`y_pred`). This metric will be optimized during the hyperparameter tuning process.

return score

11. Return Score:

Returns the calculated score, which Optuna will attempt to minimize since the direction is set to `’minimize’`.

study = optuna.create_study(direction='minimize', sampler=optuna.samplers.RandomSampler(seed=42))

12. Study Initialization:

Initializes an Optuna study for optimization with a random sampler and a specified random seed. The direction is set to minimize, indicating that the objective function should be minimized.

study.optimize(objective, n_trials=200)

13. Optimization Process:

Optimizes the `objective` function with 200 trials, attempting to find the set of hyperparameters that minimizes the mean squared error.

print("Best trial:")
trial = study.best_trial
print("Value: {:.4f}".format(trial.value))

14. Print Best Trial Information:

Prints information about the best trial found by Optuna, including the best objective value (mean squared error) achieved.

print("Params: ")
for key, value in trial.params.items():
print(" {}: {}".format(key, value))

15. Print Best Hyperparameters:

Prints the hyperparameters associated with the best trial found by Optuna. This includes the values of ‘n_estimators’, ‘max_depth’, ‘min_samples_split’, ‘min_samples_leaf’, ‘max_features’, and ‘criterion’.

Executing the Optuna model with 200 trials is a computationally intensive operation. With a system configuration of 16MB RAM and an SSD hard drive, the entire process took approximately 4 minutes to complete. The obtained results from this exhaustive hyperparameter tuning are as follows:

Optuna (200 Trails) Hyperparameter Suggestion

To ensure precision in parameter assignment for our next Random Forest model and to avoid the risk of inadvertent errors, we capture the best parameters determined by the Optuna study in a variable named ‘best_params’:

best_params = study.best_params

However, before proceeding to build the next Random Forest model with these optimized parameters, let’s dive into the visualization of Optuna graphs. These visualizations will provide insights into the optimization process, allowing us to better comprehend the evolution of hyperparameter tuning throughout the study.

Optuna Visualizations

Optimization History Plot

The Optimization History Plot visualizes the evolution of hyperparameter search, offering insights into the effectiveness and efficiency of the optimization process. This graphical representation aids in assessing whether the objective function improves over trials or quickly converges to an optimal solution. By analysing the plot, decisions can be made regarding the need for additional trials or whether a satisfactory result has been achieved.

The graphical analysis indicates that our model attained its optimal value by the 107th trial, with subsequent trials maintaining the same objective values. In summary, this plot serves as a visual summary of the hyperparameter optimization study, showcasing how the objective function evolves with each trial and providing valuable information about the optimization process’s efficiency and effectiveness.

Parallel Coordinate Plot

Parallel Coordinate Plot is used to generate a parallel coordinates plot, which is a type of visualization that helps to understand the relationships between hyperparameters and their corresponding objective function values in a hyperparameter optimization study.

this is a useful tool for gaining insights into the relationships between hyperparameters and their impact on the objective function during an optimization study. It helps us to identify promising hyperparameter configurations and understand the trade-offs and interactions between different hyperparameters.This can aid in making decisions on which hyperparameter settings to prioritize for further optimization or model tuning.

optuna.visualization.plot_parallel_coordinate(study)

The plot illustrates serval important aspects:

1) Hyperparameter Relationships: Each vertical axis represents a hyperparameter, and the horizontal lines connecting different axes illustrate how the values of these hyperparameters relate to each other during the optimization process. You can see how certain hyperparameter values are chosen together or in relation to one another.

2. Objective Function Values: The color and thickness of each line segment in the plot represent the objective function value for a specific trial or combination of hyperparameters. Darker, thicker lines often indicate better objective function values, while lighter, thinner lines correspond to poorer results.

3. Optimal Configurations: By visually inspecting the plot, we can identify regions of the plot where the lines converge, indicating successful configurations of hyperparameters that led to good objective function values. This helps you find optimal or promising combinations of hyperparameters.

4. Divergence and Exploration: The spread of lines across the plot can also indicate how widely you explored the hyperparameter search space. Tight clusters suggest that the search was focused, while scattered lines may indicate that the search space was thoroughly explored.

Slice Plot

These Plots provide insights into the relationship between specific hyperparameters and the objective function value. It visualizes how a particular hyperparameter or a combination of hyperparameters affects the optimization process.

optuna.visualization.plot_slice(study, params = ['n_estimators',
'max_depth',
'min_samples_split',
'min_samples_leaf',
'max_features',
'criterion',])

Hyperparameter Importance

This plot can provide valuable insights into which hyperparameters had the most significant impact on the objective function and the overall model performance.

optuna.visualization.plot_param_importances(study)

Assigning the Best Hyperparameters

Now that we have our best hyperparameters figured with the help of Optuna model, we assign these parameters as best parameters and insert them into our random forest model and run the whole process again. To ensure accuracy, we use the previous variable ‘best_params’ to each of the selected parameters as illustrated below:

best_n_estimators = best_params['n_estimators']
best_max_depth = best_params['max_depth']
best_min_samples_split = best_params['min_samples_split']
best_min_samples_leaf = best_params['min_samples_leaf']
best_max_features = best_params['max_features']
best_criterion = best_params['criterion']

Random Forest Model with The Best Hyperparameters

As shown below, we assign our RandomForestRegressor with its best parameters to a new variable called ‘best_model’ and run our model. Afterwards, we run our customised function ‘modelresults’ to get our metrics scores and compare them with the previous random forest model.

best_model = RandomForestRegressor( n_estimators = best_n_estimators,
                                  max_depth = best_max_depth,
                                  min_samples_split = best_min_samples_split,
                                  min_samples_leaf =  best_min_samples_leaf,
                                  )

best_model.fit(X_train, y_train)

y_pred_rfr_optuna_200_trails = best_model.predict(X_test)

modelresults(y_pred_rfr_optuna_200_trails)

Random Forest with Optuna Performance Metrics

In order to make our comparison easier to analysis, we make a new table with the help of the following function and print out the results:

def modelresults(predictions, model_name):
    mae = mean_absolute_error(y_test, predictions)
    mse = mean_squared_error(y_test, predictions)
    r2 = r2_score(y_test, predictions)

    results_dict = {
        'Model': model_name,
        'MAE': mae,
        'MSE': mse,
        'R2 Score': r2
    }
    
    print('Results for {}:'.format(model_name))
    print('Mean absolute error on model is {:.4f}'.format(mae))
    print('Mean squared error on model is {:.4f}'.format(mse))
    print('The r2 score on model is {:.4f}'.format(r2))
    
    return results_dict


results_fit = modelresults(y_pred_rfr_fit, 'Random Forest Fit')
results_optuna_200 = modelresults(y_pred_rfr_optuna_200_trails, 'Random Forest Optuna (200 trials)')

results_df = pd.DataFrame([results_fit, results_optuna_200])
print(results_df)

while the differences in metrics are subtle, the baseline Random Forest Fit model exhibits slightly superior performance in terms of MAE, MSE, and R2 Score compared to the Random Forest Optuna model with 200 trials.

How To Get More Accurate Hyperparameter Results?

To potentially get more accurate hyperparameter results, you can consider a few strategies:

1. Increase the Number of Trials (`n_trials`):

Increasing the number of trials allows Optuna to explore the hyperparameter space more thoroughly, potentially finding better configurations. However, keep in mind that a larger number of trials will also increase the computational cost.

study.optimize(objective, n_trials=1000)

2. Adjust the Hyperparameter Ranges:

Expand or refine the ranges of the hyperparameters to explore a broader or more focused search space. This adjustment can help find more optimal configurations.

n_estimators = trial.suggest_int('n_estimators', 100, 1500)
max_depth = trial.suggest_int('max_depth', 5, 100)
min_samples_split = trial.suggest_int('min_samples_split', 2, 64)
min_samples_leaf = trial.suggest_int('min_samples_leaf', 1, 64)

3. Use a Different Sampler or Search Algorithm:

Optuna provides different samplers and algorithms for exploring the hyperparameter space. Trying a different sampler or search algorithm might lead to better results. For example, TPESampler is often a good starting point. It performs well in a variety of scenarios and is efficient for both discrete and continuous hyperparameter spaces.

study = optuna.create_study(direction='minimize',
sampler= optuna.samplers.TPESampler(seed=42)

4. Increase the Number of Dimensions Sampled in Each Trial:

Optuna samples hyperparameters sequentially by default. You can change this behavior by increasing the number of dimensions sampled in each trial using `n_jobs` parameter. This parallelizes the sampling process.

study.optimize(objective, n_trials=200, n_jobs=2)

For purposes of our experiment lets create a new Optuna model study with more trials.

Optuna Study With 1000 Trails

In this experiment, we conduct an Optuna study with an increased number of trials, specifically setting the trials to 1000. The study utilizes the same hyperparameter optimization approach as the previous one but explores a more extensive search space by increasing the number of trials. This allows us to observe how the model’s performance evolves with a larger set of hyperparameter configurations.

def objective(trial):
    n_estimators = trial.suggest_int('n_estimators', 100, 1000)
    max_depth = trial.suggest_int('max_depth', 10, 50)
    min_samples_split = trial.suggest_int('min_samples_split', 2, 32)
    min_samples_leaf = trial.suggest_int('min_samples_leaf', 1, 32)
    max_features = trial.suggest_categorical('max_features', ['sqrt', 'log2'])
    criterion = trial.suggest_categorical('criterion', ["squared_error", "absolute_error", "friedman_mse", "poisson"])

    model = RandomForestRegressor(
        n_estimators=n_estimators,
        max_depth=max_depth,
        min_samples_split=min_samples_split,
        min_samples_leaf=min_samples_leaf,
        max_features=max_features,
        criterion=criterion,
        random_state=21  )

    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)

    # metric to optimize
    score = mean_squared_error(y_test, y_pred)
    
    return score

study = optuna.create_study(direction='minimize', sampler=optuna.samplers.RandomSampler(seed=42))
study.optimize(objective, n_trials=1000)

# Print the best parameters found 
print("Best trial:")
trial = study.best_trial

print("Value: {:.4f}".format(trial.value))

print("Params: ")
for key, value in trial.params.items():
    print("    {}: {}".format(key, value))

our study outputs the following:

Following the Optuna study with 1000 trials, we proceed to assign the best parameters for our new Random Forest model, employing the same methodology as in the previous study. The optimized model’s predictions are then stored in the variable ‘y_pred_rfr_optuna_1000_trials’. This enables us to evaluate and compare the model’s performance metrics against the baseline Random Forest Fit model.

best_model = RandomForestRegressor( n_estimators = best_n_estimators,
                                  max_depth = best_max_depth,
                                  min_samples_split = best_min_samples_split,
                                  min_samples_leaf =  best_min_samples_leaf,
                                  )
best_model.fit(X_train, y_train)

y_pred_rfr_optuna_1000_trails = best_model.predict(X_test)

modelresults(y_pred_rfr_optuna_1000_trails)

Random Forest with Optuna (1000 Trails) Performance Metrics

Optuna Study with adjusted hyperparameter tuning

In the final phase of our experimentation, we initiate a new Optuna study, incorporating adjusted hyperparameter tuning strategies. This study introduces modifications such as altering the number of dimensions sampled in each trial using the n_jobs parameter and adopting a different sampler, namely the ‘TPESampler’. These adjustments aim to further enhance the optimization process and explore alternative configurations to potentially improve model performance.

def objective(trial):
    n_estimators = trial.suggest_int('n_estimators', 100, 1500)
    max_depth = trial.suggest_int('max_depth', 5, 100)
    min_samples_split = trial.suggest_int('min_samples_split', 2, 64)
    min_samples_leaf = trial.suggest_int('min_samples_leaf', 1, 64)
    max_features = trial.suggest_categorical('max_features', ['sqrt', 'log2'])
    criterion = trial.suggest_categorical('criterion', ["squared_error", "absolute_error", "friedman_mse", "poisson"])

    model = RandomForestRegressor(
        n_estimators=n_estimators,
        max_depth=max_depth,
        min_samples_split=min_samples_split,
        min_samples_leaf=min_samples_leaf,
        max_features=max_features,
        criterion=criterion,
        random_state= 21
    )

    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)

    # choose metric  to optimize
    score = mean_squared_error(y_test, y_pred)
    
    return score

study = optuna.create_study(direction='minimize', sampler=optuna.samplers.TPESampler(seed=42))
study.optimize(objective, n_trials=1000, n_jobs = 2) # 2 parallel jobs

# Print the best parameters found by Optuna
print("Best trial:")
trial = study.best_trial

print("Value: {:.4f}".format(trial.value))

print("Params: ")
for key, value in trial.params.items():
    print("    {}: {}".format(key, value))

The execution of this study was computationally intensive, requiring a substantial amount of processing time to complete. The entire process took approximately 38 minutes, reflecting the complexity of the hyperparameter tuning and optimization involved. The resulting model performance metrics, as evaluated by our ‘modelresults’ function, are as follows:

Random Forest with Optuna (adjusted hyperparameter tuning) Performance Metrics

In the final step of our analysis, we utilize the ‘modelresults’ function to compare the performance metrics of all four models. This includes the baseline Random Forest Fit model, the Optuna study with 200 trials, the Optuna study with 1000 trials, and the Optuna study with adjusted hyperparameter tuning. By integrating the results of these studies, we gain a comprehensive overview of each model’s performance across various hyperparameter configurations.

def modelresults(predictions, model_name):
    mae = mean_absolute_error(y_test, predictions)
    mse = mean_squared_error(y_test, predictions)
    r2 = r2_score(y_test, predictions)

    results_dict = {
        'Model': model_name,
        'MAE': mae,
        'MSE': mse,
        'R2 Score': r2
    }
    
    print('Results for {}:'.format(model_name))
    print('Mean absolute error on model is {:.4f}'.format(mae))
    print('Mean squared error on model is {:.4f}'.format(mse))
    print('The r2 score on model is {:.4f}'.format(r2))
    
    return results_dict


results_fit = modelresults(y_pred_rfr_fit, 'Random Forest Fit')
results_optuna_200 = modelresults(y_pred_rfr_optuna_200_trails, 'Random Forest Optuna (200 trials)')
results_optuna_1000 = modelresults(y_pred_rfr_optuna_1000_trails, 'Random Forest Optuna (1000 trials)')
results_optuna_1000_super = modelresults(y_pred_rfr_optuna_1000_super_trails, 'Random Forest Optuna adjusted (1000 trials)')

results_df = pd.DataFrame([results_fit, results_optuna_200, results_optuna_1000,results_optuna_1000_super])
print(results_df)

The metrics provided (MAE, MSE, R2 Score) are commonly used to evaluate regression models. Let’s discuss each metric and provide insight into what they mean, along with considerations for what can be considered a good score.

1. Mean Absolute Error (MAE):

MAE represents the average absolute difference between the actual and predicted values. It measures the average magnitude of errors without considering their direction.

A lower MAE indicates better model performance. MAE is easy to understand and its values are in the same unit as the target variable.

2. Mean Squared Error (MSE):

MSE is the average of the squared differences between actual and predicted values. It penalizes larger errors more heavily than MAE.

Like MAE, lower MSE values indicate better model performance. However, because of squaring, MSE is more sensitive to outliers.

3. R2 Score (Coefficient of Determination):

R2 Score represents the proportion of the variance in the dependent variable that is predictable from the independent variables. It ranges from 0 to 1, where 1 indicates perfect prediction.

Higher R2 scores indicate better model fit. A score of 1 means the model explains all the variability in the target variable.

Considerations:

All the models, including the baseline Random Forest Fit and the ones tuned with Optuna, exhibit excellent performance.

The R2 scores are very close to 1, indicating that the models explain a high percentage of the variance in the target variable.

When comparing the models, slight variations in MAE, MSE, or R2 scores may not have a significant impact on the overall assessment, as they all demonstrate high-quality predictions.

In summary, our models perform very well, and the choice between them might depend on other factors such as computational resources, training time, or interpretability.