Databricks Notebook Widgets

In this blog post, I will talk about databricks notebook widgets and how to simplify the usage of the text widget. for this poc i have used python notebook and custom python function.

What is databricks notebook widgets?

Databricks notebook widgets are interactive elements that allow users to input values or make selections within a Databricks notebook. These widgets can be used to parameterize the notebook and make it more dynamic. With widgets, you can create user-friendly interfaces to customize the behaviour of your notebook without having to modify the code.

Databricks provides a variety of widget types, such as text input, dropdown checkbox and multiple select. These widgets can be added to your notebook using the dbutils.widgets module.

Here is an overview of some commonly used Databricks notebook widgets and you can find more about widgets here.
https://learn.microsoft.com/en-gb/azure/databricks/notebooks/widgets

In this article, I am going to cover text widget.

1. Text Input Widget: Allows users to enter a text value. Example: dbutils.widgets.text("widget_name", "default_value", "Label")
2. Dropdown Widget: Provides a dropdown menu to select from a list of options. Example: dbutils.widgets.dropdown("widget_name", "default_value", ["option1", "option2", "option3"], "Label")
3. Checkbox Widget: Allows users to select one or more options using checkboxes. Example: dbutils.widgets.checkbox("widget_name", True, "Label")
4. Radio Button Widget: Provides a set of options as radio buttons for single selection. Example: dbutils.widgets.radio("widget_name", "default_value", ["option1", "option2", "option3"], "Label")
5. Date Picker Widget: Enables users to select a date from a calendar view. Example: dbutils.widgets.date("widget_name", "2022-01-01", "Label")
6. File Upload Widget: Lets users upload files directly into the notebook. Example: dbutils.widgets.file("widget_name", "default_path", "Label")

Once widgets are created, their values can be accessed using the dbutils.widgets.get() method. These values can then be used as input parameters in your code to customize data processing, visualization, or analysis.

Widgets in Databricks notebooks provide a convenient way to interact with your notebook’s code and make it more versatile and user-friendly.

Now, let’s talk about complex notebook, I had a scenario where I have to pass 20 parameters to the notebook. Creation of 20 Individual parameters and maintain them it is quite painful. So came up with a solution. club all the required parameter in single json object convert them dynamically in variables. So it can be used down the line in notebook without defining then individually.

import json

def dbr_widgets_to_variables(pipeline_parameters):
    """
    Convert the pipeline parameters from JSON format to Python variables.
    
    Parameters:
    pipeline_parameters (str): JSON string representing the pipeline parameters.

    Returns:
    None
    """
    pipeline_parameters_json = json.loads(pipeline_parameters)
    print(f"{json.dumps(pipeline_parameters_json,indent=4)}")
    print(f"")
    for i in pipeline_parameters_json:
        print("%s = %s" % (i, pipeline_parameters_json[i]))
        globals()[i] = pipeline_parameters_json[i]

First, the code uses the json module to parse the “pipeline_parameters” string into a JSON object called “pipeline_parameters_json”. Then, it prints the formatted JSON string using the dumps() function and indents it for better readability.

Next, the code iterates through each key-value pair in the “pipeline_parameters_json” object. For each pair, it prints the key and value using string formatting.

Finally, the code uses globals() and the key-value pairs from “pipeline_parameters_json” to create variables in the global scope. The variables are named after the keys in “pipeline_parameters_json” and their values are the corresponding values from the JSON object.

Example: Convert Json Object to Variables

dbutils.widgets.text("pipeline_parameters", "", "Pipeline Configuration Name")
pipeline_cfg_name = dbutils.widgets.get("pipeline_parameters")
dbr_widgets_to_variables(pipeline_cfg_name)

This above code is using Databricks utility (dbutils) to create a text widget asking the user to input a pipeline configuration name.

The value entered by the user is then retrieved using dbutils.widgets.get() and assigned to the variable pipeline_cfg_name.

Finally, the dbr_widgets_to_variables() function is called with pipeline_cfg_name as an argument to convert the widget value into a variable.

Input Parameter for Demo

{
  "scope_01": "dbr-key-vault-scope",
  "spn_client_id_01": "dbr-spn-application-id",
  "spn_secret_01": "dbr-spn-secret",
  "tenant_id_01": "tenant-id",
  "expriy_of_token_01": 100,
  "comment_01": "Testing Widgts 01",
  "databricks_url_01": "https://uksouth.azuredatabricks.net",
  "scope_02": "dbr-key-vault-scope",
  "spn_client_id_02": "dbr-spn-application-id",
  "spn_secret_02": "dbr-spn-secret",
  "tenant_id_02": "tenant-id",
  "expriy_of_token_02": 100,
  "comment_02": "Testing Widgts 02",
  "databricks_url_02": "https://ukwest.azuredatabricks.net",
  "scope_03": "dbr-key-vault-scope",
  "spn_client_id_03": "dbr-spn-application-id",
  "spn_secret_03": "dbr-spn-secret",
  "tenant_id_03": "tenant-id",
  "expriy_of_token_03": 100,
  "comment_03": "Testing Widgts 03",
  "databricks_rul_03": "https://westeurope.azuredatabricks.net"
}

Results: Json Object Converted to the Python Variables Dynamically.

Converted Json Object to Variables

In Conclusion, The POC proves that It has simplify the Input parameters for databricks notebook when it comes to handle many dynamic value for a particular notebook. If you like the poc please start following me on medium. lot more to come.

Thanks