How to use config files with Python? (Cheat sheet included!)

The ultimate guide to writing reusable Python Scripts with JSON, YAML, TOML, and INI configuration files

Photo by Mpho Mojapelo on Unsplash

When writing Python scripts, I like to make them dynamic and reusable. And hardcoding the configurations goes against the purpose.

I use configuration files to pass the configuration details in my Python scripts, it gives me the flexibility to write a good-looking, functional, and reusable code.

In this blog post, I will take you through using configuration files with the most popular formats .json, .yaml and .ini in your Python script.

1: Server Configuration Management Using JSON config files in Python.

JSON provides a lightweight, easy-to-read format for data interchange, making it perfect for APIs, web applications, and configurations requiring structured data. Its simplicity and structure cater well to modern development needs.

Scenario

Sometimes we have to use Python for server configuration management. We want the automation to be used by others, they can update server details in .json config without modifying the script. It will do the same job for them.

Example

• Create a JSON Configuration File

servers_config.json

{
    "servers": [
        {
            "name": "web-server-1",
            "ip": "192.168.1.101",
            "roles": ["web", "app"]
        },
        {
            "name": "db-server-1",
            "ip": "192.168.1.102",
            "roles": ["db"]
        }
    ]
}

• Read the JSON Configuration File in Python

We will use built-in json module to read the JSON file configuration and configure the servers.

server-config.py

import json

def read_server_config(file_path):
    with open(file_path, 'r') as file:
        config = json.load(file)
        return config.get('servers', [])

def configure_servers(servers):
    for server in servers:
        print(f"Configuring {server['name']} with IP {server['ip']} and roles {server['roles']}")

if __name__ == "__main__":
    server_config_path = 'servers_config.json'
    servers = read_server_config(server_config_path)
    configure_servers(servers)

• Run the code

This script reads the server configuration from a JSON file and prints the configuration steps for each server.

If you want to see a more elaborated example of using .json config files, read this blog post.

2: Application Deployment Using YAML config files in Python

YAML stands out with its human-readable data serialization, using indentation for nested structures. Perfect for DevOps tools like Ansible, Kubernetes, and CI/CD pipelines.

Scenario

Managing application configurations through a YAML file is highly efficient when automating multi-environment deployment processes in Python.

Example

• Create a YAML Configuration File

deployment_config.yaml

applications:
  - name: "frontend"
    repo: "git@github.com:akhileshmishrabiz/Devops-zero-to-hero.git"
    deploy_path: "project4"
  - name: "backend"
    repo: "git@github.com:akhileshmishrabiz/python-for-devops.git"
    deploy_path: "python-flask-postgres-app"

• Read the YAML Configuration File in Python

Use the PyYAML library to read the YAML file and deploy the applications.

# Install PyYAML
pip install pyyaml

• Python script that will fetch the config value form.yaml config file

deploy.py

import yaml
import os

def read_deployment_config(file_path):
    with open(file_path, 'r') as file:
        config = yaml.safe_load(file)
        return config.get('applications', [])

def deploy_applications(applications):
    for app in applications:
        print(f"Deploying {app['name']} from {app['repo']} to {app['deploy_path']}")
        # Example deployment steps
        # os.system(f"git clone {app['repo']} {app['deploy_path']}")

if __name__ == "__main__":
    deployment_config_path = 'deployment_config.yaml'
    applications = read_deployment_config(deployment_config_path)
    deploy_applications(applications)

• Run the code

This script reads the application deployment configuration from a YAML file and prints the deployment steps for each application.

3: Managing Environment Variables Using INI config files in Python

INI files offer a straightforward, flat-file format for configurations, featuring sections, properties, and values. Ideal for legacy systems, they excel in managing simple key-value pairs for application settings and user preferences.

Scenario

When automating tasks in Python, managing environment-specific configurations separately from the script is essential. This example demonstrates how to read environment variables from an .ini configuration file and set them as system environment variables

Example

• Create an INI Configuration File

env_config.ini

[development]
DATABASE_URL = postgresql://localhost/devdb
SECRET_KEY = dev-secret-key 

[production]
DATABASE_URL = postgresql://localhost/proddb
SECRET_KEY = prod-secret-key

• Write the Python Code to Read the INI Configuration

We can use configparser module to read the INI file and set the environment variables.

env-config.py

import configparser
import os

def read_env_config(file_path, environment):
    config = configparser.ConfigParser()
    config.read(file_path)
    if environment in config:
        return config[environment]
    return {}

def set_environment_variables(env_vars):
    for key, value in env_vars.items():
        os.environ[key] = value
        print(f"Set {key} to {value}")

if __name__ == "__main__":
    env_config_path = 'env_config.ini'
    environment = 'development' 
    env_vars = read_env_config(env_config_path, environment)
    set_environment_variables(env_vars)

• Run the code

This script reads environment variables from an INI file and sets them in the system environment. You can change the environment variable to switch between different configurations (e.g., development, production).

4: Managing database configuration Using TOML config files in Python

TOML blends readability with support for complex data structures, making it a favored choice for configurations in languages like Rust and Python. It’s well-suited for scenarios demanding both clarity and intricate configuration options.

Scenario

In microservices-based applications, the best practice is to manage the database connection details out of the application code. We can provide the database details with a .toml config file to, and use toml library to parse the data in Python.

Example

• Create a TOML Configuration File

config.toml

[database]
host = "localhost"
port = 5432
name = "mydatabase"
user = "admin"
password = "adminpass"

[server]
host = "127.0.0.1"
port = 8000

• Install the toml library to parse TOML files in Python.

Python 3.11 onwards, tomllib is included in the Python Standard Library.

pip install toml

• Write the Python Code to Read the TOML Configuration

read_config.py

import toml

def read_toml_config(file_path: str) -> dict:
    """
    Read a TOML configuration file and return the data as a dictionary.

    :param file_path: Path to the TOML configuration file.
    :return: Dictionary containing configuration data.
    """
    with open(file_path, 'r') as file:
        config_data = toml.load(file)
    return config_data

def print_config(config_data: dict) -> None:
    """
    Print the configuration data.

    :param config_data: Dictionary containing configuration data.
    """
    print("Database Configuration:")
    print(f"  Host: {config_data['database']['host']}")
    print(f"  Port: {config_data['database']['port']}")
    print(f"  Name: {config_data['database']['name']}")
    print(f"  User: {config_data['database']['user']}")
    print(f"  Password: {config_data['database']['password']}")

    print("\nServer Configuration:")
    print(f"  Host: {config_data['server']['host']}")
    print(f"  Port: {config_data['server']['port']}")

if __name__ == "__main__":
    config_file_path = 'config.toml'
    config = read_toml_config(config_file_path)
    print_config(config)

• Run the code

This Python script demonstrates how to read and print configuration details from a TOML file. Using the toml library, the script reads configuration data for both database and server settings.

Conclusion

Use this blog post as a cheat sheet for utilizing config files with Python in various formats.

Generate requirements.txt for your Python project on the fly

If you found this blog post useful, clap, follow, and subscribe so you don’t miss my future articles.

Connect with me on Linkedin: https://www.linkedin.com/in/akhilesh-mishra-0ab886124/