Connect Devices with Python Concurrency

When we try to connect network and system devices with the Python code, we try to connect many devices. We write the scripts to collect 10 or 100 devices and configure or collect data from each other. In the previous articles, we used paramiko and netmiko modules to connect multiple devices. But in those examples, we use the for loop to connect them. With the for loop, we login to devices one by one. So, after the connection to a device, we configure or collect data from the device and connect it to the next device in the list. For loop runs each iteration in order.

If we have 100 devices, and the data collection takes 30 seconds on each device, we have 50 minutes (3000 seconds) to finish the script. It’s not an automation script in the real world. Automation must be fast and simple.

Instead of the for loop, we can use the parallelism feature of the Python language as “Concurrency”. When we use it, python code connects to multiple devices simultaneously. For example, if we set the concurrent value to 20, in each process, we connect to 20 devices. In the same scenario, for 100 devices, in the 5 processes, we finish our script which takes 2.5 minutes (150 seconds). It’s one of the real power of Python automation.

Concurrent is a built-in Python module. So, we directly import it. We specifically import the “ThreadPoolExecuter” function from the “concurrent.futures” module. We also use netmiko to connect devices. So, we also import it.

After that, we define the device IP addresses that we want to login in a list and bind them to a variable.

from netmiko import Netmiko
from concurrent.futures import ThreadPoolExecutor

hosts = ["10.1.1.1", "10.1.1.2", "10.1.1.3", "10.1.1.4", "10.1.1.5"]

We create a function as “server_reboot,” and the parameter is “ip”. Because all of our devices are Ubuntu servers here, we have the same parameters as username, password, and device type. So, we don’t define each device's information like in the previous articles. We can use the same code for other device types such as Cisco routers. The only parameter that is unique on each server is the IP address. So, we write the “ip” parameter which is defined in the function and bind to the “host” key in the device dictionary.

After that, we connect the device with the Netmiko function and send the reboot command to reboot devices. By the way, the reboot command must be run in the admin privilege in Ubuntu, so we add another key as a secret in the device information. When netmiko tries to enter admin privilege with the “sudo” command, it uses the value of the secret key. We finish functioning with the return at the end.

def server_reboot(ip):
    device = {
        "host": ip,
        "username": "admin",
        "password": "test",
        "device_type": "linux",
        "secret": "test"
    }
    net_connect = Netmiko(**device)
    net_connect.send_config_set("reboot")
    return

Our connection function is ready. We can call the concurrent feature to login devices simultaneously. In the ThreadPoolExecutor function, we define max_workers. The default value is 5. We write as 10. So, the Python code tries to connect 10 devices in each process. We bind “ThreadPoolExecutor(max_workers=10)” to “executor”. So, when we write executer, it calls “ThreadPoolExecutor(max_workers=10)”

Finally, we map 2 things in the executor in the parentheses. 1st item must be a function, and the 2nd item must be an iterable such as a list variable.

with ThreadPoolExecutor(max_workers=10) as executor:
    result = executor.map(server_reboot, hosts)

When we run the code, we connect to 5 devices at the same time and reboot all of them. We don’t wait for each loop to finish the process.

Note that when the max_workers value is high, the process that Python code takes on your local device resource such as CPU increases.

You can enroll in my “Network Automation with Python” — LIVE training HERE.

For +100 Python networking scripts, check my book HERE.

Here is the full code: