Parallel Simulations and Green Infrastructure Analysis with PySWMM

PySWMM, a Python wrapper for the EPA Storm Water Management Model (SWMM), has become a powerful tool for simulating stormwater systems and analyzing their performance. Despite its capabilities, it lacks a direct approach to parallelizing simulations, which hinders the potential for executing multiple simulations in a shorter timeframe. In this article, I will show a way to run parallel simulations using PySWMM.

The python script and the SWMM inp file are on github

The SWMM model enables the representation of various combinations of green infrastructure practices (also known as Low Impact Development (LID) techniques) to investigate their effectiveness in mitigating runoff generation and enhancing water storage. The performance of the LID controls depends on several parameters. For simplicity, this article focuses only on Bio-retention cells. The Bio-retention cell consists of 4 layers: surface, soil, storage, and drain layers, as shown in Figure 1. We will vary the following parameters:

• Sub-catchment area and width.
• Ratio of Bio-retention area to the total sub-catchment area.
• Berm height of the surface layer.
• Thickness of the soil layer.
• Thickness of the storage layer.
• Seepage rate of the storage layer.

Figure 1. Schematic figure of the bioretention unit model (source Lerer et al., 2022)

SWMM model:

Adopting a similar approach to Lerer et al., 2022, in order to distinguish the rainfall-runoff process in the catchment from the hydrological processes in the LID controls, I positioned the Bio-retention cell within a sub-catchment that receives excess runoff from the upstream sub-catchment (representing the area that drains into the Bio-retention cell), as depicted in Figure 2.

Figure 2. Schematic illustration of the water flows in the SWMM models used to simulate Bio-retention performance

Parallelizing PySWMM simuations:

The SWMM model is not thread-safe, which means that only one model can be opened at a time per ‘instance’ of Python. To overcome this limitation, we can utilize a combination of queue, threading, and subprocess. This approach enables us to run each model on its own process simultaneously. The workflow is as follows:

1- Begin by listing the parameters we need to vary and store them in a Queue (a list of job descriptions).

2- Utilize threads to handle the various jobs in the queue. Each thread represents a simulation that can run in parallel, and the number of threads depends on the system’s capabilities.

3- Within the worker function, call a subprocess to execute the ‘pyswmm_wrapper.py’ file, where we can make the desired changes to the SWMM model.

Python Script:

1- Begin by listing the parameters we need to vary and store them in a Queue (a list of job descriptions).

I generated 1000 random combinations of parameters (num_iterations) by selecting values within predefined ranges. These combinations were then stored in a queue named “simulations_queue.

# Number of simulations 
num_iterations=1000 # You can change it depending on your problem

# Parameter Range

area2_range=[0.02,0.2] # Percentage range of Bioretention cell area

# Surface layer thickness
surface_thickness=[100,600]

# soil layer thickness
soil_thickness=[300,600]

#Storage layer thickness
storage_thickness=[800,2000]

#Drain offset
drain_offset=[0,1000]

#Soil layer conductivity
seepage_rate=[0.5,120]

# Create a queue and add all the simulations to it
simulations_queue = queue.Queue()
for _ in range(num_iterations):
        params = (
            round(random.uniform(min(area2_range), max(area2_range)), 2),
            round(random.uniform(min(surface_thickness), max(surface_thickness)), 2),
            round(random.uniform(min(soil_thickness), max(soil_thickness)), 2),
            round(random.uniform(min(storage_thickness), max(storage_thickness)), 2),
            random.choice(drain_offset),
            round(random.uniform(min(seepage_rate), max(seepage_rate)), 2)
        )
        simulations_queue.put(params)
#simulations_queue.get()

2- Utilize threads to handle the various jobs in the queue. Each thread represents a simulation that can run in parallel, and the number of threads depends on the system’s capabilities.

In the following function:

• “pyswmm_wrapper.py” represents the Python code where we modify the parameters in the SWMM model and specify the desired output.
• 'SWMM.inp' refers to the SWMM model that we will modify using the 'pyswmm_wrapper.py' code.
• The list of parameters follows the same order as they are added to the simulation queue.
• Finally, the function’s output is saved in pre-created lists for further analysis.

# Function to process the simulations from the queue
def process_simulation_queue():
    while not simulations_queue.empty():
        sim_params = simulations_queue.get()
        result = subprocess.run(["python", "pyswmm_wrapper.py", "SWMM.inp",
                                 str(sim_params[0]),  # A2
                                 str(sim_params[1]),  # Surface_thickness_value
                                 str(sim_params[2]),  # Soil_thickness_value
                                 str(sim_params[3]),  # Storage_thickness_value
                                 str(sim_params[4]),  # Drain_offset_value
                                 str(sim_params[5])   # Seepage_rate_value
                                ], capture_output=True, text=True)
        
        #print(result)
        #print("---------------------------------------------------------------------------------------------------")
        
                                 
        if result.returncode == 0:
            #print("YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY")
            output_lines = result.stdout.splitlines()
            #print(output_lines)
            Results_area2.append(float(output_lines[0]))           
            Results_Bio_surface_thickness.append(float(output_lines[1]))
            Results_Bio_soil_thickness.append(float(output_lines[2]))
            Results_Bio_storage_thickness.append(float(output_lines[3]))
            Results_Bio_drain_offset.append(float(output_lines[4]))
            Results_Bio_seepage_rate.append(float(output_lines[5]))
            Results_S1_stat.append(output_lines[6])
            Results_S2_stat.append(output_lines[7])
            Results_system_stat.append(output_lines[8])

        simulations_queue.task_done()

Then, I made adjustments to the ‘pyswmm_wrapper.py’ to obtain the parameter values for each thread, perform the simulations, and subsequently return the parameter values along with the statistics of the two subcatchments and the entire system.

Finally we create and start the threads using the below code. You will need to adjust the number of threads as per your system's capabilities

# Create and start the threads
num_threads = 20  # Adjust the number of threads as per your system's capabilities
threads = []
start_time=time.time()
for _ in range(num_threads):
    thread = threading.Thread(target=process_simulation_queue)
    thread.start()
    threads.append(thread)

# Wait for all threads to finish
for thread in threads:
    thread.join()
    
print("Finish")
end_time=time.time()

elapsed_time = (end_time - start_time)/3600

print(f"Function took {elapsed_time:.6f} hrs to execute.")

With the results now available, we can proceed to store them in a dataframe, facilitating further analysis. I would like to extend my gratitude to Bryant E. McDonnell for his invaluable guidance and support in resolving this matter.

References:

Lerer, S.M., Guidje, A.H., Drenck, K.M.L., Jakobsen, C.C., Arnbjerg-Nielsen, K., Mikkelsen, P.S. and Sørup, H.J.D., 2022. Constructing an inventory for fast screening of hydraulic and hydrologic performance of stormwater control measures. Blue-Green Systems, 4(2), pp.213–229.