How to run parallel tasks in Python
concurrent.futures module provides a high-level interface for launching parallel tasks in Python.
Parallel tasks can be launched with threads or with processes.
For threads concurrent.futures has class ThreadPoolExecutor and for processes it has class ProcessPoolExecutor.
In this article we will see how to implement ThreadPoolExecutor.
Parallel execution using threads in Python
Import modules
import concurrent.futures
from timeit import default_timer as timer
import time
Calculate Time required for running the tasks without implementing ThreadPoolExecutor
import concurrent.futures
from timeit import default_timer as timer
import time
def test_func(value):
msg = f"Getting {value } from list_of_values"
time.sleep(1)
print(msg)
list_of_values = ["first", "second", "third", "fourth", "fifith", "sixth"]
start = timer()
for i in list_of_values:
test_func(i)
end = timer()
print("Time taken without parallel execution is ",end - start)
Output
Getting first from list_of_values
Getting second from list_of_values
Getting third from list_of_values
Getting fourth from list_of_values
Getting fifith from list_of_values
Getting sixth from list_of_values
Time taken without parallel execution is 6.023686041785554
So it took around 6 seconds to complete the task with parallel execution, lets implement parallel execution and calculate the execution time.
Execute the below code and check the execution time
import concurrent.futures
from timeit import default_timer as timer
import time
def test_func(value):
msg = f"Getting {value } from list_of_values"
time.sleep(1)
print(msg)
list_of_values = ["first", "second", "third", "fourth", "fifith", "sixth"]
start = timer()
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
executor.map(test_func,list_of_values)
end = timer()
print("Time taken with parallel execution is ",end-start)
Output
Getting third from list_of_values
Getting fourth from list_of_values
Getting first from list_of_values
Getting second from list_of_values
Getting fifith from list_of_values
Getting sixth from list_of_values
Time taken with parallel execution is 2.01107863221897
We can see parallel execution reduced time significantly as it took only 2 seconds this time in comparison to 6 Seconds previously without implementing ThreadPoolExecutor.
Category: Python
