Learn Python Multiprocessing Module Easily with Examples

Python’s built-in multiprocessing module is quite important if there are requirements for parallelism processing, in addition to the built-in threading module, another one is multiprocessing.

The advantage of using multiprocessing is that it can greatly avoid the impact of Python GIL on program performance, but the bad thing is that it consumes more memory. Even so, it is still a module that must be understood.

This article will learn how to use the multiprocessing module through several examples.

Requirements #

• Python 3.10

multiprocessing.Pool #

Pool is a convenient class in the multiprocessing module that provides a simple way to define the number of workers, which is the number of parallel processes to be used. For example, Pool(4) means that there will be 4 parallel processes running.

Here is an example program to send emails to 100 users using Pool. It takes 1 second to send out each email. Without parallel processing, it takes 100 seconds to send all the 100 emails using a for loop. If using 4 workers in parallel, it theoretically takes around 25 seconds (100 / 4) to process, which is the power of parallel processing.

from time import sleep, time
from multiprocessing import Pool


def send_mail(username, coupon_code, extra_sentence):
    print(username, coupon_code, extra_sentence or '')
    sleep(1)  # to simulate calling an API here


s_time = time()
with Pool(4) as pool:
    for idx in range(100):
        pool.apply_async(
            send_mail,
            (f'user{idx:0>2}', f'vipcode{idx:0>2}'),
            {'extra_sentence': f'{idx:0>4}'}
        )
    pool.close()
    pool.join()

print('total time：', time() - s_time)

After creating 4 workers (i.e. 4 processes) in line 11 of the above example, the loop in lines 12 and 13 enqueues the send_email function and its arguments to the pool via apply_async for execution.

Next, pool.close() tells the pool that no other data or tasks will be added to the queue. Note that the Workers will not start working immediately, but will start when pool.join() is called.

Below is the execution result, it can be seen that it took about 26 seconds, close to our estimated value.

user01 vipcode01 0001
user02 vipcode02 0002
user03 vipcode03 0003
user05 vipcode05 0005
user06 vipcode06 0006
user07 vipcode07 0007
user04 vipcode04 0004
user08 vipcode08 0008
...
total time： 26.362775325775146

Moreover, it is worth noting that the numbers in the above results are not consecutive, which is caused by the unevenly execution speed of workers. This is a normal phenomenon. It also means that in the world of parallel processing, we usually cannot predict the order of execution and termination. If you need 100% guarantee of the order of execution and termination, parallel processing may not be suitable for you.

The parent process shares data (shared memory) with the child process. #

After the initial experience of Pool’s features, there is a concept that needs to be known.

If a Python program (assumed to be A) uses Pool/Process to generate other programs (assumed to be B, C, D) to work collaboratively, then A is the parent process and B, C, D are child processes.

The memory between Processes is basically independent, and if they need to communicate, they have to use IPC (Inter-Process Communication) technology.

There are 4 common IPC techniques:

• By writing data via a file, e.g. A Process, to a file and then reading it by B Process, a communication effect can be achieved.
• Socket communicates through TCP/UDP and other network protocols.
• Communication is done through a shared memory block.
• Signal communicates through the signal mechanism provided by the operating system, for example, a process sends a signal to a process B, and B process responds differently to different signals to achieve communication effects.

Of course, there are more than just the four IPC technologies mentioned above; for more information, please refer to Inter-process communication - Wikipedia.

Python’s multiprocessing module also provides a Shared Memory method for us to implement communication between Processes, such as the Value('d', 0.0) in the following example, which is actually a shared memory. This shared memory is passed to the child process in line 11 through p = Process(target=add_one, args=(num, )), so that the child process can access the shared memory. After the child process adds 1, the parent process reads and prints its value in line 16.

For more information about Value, please refer to the official document here.

from multiprocessing import Process, Value


def add_one(v):
    v.value += 1


num = Value('d', 0.0)


p = Process(target=add_one, args=(num, ))
p.start()
p.join()


print(num.value)

The execution result of the above example is.

1.0

What would happen if we modify the above example to a Pool form and submit it to 100 child processes?

from multiprocessing import Pool, Value


def add_one(v):
    v.value += 1


num = Value('d', 0.0)


with Pool(4) as pool:
    for _ in range(100):
        pool.apply_async(add_one, args=(num, ), error_callback=lambda e: print(e))
    pool.close()
    pool.join()


print(num.value)

If the above example is executed, the following error will appear and the value of num.value is not expected to be 100.0:

...(skip)...
Synchronized objects should only be shared between processes through inheritance
Synchronized objects should only be shared between processes through inheritance
0.0

The reason is that when Python starts a new child process, there are different default methods due to different operating systems, which is called start method.

Currently there are 3 start methods:

• spawn is supported by both Unix and Windows systems, and is the default method used to start a new child process in macOS and Windows. In simple terms, this method starts a virtually new child process, which can be understood as the data in the memory is also new.
• fork is only supported by Unix systems and is the default method supported by Unix. This method starts a child process that is identical to the parent process, and the resources in the parent process are also inherited by the newly started child process. = forkserver is also a method supported by Unix systems. It basically works similarly to fork, but there is an additional fork server between the parent process and the child process to help proxy the forking of the child process.

If you want to know what your start method is, you can execute the following program:

import multiprocessing
multiprocessing.get_start_method()

The starting method of this article is fork. If you are using Windows system, there will be problems with the subsequent examples. It is suggested to use Google Colab for execution. For other systems, you can call multiprocessing.set_start_method('fork') to set the start method to fork.

When using the fork method, the shared memory Value will be inherited by the child processes, so when creating the child processes, Value will also be inherited, so there is no need to pass parameters.

Therefore, the example of the problem mentioned above can be changed to the following form, changing add_one to not require any parameters:

from multiprocessing import Pool, Value


num = Value('d', 0.0)


def add_one():
    num.value += 1


with Pool(4) as pool:
    for _ in range(100):
        pool.apply_async(add_one, error_callback=lambda e: print(e))
    pool.close()
    pool.join()


print(num.value)

The above example’s execution result.

96.0

It can be found that the above result is not 100.0, which is due to a race condition, some processes read the value of n before it is updated, resulting in an inaccurate value.

This article does not discuss race condition.

To correct this error, the correct way is to introduce Lock to ensure that only one process can update the value of num at the same time.

Python’s multiprocessing.Value has implemented a lock, so in order to fix the race condition problem, it can be changed to the following example, and an additional lock needs to be obtained through with num.get_lock() in order to change the value of num:

from multiprocessing import Pool, Value, get_start_method, set_start_method



def add_one():
    with num.get_lock():
        num.value += 1


num = Value('d', 0.0)


with Pool(4) as pool:
    for _ in range(100):
        pool.apply_async(add_one, error_callback=lambda e: print(e))
    pool.close()
    pool.join()


print(num.value)

multiprocessing.Manager #

In addition to the lock mechanism of the aforementioned example, Python also provides multiprocessing.Manager to make it easy for us to use locks to solve race condition situations.

Managers provide a way to create data which can be shared between different processes, including sharing over a network between processes running on different machines. A manager object controls a server process which manages shared objects. Other processes can access the shared objects by using proxies.

In addition to providing shared data between processes, Manager also provides a way for us to share data between different hosts through a networked approach.

The aforementioned example of add_one that has the potential to calculate errors can be changed to the following form using Manager:

from multiprocessing import Pool, Manager


m = Manager()
n = m.Value('d', 0.0)
lock = m.Lock()


def add_one():
    lock.acquire()
    n.value += 1
    lock.release()


with Pool(4) as pool:
    for _ in range(100):
        pool.apply_async(add_one, error_callback=lambda e: print(e))
    pool.close()
    pool.join()


print(n.value)

Actualy, Manager() returns a SyncManager.

The above example creates an instance of Manager in line 4 to manage the shared data between processes, then adds 1 shared data n in line 5 and a lock in line 6.

Before updating the value of n, a lock must be obtained, which is part of line 10. After updating, the lock should be released to enable other processes to update the value as well, which is part of line 12.

Thus, the resulting value will be 100.0 as expected!

Conclusion #

Above is a simple explanation and example of Python multiprocessing. This article serves as a beginner’s guide, allowing beginners to experience the wonders of multiprocessing through a few simple examples to understand the concept of “parallel” processing. There are actually quite a few opportunities to use multiprocessing in actual work. If you are interested, you can read the official documents of multiprocessing to learn more about multiprocessing.

Happy Coding!

References #

Multiprocessing - Process-based parallelism