python multi threading vs multi processing, and best practices, multi threading wait for all

 https://towardsdatascience.com/multithreading-and-multiprocessing-in-10-minutes-20d9b3c6a867

By formal definition, multithreading refers to the ability of a processor to execute multiple threads concurrently, where each thread runs a process. Whereas multiprocessing refers to the ability of a system to run multiple processors in parallel, where each processor can run one or more threads.

https://stackoverflow.com/questions/3044580/multiprocessing-vs-threading-python

The threading module uses threads, the multiprocessing module uses processes. The difference is that threads run in the same memory space, while processes have separate memory. This makes it a bit harder to share objects between processes with multiprocessing. Since threads use the same memory, precautions have to be taken or two threads will write to the same memory at the same time. This is what the global interpreter lock is for.

Spawning processes is a bit slower than spawning threads.

Best practices

As mentioned in the question, Multiprocessing in Python is the only real way to achieve true parallelism. Multithreading cannot achieve this because the GIL prevents threads from running in parallel.

As a consequence, threading may not always be useful in Python, and in fact, may even result in worse performance depending on what you are trying to achieve. For example, if you are performing a CPU-bound task such as decompressing gzip files or 3D-rendering (anything CPU intensive) then threading may actually hinder your performance rather than help. In such a case, you would want to use Multiprocessing as only this method actually runs in parallel and will help distribute the weight of the task at hand. There could be some overhead to this since Multiprocessing involves copying the memory of a script into each subprocess which may cause issues for larger-sized applications.

However, Multithreading becomes useful when your task is IO-bound. For example, if most of your task involves waiting on API-calls, you would use Multithreading because why not start up another request in another thread while you wait, rather than have your CPU sit idly by.

TL;DR

• Multithreading is concurrent and is used for IO-bound tasks
• Multiprocessing achieves true parallelism and is used for CPU-bound tasks

Let's say you're making multiple API calls to request some data, in this case the majority of the time is spent waiting on the network. As it awaits this network I/O, the GIL can be released to be used by the next task. However, the task will need to re-acquire the GIL in order to go to execute the rest of any python code associated with each API request, but, as the task is waiting for the network, it does not need to hold on to the GIL.  (Global interpreter lock)

Multi threading code:

https://creativedata.stream/multi-threading-api-requests-in-python/

from concurrent.futures.thread import ThreadPoolExecutor

import time

    

def call_script(ordinal, arg):

    print('Thread', ordinal, 'argument:', arg)

    time.sleep(2)

    print('Thread', ordinal, 'Finished')

    

args = ['argumentsA', 'argumentsB', 'argumentsC']

    

with ThreadPoolExecutor(max_workers=2) as executor:

    ordinal = 1

    for arg in args:

        executor.submit(call_script, ordinal, arg)

        ordinal += 1

print('All tasks has been finished')

-------

output:

Thread 1 argument: argumentsA

Thread 2 argument: argumentsB

Thread 1 Finished

Thread 2 Finished

Thread 3 argument: argumentsC

Thread 3 Finished

All tasks has been finished

Multi threading wait for all to be completed

https://stackoverflow.com/questions/11968689/wait-until-all-threads-are-finished-in-python

Using a ThreadPoolExecutor the code would be:

from concurrent.futures.thread import ThreadPoolExecutor
import time
    
def call_script(ordinal, arg):
    print('Thread', ordinal, 'argument:', arg)
    time.sleep(2)
    print('Thread', ordinal, 'Finished')
    
args = ['argumentsA', 'argumentsB', 'argumentsC']
    
with ThreadPoolExecutor(max_workers=2) as executor:
    ordinal = 1
    for arg in args:
        executor.submit(call_script, ordinal, arg)
        ordinal += 1
print('All tasks has been finished')

The output of the previous code is something like:

Thread 1 argument: argumentsA
Thread 2 argument: argumentsB
Thread 1 Finished
Thread 2 Finished
Thread 3 argument: argumentsC
Thread 3 Finished
All tasks has been finished