Python Fork() Exit Status

Most computer programs return an exit code to the operating system’s shell. Shell scripting tools can use the exit status of a program to indicate if the program exited normally or abnormally. In either case, the shell script can react depending on the outcome of the child process.

Python programs can use os.fork() to create child processes. Since the child process is a new instance of the program, it can be useful in some cases to inspect if the child process exited normally or not. For example, a GUI program may spawn a child process and notify the user if the operation completed successfully or not.

This post shows an example program taken from Programming Python: Powerful Object-Oriented Programming that demonstrates how a parent process can inspect a child process’ exit code. I added comments to help explain the workings of the program.


import os

exitstat = 0

# Function that is executed after os.fork() that runs in a new process
def child():
    global exitstat
    exitstat += 1
    print('Hello from child', os.getpid(), exitstat)
    # End this process using os._exit() and pass a status code back to the shell

# This is the parent process code
def parent():
    while True:
        # Fork this program into a child process
        newpid = os.fork()
        # newpid is 0 if we are in the child process
        if newpid == 0:
            # Call child()
        # otherwise, we are still in the parent process
            # os.wait() returns the pid and status and status code
            # On unix systems, status code is stored in status and has to
            # be bit-shifted
            pid, status = os.wait()
            print('Parent got', pid, status, (status >> 8))
            if input() == 'q':

if __name__ == '__main__':


This program is pretty basic. We have two functions, parent() and child(). When the program starts on line 38, it calls parent() to enter the parent() function. The parent() function enters and infinite loop that forks this program on line 20. The result of os.fork() is stored in the newpid variable.

Our program is executing in the child process when newpid is zero. If that case, we call our child() function. The child() function prints a message to the console on line 10 and then exits by calling os._exit() on line 13. We pass the exitstat variable to os._exit() whose value becomes the exit code for this process.

The parent process continues in the meantime. On line 32, we use os.wait() to return the pid and status of the child process. The status variable also containes the exitstat value passed to os._exit() in the child process, but to get this code, we have to perform a bit shift operation by eight bits. The following line prints the pid, status, and the child process’ exit code to the console. When the user presses ‘q’, the parent process ends.


Lutz, Mark. Programming Python. Beijing, OReilly, 2013.


Python Producer Consumer with Queue

The producer / consumer pattern is a common programming construct used in multithreaded applications where one thread acts as a producer of data while other threads consume the data. A web crawler application is a use case of the producer / consumer pattern. For example, the application may have a thread dedicated to crawling the web that gathers data (producer) while other threads index and store the data (consumers).

Producer and consumer threads need a way to share data. Python’s queue module provides one of many solutions. The Queue object is a FIFO object that lets the produce thread place data on the queue. Consumer threads are blocked by the Queue until the Queue has data for the consumer thread to read. When data becomes available, the consumer thread removes data from the Queue and does its work.

Below is an example program borrowed from Programming Python: Powerful Object-Oriented Programming that shows how to use a Queue to synchronize data between producer and consumer threads. I added my own comments to the code to help explain what is happening in the program.


# Specify the number of consumer and producer threads
numconsumers = 2
numproducers = 4
nummessages = 4

import _thread as thread, queue, time

# Create a lock so that only one thread writes to the console at a time
safeprint = thread.allocate_lock()

# Create a queue object
dataQueue = queue.Queue()

# Function called by the producer thread
def producer(idnum):
    # Produce 4 messages to place on the queue
    for msgnum in range(nummessages):
        # Simulate a delay

        # Put a String on the queue
        dataQueue.put('[producer id={}, count={}]'.format(idnum, msgnum))

# Function called by the consumer threads
def consumer(idnum):
    # Create an infinite loop
    while True:
        # Simulate a delay
            # Attempt to get data from the queue. Note that
            # dataQueue.get() will block this thread's execution
            # until data is available
            data = dataQueue.get()
        except queue.Empty:
            # Acquire a lock on the console
            with safeprint:
                # Print the data created by the producer thread
                print('consumer ', idnum, ' got => ', data)

if __name__ == '__main__':
    # Create consumers
    for i in range(numconsumers):
        thread.start_new_thread(consumer, (i,))
    # Create producers
    for i in range(numproducers):
        thread.start_new_thread(producer, (i,))
    # Simulate a delay
    time.sleep(((numproducers - 1) * nummessages) + 1)
    # Exit the program
    print('Main thread exit')

Detailed Explanation

This program shows the producer / consumer pattern in action. We begin by defining variables that specify the number of consumer threads (line 2), the number of produce threads (line 3), and the number of messages the producer threads make (line 4). The program creates a lock on line 9 so that only one thread can use the console at the same time. Then on line 12, the queue is created as a global variable.

Our first function, producer, is defined on lines 16-23. There isn’t anything fancy going on in this function. The function simply enters a for-each loop and creates 4 strings that are placed on dataQueue (line 23). Since dataQueue is a FIFO structure, worker threads will remove these Strings from dataQueue in the order they are recieved.

Lines 27-43 define the consumer thread function, consumer. This code enters an infinite loop and removes data from dataQueue and prints the String to the console. Line 36 is the critical piece of code in the consumer function. The call to get() on dataQueue removes the item at the front of the queue and stores it in the variable data. If dataQueue is empty, the consumer thread is blocked until data becomes available.

Alternatively, we could pass false to the optional block parameter on get(). That would cause the thread to continue to execute even if the queue is empty. However, we need to be prepared for situations where the queue is empty and catch the queue.Empty exception that is thrown. Our program calls pass to skip over the exception should this happen (it shouldn’t be the way, because we are using the blocking version of get()).

Lines 48-49 create our producer threads and start them. Lines 52-53 create and start the consumer threads. The producer threads call the produce function while the consumer threads call the consume function. The dataQueue object does the job of synchronizing data between threads. The produce threads write to dataQueue and consumer threads read from it. Thus, our program has created the consumer / producer pattern.


Lutz, Mark. Programming Python. Beijing, OReilly, 2013.

Python _thread Mutex

When Python programs create threads, each thread shares the same global memory as every other thread. Usually, but not always, multiple threads can safely read from shared resources without issue. Threads writing to shared resources are a different story because one thread could potentially overwrite the work of another thread.

This post demonstrates an example program shown in Programming Python: Powerful Object-Oriented Programming where threads acquire and release locks in the program. The locking mechanism ensures that only one thread has access to a shared resource at a time.


Here is an example program with my own comments added.

import _thread as thread, time

# This mutex object is created by calling
# thread.allocate_lock()
# The mutex is responsible for synchronizing threads
mutex = thread.allocate_lock()

def counter(tid, count):
    for i in range(count):
        # The standard out is a shared resource
        # Unless the program controls access to the standard out
        # multiple threads can print to standard out at the same time
        # which results in garbage output
        # Acquire a lock
        # Now only the current thread can print to the console
        print('[{}] => {}'.format(tid, i))
        # Make sure to release the lock for other threads when finished

if __name__ == '__main__':
    for i in range(5):
        thread.start_new_thread(counter, (i, 5))

    print('Main thread exiting...')


The program creates five threads, each of which needs access to the standard output stream. The standard output stream is a global object that all of the threads share, which means that each thread can call print at the same time. That isn’t ideal because we can get garbage output printed to the console if two threads call the print() statement at the same time.

The solution is to lock access to the standard output stream so that only one thread may use it at a time. We do this by creating a mutex object on line 6 in the program by using thread.allocate_lock(). When a thread needs a lock, it calls acquire() on the mutex. At that point, all other threads that need protected resources have to sit and wait for mutex.release().

It’s important to keep the operations between mutex.acquire() and mutex.release() as brief as possible. Only one thread can hold a lock at a time, so the longer one thread holds a lock, the longer other threads need to wait for their turn to use the lock. That naturally impacts the performance of the overall program.


Lutz, Mark. Programming Python. Beijing, OReilly, 2013.

Python Threading

The Python threading module provides an OOP solution to threading. The base class, threading.Thread, follows a Java like pattern to creating and joining threads. This post provides a threading demonstration with an example program found in Programming Python: Powerful Object-Oriented Programming.


This is the example program with my own comments added.

import threading

# Thread is the base class for creating OOP Style Threads
# It has a run() method that contains the code that runs in a new thread
class MyThread(threading.Thread):
    def __init__(self, myId, count, mutex):
        self.myId = myId
        self.count = count
        self.mutex = mutex

    # Everything inside of run is executed in a seperate thread
    def run(self):
        for i in range(self.count):
            with self.mutex:
                print('[{}] => {}'.format(self.myId, i))

if __name__ == '__main__':
    stdoutmutex = threading.Lock()
    threads = []
    for i in range(10):
        # Create the new Thread Object
        thread = MyThread(i, 100, stdoutmutex)
        # The thread doesn't actually start running until
        # start() is called

    for thread in threads:
        # join() is used to synchronize threads
        # Calling join() on a thread makes the parent thread wait
        # until the child thread has finished

    print('Main thread exiting...')


The OOP approach to Python threads requires developers to extend the threading.Thread class. The Thread class provides high level methods that support threading such as start() and join() which we discuss shortly. It has also an empty run() method that developers need to override. All of the code placed in the run() method runs in a new thread.

We are still free to use locks with OOP threads. On line 20, we acquire a lock by calling threading.Lock(). The mutex is passed to the thread object’s constructor on line 24 and is used by the thread on line 15 to aquire a lock to that standard output stream.

It’s important to note that the new thread doesn’t actually run until we call start(). The start() method is what submits the thread to the thread pool so that the Python runtime can use the thread. Never call run() directly on thread object because doing so will keep the program single threaded. The run() method is called by the Python environment when it’s the thread’s turn to run.

The example program also uses the join() method. The join() method is used to make a parent thread wait until a child thread completes. The example program creates 10 threads and needs to wait until all of the threads are finished. This is done by entering a for-each loop on line 31 and then calling join() on each of the threads. When join() is called, the parent thread sleeps until child thread’s run() method is finished. When all 10 threads are finished, the program exits.


Lutz, Mark. Programming Python. Beijing, OReilly, 2013.

Python _thread Basic

Python 3 has the newer thread package, but the _thread package still exists for developers who are more comfortable with the 2.x API. This is a basic example derived from Programming Python: Powerful Object-Oriented Programming that demonstrates how to create threads using the _thread module.


import _thread as thread, time

# This function will run in a new thread
def counter(tid, count):
    for i in range(count):
        # Simulate a delay
        # Print out the thread id (tid) and the current iteration
        # of our for loop
        print('[{}] => {}'.format(tid, i))

if __name__ == '__main__':
    # Enter a loop that creates 5 threads
    for i in range(5):
        # Start a new thread passing a callable and it's arguments
        # in the form of a tuple
        thread.start_new_thread(counter, (i, 5))

    print('Main thread exiting')


This script creates five new threads using _thread.start_new_thread. Each time a new thread is created, the counter function is called and is passed a tuple of (i, 5). That tuple corresponds to the tid and count parameters of the counter function. Counter enters a loop that runs 5 times since 5 was passed to the second parameter of counter on line 19. It will print the thread id and current iteration of the loop.

Meanwhile, the for loop in the parent thread continues to iterate because thread.start_new_thread does not block the for loop in the main thread. By calling start_new_thread, the program’s execution runs both the for loop in the main thread and the counter function in parallel. Allowing programs to run multiple portions of code at the same time is what gives threads their power. For example, you may wish to use a thread to handle a long running database query while the user continues to interact with the program in the main thread.

One final note about threads in Python. Threads give the appearance of allowing programs to multitask and for all intents and purposes, that is what is happening in the program. Nevertheless, what is really happening is that the Python Virtual Machine is time slicing the computer instructions and allowing a few lines of code to run before switching to another set of instructions.

In other words, if a program has three threads, A, B, and C, then Thread A runs for a few moments, then Thread B, and finally Thread C. Note that there is no guarantee to the order in which threads run. It is possible that one thread may run more often than other threads or that the order of running threads is different each time.


Lutz, Mark. Programming Python. Beijing, OReilly, 2013.

Use Python to Launch Another Program

Many Python programs operate as wrappers to other programs. For example, there are a wide variety of Python programs that work as a GUI wrapper for CLI programs. This tutorial uses an example from Programming Python: Powerful Object-Oriented Programming to demonstrate how a Python program can be used to launch another program.


Parent Process

This code represents a parent process that is capable of launching a child process. I added my own comments to help further explain the code.

import os

# Track the number of spawned children
spawned_children = 0

# Enter an infinite loop
while True:
    # Increment the number of children
    spawned_children += 1
    # Now launch a new process
    # If we are still in the parent process, pid is non-zero
    # otherwise, pid is zero and indicates that we are in the
    # child process
    pid = os.fork()
    # Test if we are in the child process
    if pid == 0:
        # Now, since we are in the new process, we use
        # os.execlp to launch a child script.
        os.execlp('python', 'python', '', str(spawned_children))
        assert False, 'error starting program'
        # If we are here, then we are still in the parent process
        print('Child is', pid)
        if input() == 'q': break

Child Process

This is a simple script that simply represents a sample child process.

import os, sys

# Parent passes the child process number
# as a command line argument
print('Hello from child', os.getpid(), sys.argv[1])


We have two core concepts used in this program. The first concept is forking the parent process into a child process. The other concept is actually launching a new program.


Forking is the process of launching a copy of the running program. The program that launched a copy of itself is called the parent process, while the copy of the program is called the child process. In theory, a parent and its children can spawn as many children as the underlying operating system allows.

The example program spawns a child process on line 16 and assigns the result to the variable called pid. When os.fork() returns, pid is zero if we are running in the child process or non-zero if we are still in the parent. The program uses an if – else statement (lines 19 and 25 respectively) to perform different operations depending if we are in the parent or child process.

The parent process executes lines 27 and 28. It simply prints out the child’s process id and then asks the user if they wish to continue the program. The user can enter ‘q’ to exit the parent process.

The child process executes code on lines 23 and 24. Line 23 uses os.execlp to start the companion child script and passes the number of spawned children as a command line argument. The child script prints its pid and the number of child processes and then exits.

Launching Programs

The Python os module provides a variety of ways to launch other programs via a Python script. This example makes use of os.execlp which looks for an executable on the operating system’s path. The first argument is the name of the program followed by any number of command line arguments.

The os.execlp is not supposed to return to the caller because if successful, the program is replaced by the new program. This is why it’s necessary to call os.fork() prior to using os.execlp (unless you wish your program to simply end with the new process). When the example script calls os.execlp, the child process created by os.fork is replaced by the new program launched by os.execlp.

Line 24 handles the case when os.execlp fails. Should os.execlp fail, control is returned back to the caller. In most cases, a develop should notify the user that launching the new program has failed. The example script accomplishes this by using an assert False statement with an error message.


Lutz, Mark. Programming Python. Beijing, OReilly, 2013.

Python Basic Forking

Many programs need to execute tasks simultaneously and Python provides us with a few different mechanisms for concurrent programming. One of those mechanisms is called forking, where a call is made to the underlying operating system to create a working copy of a program that’s already running. The program that created the new process is called the parent process, while the processes that are created by the parent are called the child process.

This post shows the most basic form of creating processes in Python and helps serve as a foundation to understanding forking. The example is derived from Programming Python: Powerful Object-Oriented Programming, and I added my own comments to help better explain the program.

import os

# This is a function called by the child process
def child():
    # Use os.getpid() to get the pid for this process
    print('Hello from child', os.getpid())

    # force the child process to exit right away
    # or the child process will return to the infinite loop in parent()

def parent():
    while True:
        # Attempt to fork this program into a new process. When forking is complete
        # newpid will be non-zero for the original process, but it wil be
        # 0 in the child process
        newpid = os.fork()

        # The program now goes in two different directions at the same time
        # When newpid is 0, we call child() and the child process exits
        if newpid == 0:  # Test if this is a child process
            # If are here, then we are still in the parent process
            # We print the pid of the parent and the child process (newpid)
            print('Hello from parent', os.getpid(), newpid)
        if input() == 'q':

if __name__ == '__main__':

When run on my machine, the program shows the following output

Hello from parent 87800 87802
Hello from child 87802
Hello from parent 87800 87803
Hello from child 87803
Hello from parent 87800 87804
Hello from child 87804
Hello from parent 87800 87805
Hello from child 87805


The hardest part to grasp about this program is that when os.fork() is called on line 19, the program actually launches a copy of itself. The operating system creates the new process and that new process gets a copy of all variables in memory and execution of the new and old programs continue after line 19. (Note: The OS may not exactly copy the parent process, but functionally speaking, the child process can be considered to be a copy of the parent).

The os.fork() function returns a number called a PID (process ID). We can test the pid to see if we are running in the parent or child process. When we are in the child process, the value returned by os.fork() is zero. So on line 23, we test for 0 and if newpid is zero, we call the child() function.

The alternative case is that we are still running in the parent process (bearing in mind, that the child process is also running at this point in time as well). If we are still in the parent process os.fork() returns a non-zero value. In that case, we use the else block to print the parent and child PID.

The parent process continues to loop until the user enters q to quit. Each time the loop iterates, a new child process is created by the parent. The parent prints its own PID (using os.getpid()) and the pid of the child on line 28.

The child process also uses os.getpid() to get its own PID. It prints its own PID on line 7 and then on line 11, we use os._exit(0) to force the child process to shut down. This is a critical step for this program! If we were to omit the call to os._exit on line 11, the child process would return to the parent function and enter the same infinite loop the parent is using.


This is the most basic example of creating child processes using Python. Keep in mind that processes do not share memory (unlike threads). In real world programs, processes often need to sycnchronize data from one process to another process using tools such as network sockets, databases, or files. When a child process is spawned it gets a copy of the memory of the parent process, but then functions as an independent program.


Lutz, Mark. Programming Python. Beijing, OReilly, 2013