VolatileThunk

In this era of threads and asynchronous abstractions, applications and processes have become almost synonymous. A process is widely seen as the operating system's underlying representation of a whole running application. However, by limiting ourselves to this model we cut outselves off from an elegant set of tools for parallelism and concurrency.

In case you thought this blog's design looked prehistoric enough, it's starting with a post about following concurrency patterns rooted in the era in which the mouse was a keen invention.

The key construct behind process-based Unix concurrency is the fork system call. It's practically a paradox: one program calls it yet two programs finish calling it to move on. This appears as quite the oddity to programmers of contemporary environments like Java and the JavaScript; how can a mere function call violate the fundamental laws of how a written program executes?

Most environments, despite having a set of rules its programmers can depend on, have occasional strange artefacts on the surface that violate such rules and are exposed by the underlying system.

A Java program holding a reference to an object ensures that object remains alive, but the system exposes a different type of reference tracking with the java.lang.ref.WeakReference type insofar as allowing an object to be deleted while the programmer is still holding onto it. Likewise, storing a string literal into a object will never block execution in JavaScript, but there's an exception that exposes the underlying browser environment: window.location. Assigning a string to that cancels the current flow of execution by redirecting the page, halting the current JavaScript environment and throwing it away.

In C, or even other higher-level languages like Python or Ruby, the environment is not a language-specific world unto itself like Java or a web browser; it is instead our underlying operating system. Python and Ruby build their own abstractions on top, but they are not as aggressive about hiding the underlying operating system as programming environments like Java and in-browser JavaScript.

Apart from your operating system, languages like Python and Ruby provide extra gizmos through their runtimes, such as stack-unwinding, garbage collection, and some introspection capabilities. System calls from these languages, like the previous examples, provide capabilities that can step outside of the normal rules of the language you are using. Unix doesn't care about the finally blocks that your language runtime makes "guarantees" about running; if you exec, the whole process' runtime image is being swapped out and execution is jumping. Those finally blocks will be deep sixed.

For the demonstrations in this article, Python 3 is used. All the examples use only its standard library and can be run with python3 program.py.

Invoking the fork system call in Python is straightforward:

import os

if os.fork() == 0:
    print("running the child process")
else:
    print("running the parent process")

print("finished a process")

fork returns 0 in the newly created child process and the child's process ID in the parent process.

This program demonstrates that oxymoron of one program entering fork and two leaving it; how else could both branches be run in a single if? This allows for a form of concurrency and parallelism:

$ python3 program.py
running the parent process
finished a process
running the child process
finished a process

Running operations side-by-side is hardly an elusive trick. Anyone spinning up a thread in Java or interleaving two setTimeouts in JavaScript could replicate something similar. Unlike the latter, however, the application can still continue while one of the tasks infinitely loops:

import os

if os.fork() == 0:
    while True:
        pass
else:
    print("finished")

$ python3 program.py
finished

These clearly aren't events being dispatched into an event loop behind the scenes, otherwise the infinite loop would block the event loop, stopping the second event print("finished") from running. Single events blocking the entire event loop and grinding large applications to a halt is not a theoretical problem.

Threads sidestep this problem by utilising the operating system's scheduler which slices up time on the computer between competing tasks to avoid any one of them starving the whole system of resources, but they come with their own arsenel of footguns in many shapes and sizes:

import os
import threading
import time


class UserCounter:
    def __init__(self):
        self.count = 0

    def increment(self):
        count = self.count
        print(f"User {count} visited")
        # Unlike += under CPython's GIL, print can yield between this stale read and write.
        self.count = count + 1


user_counter = UserCounter()


def handle_requests():
    while True:
        # Pretend to serve a user on the network.
        time.sleep(0.3)

        user_counter.increment()


for _ in range(os.cpu_count()):
    thread = threading.Thread(target=handle_requests)
    thread.start()

An operation is divided and conquered, handling user requests across as many threads as the machine has processor cores. Unfortunately, user_counter is shared across all threads. Loading and incrementing its value are separate operations which can be interleaved with other concurrent threads, causing the counter to be wrong.

$ python3 program.py
User 0 visited
User 0 visited
User 1 visited
User 1 visited
User 1 visited

We must remember quite a few rules to avoid shooting ourselves in the foot in multithreaded systems. Just a few.

Between dealing with data races, deadlocks, stale reads, and other threading esoteria, programming with threads is playing Russian Roulette with a fully loaded uzi. An uzi that jams a lot too, as adding too many critical regions to synchronise threaded code bottlenecks your otherwise concurrent program into single-threaded hotspots that can end up throttling your application's performance.

Have you managed to get the locking fine-grained enough to get good performance while avoiding those pitfalls? Well, hopefully you're not building any more abstractions on top of it, as locks do not compose.

Operating system processes are bulky and cannot be spun up as fast as threads, but are more isolated from one another. They have their own memory space, meaning one buggy process can't corrupt the in-memory data of another. Unlike threads, misbehaving processes can actually be killed without causing strange, hard to trace bugs in the underlying system.

Processes also encourage communication via message-passing mechanisms like signalling, domain sockets, and networking connections. It turns out that these solutions are easier to scale across multiple physical machines than shared memory communication, as others also discovered quite a long time ago.

Signalling is the easiest way to get your feet wet with Unix IPC, Inter-Process Communication:

from os import _exit, fork, getppid, kill, waitpid
from signal import SIGCONT, sigwait
from time import sleep


def expensive_operation_1():
    sleep(5)
    kill(getppid(), SIGCONT)
    sigwait((SIGCONT,))
    print("Very expensive operation 1 complete.")


def expensive_operation_2():
    sleep(2)
    kill(getppid(), SIGCONT)
    sigwait((SIGCONT,))
    print("Slightly expensive operation 2 complete.")


def start_children():
    pid_1 = fork()
    if pid_1 == 0:
        expensive_operation_1()
        _exit(0)

    pid_2 = fork()
    if pid_2 == 0:
        expensive_operation_2()
        _exit(0)

    return pid_1, pid_2


def wait_for_all(pids):
    for _ in pids:
        sigwait((SIGCONT,))


def display_in_order(pids):
    for pid in pids:
        kill(pid, SIGCONT)
        waitpid(pid, 0)


pids = start_children()
print("All children started.")
wait_for_all(pids)
print("All children finished main tasks; asking them to display results in order.")
display_in_order(pids)

This is a more complex example, so let's go through each component piece-by-piece.

The two operations take different amounts of time. Once each completes its main task, it sends SIGCONT to its parent and waits for the same signal in return. start_children forks a process for each operation, while wait_for_all waits for one completion signal per process. Finally, display_in_order sends SIGCONT to each child in definition order and waits for that child to exit.

Running this yields:

$ python3 program.py
All children started.
All children finished main tasks; asking them to display results in order.
Very expensive operation 1 complete.
Slightly expensive operation 2 complete.

The slightly expensive operation, despite being quicker, displays its output after the longer running one. Both of them ran at the same time; it waited for 5 seconds, not 7. To recap, combining fork with Unix process signalling, the following was organised:

• Multiple tasks are forked in seperate processes. This not only allows IO interleaving like event systems, but also utilisation of multiple processor cores if we assumed the mock sleeps are actually computationally expensive operations.
• The parent process waits for a signal, specifically SIGCONT, to indicate a child process finished its main task. It waits for the same amount of signals as child processes. This means it does not move on until they all declare having finished.
• It iterates over the processes in the order they were defined, sends a message to each to display their results, and waits for them to complete.

The whole process not only parallelises the compution, but it linearises the results. Notice the lack of locks, shared queues, and polling.

Running man 7 signal on a Unix device tells us what signals exist. Choosing SIGCONT was an arbritary decision, as most of the signals here could have been used. SIGCONT just so happens to best describe what it was doing: continuing after the tasks had finished waiting for something else.

This relevant section of Linux's manpage for signal is illustrative:

Signal      Standard   Action   Comment
────────────────────────────────────────────────────────────────────────
SIGABRT      P1990      Core    Abort signal from abort(3)
SIGALRM      P1990      Term    Timer signal from alarm(2)
SIGBUS       P2001      Core    Bus error (bad memory access)
SIGCHLD      P1990      Ign     Child stopped or terminated
SIGCLD         -        Ign     A synonym for SIGCHLD
SIGCONT      P1990      Cont    Continue if stopped
SIGEMT         -        Term    Emulator trap
SIGFPE       P1990      Core    Floating-point exception
SIGHUP       P1990      Term    Hangup detected on controlling terminal
                                or death of controlling process
SIGILL       P1990      Core    Illegal Instruction
SIGINFO        -                A synonym for SIGPWR
SIGINT       P1990      Term    Interrupt from keyboard
SIGIO          -        Term    I/O now possible (4.2BSD)
SIGIOT         -        Core    IOT trap. A synonym for SIGABRT
SIGKILL      P1990      Term    Kill signal
SIGLOST        -        Term    File lock lost (unused)
SIGPIPE      P1990      Term    Broken pipe: write to pipe with no
                                readers; see pipe(7)
SIGPOLL      P2001      Term    Pollable event (Sys V);
                                synonym for SIGIO
SIGPROF      P2001      Term    Profiling timer expired
SIGPWR         -        Term    Power failure (System V)
SIGQUIT      P1990      Core    Quit from keyboard
SIGSEGV      P1990      Core    Invalid memory reference
SIGSTKFLT      -        Term    Stack fault on coprocessor (unused)
SIGSTOP      P1990      Stop    Stop process
SIGTSTP      P1990      Stop    Stop typed at terminal
SIGSYS       P2001      Core    Bad system call (SVr4);
                                see also seccomp(2)
SIGTERM      P1990      Term    Termination signal
SIGTRAP      P2001      Core    Trace/breakpoint trap
SIGTTIN      P1990      Stop    Terminal input for background process
SIGTTOU      P1990      Stop    Terminal output for background process
SIGUNUSED      -        Core    Synonymous with SIGSYS
SIGURG       P2001      Ign     Urgent condition on socket (4.2BSD)
SIGUSR1      P1990      Term    User-defined signal 1
SIGUSR2      P1990      Term    User-defined signal 2
SIGVTALRM    P2001      Term    Virtual alarm clock (4.2BSD)
SIGXCPU      P2001      Core    CPU time limit exceeded (4.2BSD);
                                see setrlimit(2)

SIGXFSZ      P2001      Core    File size limit exceeded (4.2BSD);
                                see setrlimit(2)
SIGWINCH       -        Ign     Window resize signal (4.3BSD, Sun)

The signals SIGKILL and SIGSTOP cannot be caught, blocked, or ignored.

Up to and including Linux 2.2, the default behavior for SIGSYS, SIGXCPU, SIGXFSZ, and (on architectures other than SPARC and MIPS) SIGBUS was to terminate
the process (without a core dump).  (On some other UNIX systems the default action for SIGXCPU and SIGXFSZ is to terminate  the  process  without  a  core
dump.)  Linux 2.4 conforms to the POSIX.1-2001 requirements for these signals, terminating the process with a core dump.

SIGEMT  is  not  specified  in  POSIX.1-2001,  but nevertheless appears on most other UNIX systems, where its default action is typically to terminate the
process with a core dump.

SIGPWR (which is not specified in POSIX.1-2001) is typically ignored by default on those other UNIX systems where it appears.

SIGIO (which is not specified in POSIX.1-2001) is ignored by default on several other UNIX systems.

Some of those signals have special behaviour, like being impossible to handle such as SIGKILL, or being handled by some language runtimes for us, like Python translating SIGINT into a KeyboardInterrupt exception. SIGUSR1 and SIGUSR2 are good for non-standard signals for application-specific events.

Some notes about the previous code before moving on:

• _exit stops a process without the usual cleanup. Only the parent needs to clean up normally; each child performs its allocated task and immediately stops.
• Despite its name, kill can send any signal to a process. Passing SIGCONT turns it into a general signalling mechanism rather than terminating the recipient.
• sigwait((SIGCONT, )) uses Python's syntax for a single-item tuple. The comma disambiguates it from a grouped expression.
• wait_for_all does nothing with each process it iterates over. Iterating once per process is what makes it wait for the expected number of signals.

We might want to wait for a signal, but not if it takes too long:

from os import _exit, fork, kill
from signal import (
    SIGALRM,
    SIGCHLD,
    SIGKILL,
    SIG_BLOCK,
    alarm,
    pthread_sigmask,
    sigwait,
)
from time import sleep

signals = (SIGCHLD, SIGALRM)
pthread_sigmask(SIG_BLOCK, signals)


def set_five_second_alarm():
    alarm(5)


def remove_alarm():
    alarm(0)


def run_slow_operation():
    sleep(10)
    print("finished slow operation")
    _exit(0)


def wait_for(pid):
    set_five_second_alarm()
    try:
        if sigwait(signals) == SIGALRM:
            print("too late; killing child process")
            kill(pid, SIGKILL)
        else:
            print("child process finished")
    finally:
        remove_alarm()


pid = fork()
if pid == 0:
    run_slow_operation()
else:
    wait_for(pid)

$ python3 program.py
too late; killing child process

Alarms allow timers to be interleaved with events. 0 disables any active alarm, so the finally block guarantees that the alarm is removed even if an error occurs. SIGCHLD indicates that a child process has stopped running.

pthread_sigmask blocks SIGCHLD and SIGALRM before the process forks. This allows sigwait to consume either signal synchronously rather than racing an asynchronous signal handler.

If we wanted, we could put all of our event handling logic directly in signal handlers rather than waiting with sigwait. However, handling them inline like that opens us up to the gnarly world of "asynchronous signal-safe behaviour". It's better to do the bare minimum in the handler, just dispatching an event that can be handled by normal code in the usual execution flow. A lot of libraries will dispatch signals as events via a wrapper to avoid this exact footgun.

Be careful of default signal behaviour. Signal masks are inherited across fork, so a parent process can pass its mask to its children. If a signal must be picked up with sigwait, add it to the current process's signal mask first. Remember that threads must also be considered.

Signalling is one of the simplest forms of Unix IPC. It's enough to coordinate processes, but does not allow sending messages with payloads. Domain sockets, networking connections, and other IPC systems allow a programmer to go a lot further.

If Unix IPC is such a powerful and battle-tested standard for parallelisation and concurrency, why isn't it the primary port of call for solving such problems today? Well, for starters it's slow and bulky even with modern optimisations like copy-on-write for copying processes' memory spaces to forked children. Although shared memory has problems, it is sometimes the best way of solving certain problems and it's easier with threads. Event-driven systems avoid many of the problems with threads and handle the majority of concurrency use cases in modern webservices, so managing processes manually becomes unnecessary.

Many applications written in the likes of Node.js use processes just to utilize as many processor cores as possible, but hide process management behind modules like cluster. Processes are used to parallelise, but the concurrency is handled by abstractions built atop an event loop. Using a process per request would destroy performance as they are too coarse for that level of fine-grained concurrency. In fact, that's how web applications were handled many years ago in CGI scripts. There's a reason it isn't done that way anymore.

Despite these problems, Unix IPC mechanisms are sometimes still the best way of tackling certain concurrency and parallelism problems, so it's worth keeping those dusty old '70s techniques in the toolbox.