Inter-Process Communication

The Problem#

I'm trying to create a complex system incorporating input, processing, and display

  • Frontend: Webpage written in PHP
  • Processing: High-Performance C/C++ (of Fortran)
  • Sensors: Python-based drivers

How do they all communicate?

Solution 1#

Just pick a single language and go with it

Solution 2#

Pick a single language and shoehorn everything else into it

  • DPI - Direct Programmer Interface
  • Library/wrappers
  • exec commands

Solution 3#

  • Interprocess communications (IPC): use mechanisms to share data between independent processes

Why is this so hard?#

All code gets converted to machine code, right? Why can't a python program directly point to a C program's memory and visa versa?

The OS will not let you do that (intentionally)

Fundamentals of IPC#

  • Requirements
    • Shared token
    • API/data format/data encoding
    • Access control mechanism

Simplest/Slowest IPC: Shared File(s)#

Applications communicate by writing/reading data to/from one or more files used for this purpose. FIles can be text or binary

  • Shared token: Filename/path
  • Access control: POSIX file locking (fcntl) may be safest, can also use custom

Fastest IPC: Shared Memory#

Multiple applications use an OS API to map a shared segment of memory. Data is passed by reading/writing to that chunk of memory

  • Shared token: API specific, used so that processes can connect to the right shared memory. Ex: integer "key"
  • Flow control (POSIX) Semaphores are safest, or can be custom

Safer IPC: Named Pipes ("FIFO")#

A named pipe, or software "FIFO" is used to pass data in strict byte-order between processes. This is implemented in memory with a "backing file" that exists to ensure some data persistence until all linked processes unlink from it

  • Shared token: backing file name
  • Flow Control: file access mechanisms (only one process should open for write)

Safer IPC: Sockets#

Sockets, like those used for communicating across a network, can also be used for local streaming communications between processes in a client/server model

  • Shared token: socket number
  • Flow control: socket binding, send/receive operations

Complication#

  • One process sends you 0x7A, what does that mean?
  • What type is the data?
    • What does that mean for weakly typed language like Python?
  • Need some pre-defined API
  • Can be as complicated or simple as necessary depending on the communication
    • Can be raw data (ints, bytes, etc.)
    • Can be more complicated messages with things like operation, data, metadata (string, struct, dict, objects, etc.)

Complication: What Type is the Data?#

  • Different languages store different data structures differently in memory

    • A python list may not be encoded the same way as a C array
    • A Python object is probably encoded differently from an equivalent C object

  • Solution: Use a language independent encoding mechanism where possible

    • JSON
    • CBOR
    • Smile
    • BSON
Edit this page
Last updated on