Skip to content

run a dedicated supervisor/reaper for each job #85

Description

@mheily

Each job should have a dedicated supervisor/subreaper process. In practice, this means that jobd calls fork() one more time, and makes the parent process the supervisor. This should look like:

jobd
  ||
supervisor/reaper
  ||
actual job process, defined by Program in the manifest

The subreaper function depends on calling procctl(2) or prctl(2) to enable subreaping. Not all kernels support this, so it is allowed to not have this function. The effect of not having a subreaper is that jobs might not be fully cleaned up if they spawn additional daemons.

The supervisor is actually a bit of a misnomer when compared to things like runit/s6/daemontools. The jobd supervisor will not actually keep the job alive by restarting it when it dies. Instead, it is very dumb process that will just sit in a wait() call on it's child process. If the child dies, the supervisor will hang around long enough to kill all the stray children, update the job database to reflect that the job is dead, and then exit.

If you send a SIGTERM to the supervisor, it will send a SIGTERM to the child and then wait around for all grandchildren to exit. After a certain amount of time, it will forcibly kill all grandchildren.

So in effect, the supervisor/subreaper is really just a layer between jobd and it's running jobs, with the purpose of handling reaping and process monitoring, doing so independently of jobd itself. Jobd will only be notifed when a job and all it's related processes are fully terminated. This simplifies jobd's internal state machine and child termination handling, and should make it easier for jobd itself to crash or be restarted without losing track of jobs that are in the middle of stopping.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions