• Changing systemd services on startup

    Systemd is fast becoming the standard for PID 1 on Linux systems, replacing the ancient init script approach. It takes a bit of time to wrap your head around its concurrent and implicit ordering behaviour, but it does bring some very welcome improvements, with init.d files replaced with much simpler, declarative service definition files, along with integrated logging, eventing and automatic restarting among other things.

    One of the things you can't do any more is start services from a script during startup because systemd evaluates the service dependency tree on launch and starts services accordingly, so modifications made to it afterwards are ignored until systemd is reloaded, which you can't do while it is launching. This broke the ability for my fixed-image servers to change which server-specific services ran on boot (the servers boot from a common image, then a pre-configured service on startup mounts and applies an overlay containing configuration files and commands specific to it). As far as I know there are no hooks in systemd to allow modification before the dependency tree is determined, which is fair enough given my use case definitely isn't a common one.

    Without any hooks, in order for me to get my server-specific services started, I needed to reload systemd after the current run had completed. Luckily, the systemd on my server image has DBus support, meaning systemd will broadcast events during its execution. I wrote up a Python script that is executed by a custom service scheduled early on by systemd (,, That script hooks into DBus and waits for systemd's StartupFinished message. When it receives it, it tells systemd to reload, then starts the default target again, causing the service dependency tree to be evaluated and executed once more, this time with the added services. Systemd is smart enough to know which services are already running and won't reload them, and can also stop services that have since been disabled.

    Systemd service definition

    Description=Local overlay loader

    The /root/ script mounts the overlay, executes the overlay script and finally runs the following Python script in the background (i.e. with &). The Python DBus bindings need to be installed for this to work.

    Note that because of the RemainAfterExit=True parameter, this service will not be re-executed when systemd is reloaded, because that flag tells systemd that this service is still considered to be running even though the executed process isn't.

    DBus StartupFinished hook script

    #!/usr/bin/env python
    # Adds a signal handler to the 'startup finished' signal on systemd.
    # We then re-execute the default systemd target so it activates the systemd
    # changes made during the overlay script.
    import dbus
    import gobject
    from dbus.mainloop.glib import DBusGMainLoop
    import logging
    import subprocess
        format='%(asctime)s [%(name)s][%(levelname)s] %(message)s'
    def startup_finished_handler(firmware, loader, kernel, initrd, userspace, total):
  'systemd "startup finished" signal detected. Re-activating')
  'calling systemctl daemon-reload...')
            p = subprocess.Popen(['/usr/bin/systemctl', 'daemon-reload'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            stdout, stderr = p.communicate()
  'systemctl daemon-reload completed.')
            if stdout:
      'systemctl daemon-reload stdout:\n' + stdout)
            if stderr:
                logging.warning('systemctl daemon-reload stderr:\n' + stderr)
  'calling systemctl start')
            p = subprocess.Popen(['/usr/bin/systemctl', 'start', ''], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            stdout, stderr = p.communicate()
  'systemctl start completed.')
            if stdout:
      'systemctl start stdout:\n' + stdout)
            if stderr:
                logging.warning('systemctl start stderr:\n' + stderr)
            # handler completed; quit now
  'handler finished, quitting event loop.')
        except Exception as e:
            logging.exception('Exception occurred in startup_finished handler.', e)
    # hook up the handler
    bus = dbus.SystemBus()
    # start the event loop'Starting the DBus event loop...')
    loop = gobject.MainLoop()


  • Big tech company problems

  • Declutterising

  • Being funemployed

  • Run SuSE Studio images without persistence for OpenSuSE 12.3