• Changing systemd services on startup

    Systemd is fast becoming the standard for PID 1 on Linux systems, replacing the ancient init script approach. It takes a bit of time to wrap your head around its concurrent and implicit ordering behaviour, but it does bring some very welcome improvements, with init.d files replaced with much simpler, declarative service definition files, along with integrated logging, eventing and automatic restarting among other things.

    One of the things you can't do any more is start services from a script during startup because systemd evaluates the service dependency tree on launch and starts services accordingly, so modifications made to it afterwards are ignored until systemd is reloaded, which you can't do while it is launching. This broke the ability for my fixed-image servers to change which server-specific services ran on boot (the servers boot from a common image, then a pre-configured service on startup mounts and applies an overlay containing configuration files and commands specific to it). As far as I know there are no hooks in systemd to allow modification before the dependency tree is determined, which is fair enough given my use case definitely isn't a common one.

    Without any hooks, in order for me to get my server-specific services started, I needed to reload systemd after the current run had completed. Luckily, the systemd on my server image has DBus support, meaning systemd will broadcast events during its execution. I wrote up a Python script that is executed by a custom service scheduled early on by systemd (,, That script hooks into DBus and waits for systemd's StartupFinished message. When it receives it, it tells systemd to reload, then starts the default target again, causing the service dependency tree to be evaluated and executed once more, this time with the added services. Systemd is smart enough to know which services are already running and won't reload them, and can also stop services that have since been disabled.

    Systemd service definition

    Description=Local overlay loader

    The /root/ script mounts the overlay, executes the overlay script and finally runs the following Python script in the background (i.e. with &). The Python DBus bindings need to be installed for this to work.

    Note that because of the RemainAfterExit=True parameter, this service will not be re-executed when systemd is reloaded, because that flag tells systemd that this service is still considered to be running even though the executed process isn't.

    DBus StartupFinished hook script

    #!/usr/bin/env python
    # Adds a signal handler to the 'startup finished' signal on systemd.
    # We then re-execute the default systemd target so it activates the systemd
    # changes made during the overlay script.
    import dbus
    import gobject
    from dbus.mainloop.glib import DBusGMainLoop
    import logging
    import subprocess
        format='%(asctime)s [%(name)s][%(levelname)s] %(message)s'
    def startup_finished_handler(firmware, loader, kernel, initrd, userspace, total):
  'systemd "startup finished" signal detected. Re-activating')
  'calling systemctl daemon-reload...')
            p = subprocess.Popen(['/usr/bin/systemctl', 'daemon-reload'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            stdout, stderr = p.communicate()
  'systemctl daemon-reload completed.')
            if stdout:
      'systemctl daemon-reload stdout:\n' + stdout)
            if stderr:
                logging.warning('systemctl daemon-reload stderr:\n' + stderr)
  'calling systemctl start')
            p = subprocess.Popen(['/usr/bin/systemctl', 'start', ''], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            stdout, stderr = p.communicate()
  'systemctl start completed.')
            if stdout:
      'systemctl start stdout:\n' + stdout)
            if stderr:
                logging.warning('systemctl start stderr:\n' + stderr)
            # handler completed; quit now
  'handler finished, quitting event loop.')
        except Exception as e:
            logging.exception('Exception occurred in startup_finished handler.', e)
    # hook up the handler
    bus = dbus.SystemBus()
    # start the event loop'Starting the DBus event loop...')
    loop = gobject.MainLoop()





That's a good one!

Need a little advice: In my project, I create a customize centos 6 os where I do create and mount partitions/lvs/fs/disks  by my script (say which is hooked up with rc.sysinit. This script creates everything like /boot, /root, swap etc.etc. partitions and filesystems on no. of disks provided.

Now, I am switiching to Centos 7 os for the same project and have to deal with systemd.

Where do you think I should hook up  my script? I tried creating a service which calls my script and has but no luck! Got error saying "Job deleted to break ordering cycle starting with systemd-update-done.service/start". I chose b coz I think thisis the stage where fs is not mounted yet. So, I wanna create and mount it by my script.

Please advise.

Thanks in advance.




Posted by Neo, 10th December 2016 9:08 AM

FYI, attempts to use this on Ubuntu 16.04.2 running systemd 229.

Out the box, fails with an error:[507]: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.FileNotFound: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

Adding 'Requires=dbus.service' and 'After=dbus.service' helps, but then I get into a deadlock where overlay.service is perpetually in state 'activating', because the Python script is waiting for the StartupFinished signal, but StartupFinished never comes because overlay.service is still loading ('systemctl is-system-running' = 'starting').

I fix this (I think) by changing Type=oneshot to Type=simple. Boot then proceeds to 'running' but the overlay service fails:

# journalctl -o cat -u overlay
Started Local overlay loader.
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/", line 861, in emit
    msg = self.format(record)
  File "/usr/lib/python2.7/logging/", line 734, in format
    return fmt.format(record)
  File "/usr/lib/python2.7/logging/", line 465, in format
    record.message = record.getMessage()
  File "/usr/lib/python2.7/logging/", line 329, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Logged from file, line 50


At this point I gave up. Hope this helps someone.

Posted by Jeff, 21st February 2017 12:57 PM

Leave a comment