Changing systemd services on startup
Systemd is fast becoming the standard for PID 1 on Linux systems, replacing the ancient init script approach. It takes a bit of time to wrap your head around its concurrent and implicit ordering behaviour, but it does bring some very welcome improvements, with init.d files replaced with much simpler, declarative service definition files, along with integrated logging, eventing and automatic restarting among other things.
One of the things you can't do any more is start services from a script during startup because systemd evaluates the service dependency tree on launch and starts services accordingly, so modifications made to it afterwards are ignored until systemd is reloaded, which you can't do while it is launching. This broke the ability for my fixed-image servers to change which server-specific services ran on boot (the servers boot from a common image, then a pre-configured service on startup mounts and applies an overlay containing configuration files and commands specific to it). As far as I know there are no hooks in systemd to allow modification before the dependency tree is determined, which is fair enough given my use case definitely isn't a common one.
Without any hooks, in order for me to get my server-specific services started, I needed to reload systemd after the current run had completed. Luckily, the systemd on my server image has DBus support, meaning systemd will broadcast events during its execution. I wrote up a Python script that is executed by a custom service scheduled early on by systemd (Requires=local-fs-pre.target, After=local-fs-pre.target, Before=basic.target). That script hooks into DBus and waits for systemd's StartupFinished message. When it receives it, it tells systemd to reload, then starts the default target again, causing the service dependency tree to be evaluated and executed once more, this time with the added services. Systemd is smart enough to know which services are already running and won't reload them, and can also stop services that have since been disabled.
Systemd service definition
[Unit] Description=Local overlay loader DefaultDependencies=false Requires=local-fs-pre.target After=local-fs-pre.target Before=basic.target [Service] Type=oneshot ExecStart=/root/loadoverlay.sh RemainAfterExit=true [Install] WantedBy=local-fs.target
The /root/loadoverlay.sh script mounts the overlay, executes the overlay script and finally runs the following Python script in the background (i.e. with &). The Python DBus bindings need to be installed for this to work.
Note that because of the RemainAfterExit=True parameter, this service will not be re-executed when systemd is reloaded, because that flag tells systemd that this service is still considered to be running even though the executed process isn't.
DBus StartupFinished hook script
#!/usr/bin/env python ## # Adds a signal handler to the 'startup finished' signal on systemd. # # We then re-execute the default systemd target so it activates the systemd # changes made during the overlay script. ## import dbus import gobject from dbus.mainloop.glib import DBusGMainLoop import logging import subprocess logging.basicConfig( filename='/var/log/dbus_startup_finished_hook.log', level=logging.DEBUG, format='%(asctime)s [%(name)s][%(levelname)s] %(message)s' ) def startup_finished_handler(firmware, loader, kernel, initrd, userspace, total): try: logging.info('systemd "startup finished" signal detected. Re-activating default.target...') logging.info('calling systemctl daemon-reload...') p = subprocess.Popen(['/usr/bin/systemctl', 'daemon-reload'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) stdout, stderr = p.communicate() logging.info('systemctl daemon-reload completed.') if stdout: logging.info('systemctl daemon-reload stdout:\n' + stdout) if stderr: logging.warning('systemctl daemon-reload stderr:\n' + stderr) logging.info('calling systemctl start default.target...') p = subprocess.Popen(['/usr/bin/systemctl', 'start', 'default.target'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) stdout, stderr = p.communicate() logging.info('systemctl start default.target completed.') if stdout: logging.info('systemctl start default.target stdout:\n' + stdout) if stderr: logging.warning('systemctl start default.target stderr:\n' + stderr) # handler completed; quit now logging.info('handler finished, quitting event loop.') loop.quit() except Exception as e: logging.exception('Exception occurred in startup_finished handler.', e) # hook up the handler dbus.mainloop.glib.DBusGMainLoop(set_as_default=True) bus = dbus.SystemBus() bus.add_signal_receiver( startup_finished_handler, dbus_interface='org.freedesktop.systemd1.Manager', signal_name='StartupFinished' ) # start the event loop logging.info('Starting the DBus event loop...') loop = gobject.MainLoop() loop.run()