Finally, TIL, what can all be the reason for systemd services to hang indefinitely. The internet is flooded with numerous reports on this topic but no clear answers. So no more uselessly marked workarounds like:
systemctl daemon-reload and
systemctl-daemon-reexec for this scenario.
The scene would be something along the lines of:
rrs 6467 0.0 0.0 23088 15852 pts/1 Ss 12:53 0:00 | | \_ /bin/bash rrs 11512 0.0 0.0 14876 4608 pts/1 S+ 13:18 0:00 | | | \_ systemctl restart snapper-timeline.timer rrs 11513 0.0 0.0 14984 3076 pts/1 S+ 13:18 0:00 | | | \_ /bin/systemd-tty-ask-password-agent --watch rrs 11514 0.0 0.0 234756 6752 pts/1 Sl+ 13:18 0:00 | | | \_ /usr/bin/pkttyagent --notify-fd 5 --fallback
snapper-timeline service is important to me and it not running for months is a complete failure. Disappointingly, commands like
systemctl --failed do not report of this oddity. The overall system status is reported to be fine, which is completely incorrect.
Thankfully, a kind soul’s comment gave the hint. The problem is that you could be having certain services in
Activating status, which thus blocks all other services; quietly. So much for the unnecessary fun.
Looking further, in my case, it was:
rrs@priyasi:~$ systemctl list-jobs JOB UNIT TYPE STATE 81 timers.target start waiting 85 man-db.timer start waiting 88 fstrim.timer start waiting 3832 snapper-timeline.service start waiting 83 snapper-timeline.timer start waiting 39 systemd-time-wait-sync.service start running 87 logrotate.timer start waiting 84 debspawn-clear-caches.timer start waiting 89 plocate-updatedb.timer start waiting 91 dpkg-db-backup.timer start waiting 93 e2scrub_all.timer start waiting 40 time-sync.target start waiting 86 apt-listbugs.timer start waiting 13 jobs listed. 13:12 ♒ ॐ ♅ ♄ ⛢ ☺ 😄
That was it. I knew the
systemd-timesyncd service, in the past, had given me enough headaches. And so was it this time, just quietly doing it all again.
rrs@priyasi:~$ systemctl status systemd-time-wait-sync.service ● systemd-time-wait-sync.service - Wait Until Kernel Time Synchronized Loaded: loaded (/lib/systemd/system/systemd-time-wait-sync.service; enabled; vendor preset> Active: activating (start) since Fri 2022-04-22 13:14:25 IST; 1min 38s ago Docs: man:systemd-time-wait-sync.service(8) Main PID: 11090 (systemd-time-wa) Tasks: 1 (limit: 37051) Memory: 836.0K CPU: 7ms CGroup: /system.slice/systemd-time-wait-sync.service └─11090 /lib/systemd/systemd-time-wait-sync Apr 22 13:14:25 priyasi systemd: Starting Wait Until Kernel Time Synchronized... Apr 22 13:14:25 priyasi systemd-time-wait-sync: adjtime state 5 status 40 time Fri 2022-> 13:16 ♒ ॐ ♅ ♄ ⛢ ☹ 😟=> 3
Dear LazyWeb, anybody knows of why the
systemd-time-wait-sync service would hang indefinitely? I’ve had identical setups on many machines, in the same network, where others don’t exhibit this problem.
rrs@priyasi:~$ systemctl cat systemd-time-wait-sync.service ...snipped... [Service] Type=oneshot ExecStart=/lib/systemd/systemd-time-wait-sync TimeoutStartSec=infinity RemainAfterExit=yes [Install] WantedBy=sysinit.target
TimeoutStartSec=infinity is definitely an attribute that shouldn’t be shipped in any system services. There are use cases for it but that should be left for local admins to explicitly decide. Hanging for
infinity is not a desired behavior for a system service.
In figuring all this out, today I learnt the handy
systemctl list-jobs command, which will give the list of active