Cdtjkyj

The Ubuntu 16.04 server VM image apparently starts the "apt-daily.service" every
12 hours or so; this service performs various APT-related tasks like refreshing
the list of available packages, performing unattended upgrades if needed, etc.

When starting from a VM "snapshot", the service is triggered immediately, as (I
presume) systemd realizes quickly that the timer should have gone off long ago.

However, a running APT prevents other apt processes from running as it holds a
lock on /var/lib/dpkg. I need to disable this automated APT task until Ansible
has completed the machine setup (which typically involves installing packages);
see https://github.com/gc3-uzh-ch/elasticluster/issues/304 for more info and
context.

The error message indicating this looks like this:

E: Could not get lock /var/lib/dpkg/lock-frontend - open (11: Resource temporarily unavailable)

E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?

I have tried various options to disable the "unattended upgrades" feature
through a "user data" script for cloud-init, but all of them have failed so
far.

1. Disable the systemd task

systemd task apt-daily.service is triggered by apt-daily.timer. I have tried
to disable one or the other, or both, with various cobinations of the following
commands; still, the apt-daily.service is started moments after the VM becomes
ready to accept SSH connections::

    #!/bin/bash



    systemctl stop apt-daily.timer

    systemctl disable apt-daily.timer

    systemctl mask apt-daily.service

    systemctl daemon-reload

2. Disable config option APT::Periodic::Enable

Script /usr/lib/apt/apt.systemd.daily reads a few APT configuration
variables; the setting APT::Periodic::Enable disables the functionality
altogether (lines 331--337). I have tried disabling it with the following
script::

    #!/bin/bash



    # cannot use /etc/apt/apt.conf.d/10periodic as suggested in

    # /usr/lib/apt/apt.systemd.daily, as Ubuntu distributes the

    # unattended upgrades stuff with priority 20 and 50 ...

    # so override everything with a 99xxx file

    cat > /etc/apt/apt.conf.d/99elasticluster <<__EOF

    APT::Periodic::Enable "0";

    // undo what's in 20auto-upgrade

    APT::Periodic::Update-Package-Lists "0";

    APT::Periodic::Unattended-Upgrade "0";

    __EOF

However, despite APT::Periodic::Enable having value 0 from the command line
(see below), the unattended-upgrades program is still run...

    ubuntu@test:~$ apt-config shell AutoAptEnable APT::Periodic::Enable

    AutoAptEnable='0'

3. Remove /usr/lib/apt/apt.systemd.daily altogether

The following cloud-init script removes the unattended upgrades script
altogether::

    #!/bin/bash



    mv /usr/lib/apt/apt.systemd.daily /usr/lib/apt/apt.systemd.daily.DISABLED

Still, the task runs and I can see it in the process table! although the file
does not exist if probed from the command line::

ubuntu@test:~$ ls /usr/lib/apt/apt.systemd.daily

ls: cannot access '/usr/lib/apt/apt.systemd.daily': No such file or directory

It looks as though the cloud-init script (together with the SSH command-line)
and the root systemd process execute in separate filesystems and process
spaces...

Questions

Is there something obvious I am missing? Or is there some namespace magic going
on which I am not aware of?

Most importantly: how can I disable the apt-daily.service through a
cloud-init script?

Yes, there was something obvious that I was missing.

Systemd is all about concurrent start of services, so the cloud-init script is
run at the same time the apt-daily.service is triggered. By the time
cloud-init gets to execute the user-specified payload, apt-get update is
already running. So the attempts 2. and 3. failed not because of some namespace
magic, but because they altered the system too late for apt.systemd.daily to
pick the changes up.

This also means that there is basically no way of preventing
apt.systemd.daily from running -- one can only kill it after it's started.

This "user data" script takes this route::

#!/bin/bash



systemctl stop apt-daily.service

systemctl kill --kill-who=all apt-daily.service



# wait until `apt-get updated` has been killed

while ! (systemctl list-units --all apt-daily.service | fgrep -q dead)

do

  sleep 1;

done



# now proceed with own APT tasks

apt install -y python

There is still a time window during which SSH logins are possible yet apt-get
will not run, but I cannot imagine another solution that can works on the stock
Ubuntu 16.04 cloud image.

Note: Unfortunately part of the solution below doesn't work on Ubuntu 16.04 systems (such as that of the questioner) because the suggested systemd-run invocation only works on Ubuntu 18.04 and above (see the comments for details). I'll leave the answer here because this question is still a popular hit regardless of which Ubuntu version you are using...

On Ubuntu 18.04 (and up) there may be up to two services involved in boot time apt updating/upgrading. The first apt-daily.service refreshes the list of packages. However there can be a second apt-daily-upgrade.service which actually installs security critical packages. An answer to the "Terminate and disable/remove unattended upgrade before command returns" question gives an excellent example of how to wait for both of these to finish (copied here for convenience):

systemd-run --property="After=apt-daily.service apt-daily-upgrade.service" --wait /bin/true

(note this has to be run as root). If you are trying to disable these services on future boots you will need to mask BOTH services:

systemctl mask apt-daily.service apt-daily-upgrade.service

Alternatively you can systemctl disable both services AND their associated timers (i.e. apt-daily.timer and apt-daily-upgrade.timer).

Note the masking/disabling techniques in this answer only prevent the update/upgrade on future boots - they won't stop them if they are already running in the current boot.

You can disable this via the "bootcmd" cloud-init module. This runs before network is brought up, which is required before apt update can get a chance to run.

#cloud-config

bootcmd:

    - echo 'APT::Periodic::Enable "0";' > /etc/apt/apt.conf.d/10cloudinit-disable

    - apt-get -y purge update-notifier-common ubuntu-release-upgrader-core landscape-common unattended-upgrades

    - echo "Removed APT and Ubuntu 18.04 garbage early" | systemd-cat

Once you ssh into the instance, you should also wait for the final phases of cloud-init to finish, since it moves apt sources / lists around.

# Wait for cloud-init to finish moving apt sources.list around... 

# a good source of random failures

# Note this is NOT a replacement for also disabling apt updates via bootcmd

while [ ! -f /var/lib/cloud/instance/boot-finished ]; do

    echo 'Waiting for cloud-init to finish...'

    sleep 3

done

This is also helpful to see how early the bootcmd runs:

# Show microseconds in systemd journal

journalctl -r -o short-precise

You can verify this worked as follows:

apt-config dump | grep Periodic



# Verify nothing was updated until we run apt update ourselves.

cd /var/lib/apt/lists

sudo du -sh .   # small size

ls -ltr         # old timestamps

Woudn't ist be easier to mask the unit

systemctl mask apt-daily.service

?