[services] Job for sonic.target canceled.

[vivekrnv]

Description

This is seen intermittently. The sonic.target job is cancelled when the config reload is run. This is also seen in the cases of applying ZTP Plugins.

sonic.target is introduced in this PR: #5705

Note that this doesn't imply a problem with the sonic.target. According to systemd, even when the job is cancelled, the service won't go into the failed state. Also, the config reload will likely work the very next time you run it.

A potential fix for ZTP might be to, run the config reload again if this happens.

Steps to reproduce the issue:

In the case of ZTP plugins

1.Boot sonic with ZTP enabled feature
2.Announce DHCP option 67 pointing to ztp_config.json (see at the bottom)
3.Put all necessary plugins and configuration files using in ZTP process on WEB server.
4. Start ZTP process

ztp erase
ztp run

5.Wait till ZTP process has finished.
6.Verify the status of ZTP execution:
7.sudo show ztp status --verbose
8.Check /var/log/ztp.log and syslog

Describe the results you received:

Apr 19 12:53:54.947222 sonic INFO sonic-ztp[15343]: Stopping SONiC target ...
..........
..........
Apr 19 12:53:55.153250 sonic INFO sonic-ztp[15374]: Job for sonic.target canceled.
Apr 19 12:53:55.729680 sonic ERR sonic-ztp[13308]: configdb-json: Command 'config reload -y /tmp/config_dl.json' failed with exit code(1).

Describe the results you expected:

config reload should complete and so does the plugin of ZTP

Output of show version:

Output of show techsupport:

sonic_dump_DUT-1_20210419_130556.tar.gz

ZTP Plugins File

Additional information you deem important (e.g. issue happens only occasionally):

[anshuv-mfst]

@adam-yeung could you please take a look, thanks.

[dgsudharsan]

@adam-yeung could you please take a look, thanks.

Hi Adam,
Can you please provide an update?

[adyeung]

@rajendra-dendukuri pls help check

[vivekrnv]

Update: sonic.target cancelled can be easily reproduced even without running config reload -y.

root@sonic:~$ systemctl stop sonic.target

In paralell, 
admin@sonic:~$ sudo systemctl start dhcp_relay.service

Which would result in 
root@sonic:~$ systemctl stop sonic.target
Job for sonic.target canceled.

This happens because, stop job of sonic.target gets cancelled if there is a conflicting job installed even for one of it's dependent stop jobs

[qiluo-msft]

Update: sonic.target cancelled can be easily reproduced even without running config reload -y.

root@sonic:~$ systemctl stop sonic.target

In paralell, 
admin@sonic:~$ sudo systemctl start dhcp_relay.service

Which would result in 
root@sonic:~$ systemctl stop sonic.target
Job for sonic.target canceled.

This happens because, stop job of sonic.target gets cancelled if there is a conflicting job installed even for one of it's dependent stop jobs

This observation makes sense. What is wrong from your point of view?

[vivekrnv]

Agree, but this is affecting config reload and thus ZTP which depends on the config reload to proceed.