Service gets stuck when calling "docker service update" and won't progress

[fooflington]

It appears that the service is getting stuck sometimes when asking docker to docker service update… on one cluster it's been stuck for over 24 hours…

$ docker exec -ti 97926cd69f86 ash
/ # ps -ef
PID   USER     TIME  COMMAND
    1 root      0:00 {shepherd} /bin/bash /usr/local/bin/shepherd
  301 root      6:35 docker service update x --detach=false --with-registry-auth --image=x
  315 root      0:00 ash
  321 root      0:00 ps -ef

If I kill pid=301 then it unblocks and continues on.

Consider calling docker service update via timeout to catch this?

/~https://github.com/djmaze/shepherd/blob/master/shepherd#L101

if ! timeout ${TIMEOUT_SECONDS:-300} docker "${config_flag[@]}" service update "$name" $detach_option $registry_auth $no_resolve_image_flag ${UPDATE_OPTIONS} --image="$image" > /dev/null

# timeout --help
BusyBox v1.34.1 (2021-11-23 00:57:35 UTC) multi-call binary.

Usage: timeout [-s SIG] SECS PROG ARGS

Run PROG. Send SIG to it if it is not gone in SECS seconds.
Default SIG: TERM.

[fooflington]

The underlying "hang" appears to be related to docker/cli#627 (the service being updated is at replicas=0 in my case because it's a "cronjob" managed by crazymax/swarm-cronjob)