api-version negotiation sometimes fails over SSH connection

[thaJeztah]

Description

• related implement docker trust as plugin #6121 (comment)

API-version context negotiation seems to fail sometimes. In CI, this also shows with an error/warning (but in my reproduction steps, that didn't show for some reason, but possibly was suppressed by the output format)

Reproduce

docker context create --docker host=ssh://swarm-test-02 remote
remote
Successfully created context "remote"

The debug log comes from commandcon.New, so not 100% sure yet why it's triggered twice;

cli/cli/connhelper/commandconn/commandconn.go

Lines 34 to 73 in 9e50654

	// New returns net.Conn
	func New(ctx context.Context, cmd string, args ...string) (net.Conn, error) {
	// Don't kill the ssh process if the context is cancelled. Killing the
	// ssh process causes an error when go's http.Client tries to reuse the
	// net.Conn (commandConn).
	//
	// Not passing down the Context might seem counter-intuitive, but in this
	// case, the lifetime of the process should be managed by the http.Client,
	// not the caller's Context.
	//
	// Further details;;
	//
	// - https://github.com/docker/cli/pull/3900
	// - https://github.com/docker/compose/issues/9448#issuecomment-1264263721
	ctx = context.WithoutCancel(ctx)
	c := commandConn{cmd: exec.CommandContext(ctx, cmd, args...)}
	// we assume that args never contains sensitive information
	logrus.Debugf("commandconn: starting %s with %v", cmd, args)
	c.cmd.Env = os.Environ()
	c.cmd.SysProcAttr = &syscall.SysProcAttr{}
	setPdeathsig(c.cmd)
	createSession(c.cmd)
	var err error
	c.stdin, err = c.cmd.StdinPipe()
	if err != nil {
	return nil, err
	}
	c.stdout, err = c.cmd.StdoutPipe()
	if err != nil {
	return nil, err
	}
	c.cmd.Stderr = &stderrWriter{
	stderrMu: &c.stderrMu,
	stderr: &c.stderr,
	debugPrefix: fmt.Sprintf("commandconn (%s):", cmd),
	}
	c.localAddr = dummyAddr{network: "dummy", s: "dummy-0"}
	c.remoteAddr = dummyAddr{network: "dummy", s: "dummy-1"}
	return &c, c.cmd.Start()
	}

In this case it was run twice, and (see API version: 1.50) it didn't negotiate an API version;

docker context create --docker host=ssh://swarm-test-02 remote
remote
Successfully created context "remote"

docker --debug --context remote version
time="2025-06-10T14:29:31+02:00" level=debug msg="commandconn: starting ssh with [-o ConnectTimeout=30 -T -- swarm-test-02 docker system dial-stdio]"
time="2025-06-10T14:29:33+02:00" level=debug msg="commandconn: starting ssh with [-o ConnectTimeout=30 -T -- swarm-test-02 docker system dial-stdio]"
Client:
 Version:           28.2.2
 API version:       1.50
 Go version:        go1.24.3
 Git commit:        e6534b4
 Built:             Fri May 30 12:07:35 2025
 OS/Arch:           darwin/arm64
 Context:           remote

Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:35:23 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

On a second attempt, it was only ran once, and did negotiate (API version: 1.43 (downgraded from 1.50));

docker --debug --context remote version
time="2025-06-10T14:30:05+02:00" level=debug msg="commandconn: starting ssh with [-o ConnectTimeout=30 -T -- swarm-test-02 docker system dial-stdio]"
Client:
 Version:           28.2.2
 API version:       1.43 (downgraded from 1.50)
 Go version:        go1.24.3
 Git commit:        e6534b4
 Built:             Fri May 30 12:07:35 2025
 OS/Arch:           darwin/arm64
 Context:           remote

Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:35:23 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Expected behavior

No response

docker version

Client:
 Version:           28.2.2
 API version:       1.50
 Go version:        go1.24.3
 Git commit:        e6534b4
 Built:             Fri May 30 12:07:35 2025
 OS/Arch:           darwin/arm64
 Context:           remote

docker info

Likely not relevant

Additional Info

No response

[thaJeztah]

Looks like it's related to the ssh host being used for the first time; sometime possibly caching things?

After creating a context for a different SSH host, it showed the first time;

docker --debug --context remote2 version
time="2025-06-10T14:45:38+02:00" level=debug msg="commandconn: starting ssh with [-o ConnectTimeout=30 -T -- swarm-test-03 docker system dial-stdio]"
time="2025-06-10T14:45:40+02:00" level=debug msg="commandconn: starting ssh with [-o ConnectTimeout=30 -T -- swarm-test-03 docker system dial-stdio]"
Client:
 Version:           28.2.2
 API version:       1.50
 Go version:        go1.24.3
 Git commit:        e6534b4
 Built:             Fri May 30 12:07:35 2025
 OS/Arch:           darwin/arm64
 Context:           remote2

Second time, it connected faster and was successful;

docker --debug --context remote2 version
time="2025-06-10T14:45:55+02:00" level=debug msg="commandconn: starting ssh with [-o ConnectTimeout=30 -T -- swarm-test-03 docker system dial-stdio]"
Client:
 Version:           28.2.2
 API version:       1.42 (downgraded from 1.50)
 Go version:        go1.24.3
 Git commit:        e6534b4
 Built:             Fri May 30 12:07:35 2025
 OS/Arch:           darwin/arm64
 Context:           remote2

Trying to create a new context for the same host made no difference; it doesn't reproduce if it's for the same host, so possibly SSH caching things / writing things for the first connection?

[thaJeztah]

Also worth noting that while the Client info in the docker version shows "1.50", the daemon connection succeeded, and shows "1.42"