SSL_CERT_FILE in podman machine's systemd environment

[cluening]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I work in an environment that uses an interception proxy, meaning I need to set the normal proxy variables as well as the SSL_CERT_FILE variable when using podman machine on my mac. In recent podman releases, the certificate file gets copied into the VM correctly and almost all of the environment variables get set correctly, except one: the SSL_CERT_FILE variable does not get set at the systemd level to point at the copied-in certificate file. As a result, I am unable to pull images when on my corporate network:

laptop:~ cdlueni$ env | grep -i SSL_
SSL_CERT_FILE=/Users/cdlueni/company.crt
laptop:~ cdlueni$ podman machine init
laptop:~ cdlueni$ podman machine start
laptop:~ cdlueni$ podman pull alpine
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Error: initializing source docker://alpine:latest: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": x509: certificate signed by unknown authority

The error on the last line is due to the proxy's certificate not being set for processes being spawned by systemd. By manually editing /etc/systemd/system/envset-fwcfg.service to force the SSL_CERT_FILE variable to be added to the systemd config and then restarting the podman machine I was able to successfully pull images:

The changed line in envset-fwcfg.service:

ExecStart=/usr/bin/bash -c '/usr/bin/test -f ${FWCFGRAW} &&\
        echo "[Manager]\n#Got from QEMU FW_CFG\nDefaultEnvironment=$(/usr/bin/base64 -d ${FWCFGRAW} | sed -e "s+|+ +g") SSL_CERT_FILE=/etc/containers/certs.d/company.crt\n" > ${SYSTEMD_CONF} ||\
        echo "[Manager]\n#Got nothing from QEMU FW_CFG\n#DefaultEnvironment=\n" > ${SYSTEMD_CONF}'

Successful pull afterward:

laptop:~ cdlueni$ podman machine stop
laptop:~ cdlueni$ podman machine start
laptop:~ cdlueni$ podman pull alpine
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob sha256:9b18e9b68314027565b90ff6189d65942c0f7986da80df008b8431276885218e
Copying config sha256:a6215f271958c760a2975a6765016044115dbae4b90f414eba3a448a6a26b4f6
Writing manifest to image destination
Storing signatures
a6215f271958c760a2975a6765016044115dbae4b90f414eba3a448a6a26b4f6

It looks to me like, if the SSL_CERT_FILE variable is set on the host, then the correct path to the copy in the VM needs to be added to the systemd environment at VM boot time.

Steps to reproduce the issue:

1. Start a podman machine on a mac with proxy and certificate variables set

2. Try to pull an image

Describe the results you received:

SSL certificate errors

Describe the results you expected:

Succesful image pull

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client:       Podman Engine
Version:      4.2.1
API Version:  4.2.1
Go Version:   go1.19.1
Built:        Wed Dec 31 17:00:00 1969
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      4.2.0
API Version:  4.2.0
Go Version:   go1.18.4
Built:        Thu Aug 11 08:43:11 2022
OS/Arch:      linux/arm64

Output of podman info:

(paste your output here)

Package info (e.g. output of rpm -q podman or apt list podman or brew info podman):

port info podman
podman @4.2.1_1 (sysutils)

Description:          Podman is a tool for running Linux containers. You can do
                      this from a MacOS desktop as long as you have access to a
                      linux box either running inside of a VM on the host, or
                      available via the network. You need to install the remote
                      client and then setup ssh connection information.
Homepage:             /~https://github.com/containers/podman

Build Dependencies:   go, go-md2man
Runtime Dependencies: gvisor-tap-vsock, qemu
Platforms:            darwin, freebsd, linux
License:              Apache-2
Maintainers:          Email: judaew@macports.org, GitHub: judaew
                      Policy: openmaintainer

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (/~https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[bjorndown]

Related PR: #12748

/etc/profile.d/ssl_cert_file.sh is written during ignition which exports SSL_CERT_FILE.

Scripts in /etc/profile.d are usually only sourced in interactive environments which might be why SSL_CERT_FILE is not available to systemd services.

[bjorndown]

Passing SSL_CERT_FILE via -fw_cfg could be an option. This is also how proxy settings are propagated.

/~https://github.com/containers/podman/blob/main/pkg/machine/qemu/machine.go#L566-L574

[rhatdan]

@ashley-cui @baude WDYT?

[ashley-cui]

Sounds reasonable to me! @bjorndown Would you be interested in opening a PR?

[bjorndown]

@ashley-cui Yes, I will give it a try.

#16413 does look related. Should I include SSL_CERT_DIR as well? So far it is not being propagated, or at least I have not found references in the code. But for that to work we would need to copy the contents of SSL_CERT_DIR, similar to /~https://github.com/containers/podman/blob/main/pkg/machine/ignition.go#L494