Error: database libpod root directory (staticdir) does not match our libpod root directory (staticdir) #20872

kisame217 · 2023-12-01T15:18:11Z

Issue Description

Starting with version 4.8.0, I get the error below.

Error: database libpod root directory (staticdir) "/home/<user>/.local/share/containers/storage/libpod" does not match our libpod root directory (staticdir) "/var/home/<user>/.local/share/containers/storage/libpod": database configuration mismatch

I'm on Fedora Kinoite 39. No problems with version 4.7.2.

Steps to reproduce the issue

Upgraded to 4.8.0 from 4.7.2.
Try to start a container

podman --log-level debug start postgres

Describe the results you received

INFO[0000] podman filtering at log level debug          
DEBU[0000] Called start.PersistentPreRunE(podman --log-level debug start postgres) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using boltdb as database backend             
DEBU[0000] Initializing boltdb state at /var/home/<user>/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Overriding graph root "/var/home/<user>/.local/share/containers/storage" with "/home/<user>/.local/share/containers/storage" from database 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/<user>/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/<user>/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/<user>/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
Error: database libpod root directory (staticdir) "/home/<user>/.local/share/containers/storage/libpod" does not match our libpod root directory (staticdir) "/var/home/<user>/.local/share/containers/storage/libpod": database configuration mismatch
DEBU[0000] Shutting down engines

Describe the results you expected

Container should start normally.

podman info output

Error: database libpod root directory (staticdir) "/home/<user>/.local/share/containers/storage/libpod" does not match our libpod root directory (staticdir) "/var/home/<user>/.local/share/containers/storage/libpod": database configuration mismatch

OS: Fedora Kinoite 39
Arch: x86_64

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Running of bare metal.

Additional information

None.

The text was updated successfully, but these errors were encountered:

yann-soubeyrand · 2023-12-01T16:04:48Z

Same on Fedora 39 Silverblue (I guess it’s expected since it affects Kinoite 39).

Luap99 · 2023-12-01T16:36:33Z

What is the value of $HOME?

Luap99 · 2023-12-01T16:47:13Z

Going through the git log #20088 looks like the only think which touched this area (although I cannot say for sure it caused this until I can reproduce).
cc @mheon @vrothberg

yann-soubeyrand · 2023-12-01T16:50:59Z

What is the value of $HOME?

/home/yann for me. It’s the correct path to my home directory. However, /home is a symlink to /var/home on Silverblue (and I guess on Kinoite too).

Luap99 · 2023-12-01T16:55:39Z

Yeah I think the issue is because of this, previously we did not expand the symlink and now we do. Thus our db logic thinks you changed the config and errors out.

Luap99 · 2023-12-01T16:56:13Z

Does it work if you set HOME=/var/home/yann?

mheon · 2023-12-01T16:59:05Z

We can probably fix this in the logic for checking the current config against the DB - eval symlinks on both paths (the one in the DB and the one we're trying to use) and compare the final paths. We'd need fixes for both Bolt and SQLite that way though

Luap99 · 2023-12-01T17:08:25Z

reproducer on a not rpm-ostree system:

# create symlink for home
$ ln -s /home /tmp/home
# run system reset to start fresh
$ podman system reset

# run old podman 4.7.2, you need to change HOME var to your username of course
$ HOME=/tmp/home/pholzing podman ps
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
# run podman 4.8 or main
$ HOME=/tmp/home/pholzing bin/podman ps
Error: database libpod root directory (staticdir) "/tmp/home/pholzing/.local/share/containers/storage/libpod" does not match our libpod root directory (staticdir) "/home/pholzing/.local/share/containers/storage/libpod": database configuration mismatch

yann-soubeyrand · 2023-12-01T17:21:38Z

Does it work if you set HOME=/var/home/yann?

Unfortunately I cannot test for now, I’ve rolled back my system (it’s my workstation). However, if I understand correctly, you’re now pretty sure this issue is due to symlink expansion and me confirming it doesn’t seem to be needed anymore.

mheon · 2023-12-01T17:58:57Z

#20874 to fix

aanno · 2023-12-02T14:31:00Z

#20874 is still open. Is there a work-around? I tried to change HOME, but with no success.

kisame217 · 2023-12-02T15:50:46Z

#20874 is still open. Is there a work-around? I tried to change HOME, but with no success.

How did you change home? I set my home in /etc/passwd file. Also had to delete the containers folder in .local/share to reset the db config after I took backups of my container.

mheon · 2023-12-02T16:24:46Z

Setting static_dir in containers.conf is more likely to work. If your error message is:

Error: database libpod root directory (staticdir) "/home/<user>/.local/share/containers/storage/libpod" does not match our libpod root directory (staticdir) "/var/home/<user>/.local/share/containers/storage/libpod": database configuration mismatch

Then set static_dir to the "database libpod root directory" (in this case /home/<user>/.local/share/containers/storage/libpod)

elken · 2023-12-02T16:34:44Z

Looks like I timed it well to run into this.

Setting static_dir in containers.conf is more likely to work. If your error message is:
Error: database libpod root directory (staticdir) "/home/<user>/.local/share/containers/storage/libpod" does not match our libpod root directory (staticdir) "/var/home/<user>/.local/share/containers/storage/libpod": database configuration mismatch
Then set static_dir to the "database libpod root directory" (in this case /home/<user>/.local/share/containers/storage/libpod)

Have just tried with the following file to no luck

Have tried both combinations of home paths.

mheon · 2023-12-02T16:38:05Z

Looks pile it is now complaining about volume path, also in containers.conf (I think named volume_path but not sure?). Same procedure for the new error but different key in containers.conf basically.

…

On Sat, Dec 2, 2023 at 11:34 Ellis Kenyő ***@***.***> wrote: Looks like I timed it well to run into this. Setting static_dir in containers.conf is more likely to work. If your error message is: Error: database libpod root directory (staticdir) "/home/<user>/.local/share/containers/storage/libpod" does not match our libpod root directory (staticdir) "/var/home/<user>/.local/share/containers/storage/libpod": database configuration mismatch Then set static_dir to the "database libpod root directory" (in this case /home/<user>/.local/share/containers/storage/libpod) Have just tried with the following file to no luck image.png (view on web) </~https://github.com/containers/podman/assets/2872862/a69e326a-76ad-40c0-8281-6e6ac4a92072> Have tried both combinations of home paths. — Reply to this email directly, view it on GitHub <#20872 (comment)>, or unsubscribe </~https://github.com/notifications/unsubscribe-auth/AB3AOCFNHXPU3P7MGHWJ2LTYHNKC7AVCNFSM6AAAAABAC7ZUHKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZXGE4TOMZRGU> . You are receiving this because you were assigned.Message ID: ***@***.***>

elken · 2023-12-02T16:42:59Z

Looks pile it is now complaining about volume path, also in containers.conf (I think named volume_path but not sure?). Same procedure for the new error but different key in containers.conf basically.

Perfect! That works for podman at least

For future finders my file looks like

[engine]
static_dir = "/home/lkn/.local/share/containers/storage/libpod"
volume_path = "/home/lkn/.local/share/containers/storage/libpod"

Now trying to access existing containers with distrobox I get an error like

ERRO[0000] Unmounting /home/lkn/.local/share/containers/storage/overlay/f79b472b5b98d8e7a728c7d52415ea60b1043ae60cb1d08ee766f8dc71c08963/merged: invalid argument 
Error: unable to start container "ff34fb71718140a638f7c834ed6d116f413d8a50f8bfbc88307a38d717f7ff52": mounting storage for container ff34fb71718140a638f7c834ed6d116f413d8a50f8bfbc88307a38d717f7ff52: creating overlay mount to /home/lkn/.local/share/containers/storage/overlay/f79b472b5b98d8e7a728c7d52415ea60b1043ae60cb1d08ee766f8dc71c08963/merged, mount_data="lowerdir=/home/lkn/.local/share/containers/storage/overlay/l/6I7L4CUQJIQPFUYHFX5OZBL4JK:/home/lkn/.local/share/containers/storage/overlay/l/6I7L4CUQJIQPFUYHFX5OZBL4JK/../diff1:/home/lkn/.local/share/containers/storage/overlay/l/7JTTU5TJDPKSG76SE4CVJCD5OT:/home/lkn/.local/share/containers/storage/overlay/l/BFAWGXFYIDBJFEOEEW66NT5TUI,upperdir=/home/lkn/.local/share/containers/storage/overlay/f79b472b5b98d8e7a728c7d52415ea60b1043ae60cb1d08ee766f8dc71c08963/diff,workdir=/home/lkn/.local/share/containers/storage/overlay/f79b472b5b98d8e7a728c7d52415ea60b1043ae60cb1d08ee766f8dc71c08963/work,userxattr,context=\"system_u:object_r:container_file_t:s0:c1022,c1023\"": invalid argument

But I'm happy to take this elsewhere if it's not directly related to podman 😄

EDIT: Managed to rescue, turns out it was SELinux. Turning it off but not fully disabled allowed me to carry on using my old containers; thanks @mheon you're a life-saver 😄

When Podman starts, it checks a number of critical runtime paths against stored values in the database to make sure that existing containers are not broken by a configuration change. We recently made some changes to this logic to make our handling of the some options more sane (StaticDir in particular was set based on other passed options in a way that was not particularly sane) which has made the logic more sensitive to paths with symlinks. As a simple fix, handle symlinks properly in our DB vs runtime comparisons. The BoltDB bits are uglier because very, very old Podman versions sometimes did not stuff a proper value in the database and instead used the empty string. SQLite is new enough that we don't have to worry about such things. Fixes containers#20872 Signed-off-by: Matt Heon <mheon@redhat.com>

includes workaround for containers/podman#20872

When Podman starts, it checks a number of critical runtime paths against stored values in the database to make sure that existing containers are not broken by a configuration change. We recently made some changes to this logic to make our handling of the some options more sane (StaticDir in particular was set based on other passed options in a way that was not particularly sane) which has made the logic more sensitive to paths with symlinks. As a simple fix, handle symlinks properly in our DB vs runtime comparisons. The BoltDB bits are uglier because very, very old Podman versions sometimes did not stuff a proper value in the database and instead used the empty string. SQLite is new enough that we don't have to worry about such things. Fixes containers#20872 Signed-off-by: Matt Heon <mheon@redhat.com>

nariba · 2023-12-07T08:54:19Z

Thank you very much!
I was having the same problem and this issue helped me solve it.

In my environment, there was no directory, not even containers.conf, but if I create the directory and file, it should be OK.
The containers.conf mentioned in this comment is in the following path,
/home/lkn/.config/containers/containers.conf

debarshiray · 2023-12-07T12:05:08Z

I couldn't reproduce this on my Fedora 39 Silverblue:

rishi@fedora:~$ podman version
Client:       Podman Engine
Version:      4.8.0
API Version:  4.8.0
Go Version:   go1.21.4
Built:        Tue Nov 28 10:58:32 2023
OS/Arch:      linux/amd64
rishi@fedora:~$ 
rishi@fedora:~$ echo $HOME
/var/home/rishi
rishi@fedora:~$

... but clearly a lot of people can. Based on these two comments, I think this affects those who have been upgrading their installations in-place for a while.

I usually reinstall once every year, and I remember that we changed the default value of HOME from /home/foo to /var/home/foo at some point to avoid relying on the symbolic link.

walking-octopus · 2023-12-08T08:17:17Z

stas@thinkpad-t14 /h/stas [1]> rpm-ostree status -b
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
BootedDeployment:
● fedora:fedora/39/x86_64/kinoite
                  Version: 39.20231205.0 (2023-12-05T01:04:12Z)
               BaseCommit: 694f995d4b7ca42e87b393ff3058949dad660e5f2d692338dacaaceab06cd754
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: 'gcc-c++' adb alsa-lib-devel automake cmake duperemove ffmpeg-free fish gcc htop
                           jetbrains-mono-fonts-all make mozilla-openh264 podman-compose podman-docker
                           powertop systemd-devel tmux xeyes

AvailableUpdate:
           Diff: 7 upgraded
stas@thinkpad-t14 ~> echo $HOME
/home/stas
stas@thinkpad-t14 ~> export HOME=/var/home/stas
stas@thinkpad-t14 /h/stas> echo $HOME
/var/home/stas
stas@thinkpad-t14 /h/stas> podman --version
podman version 4.8.0
stas@thinkpad-t14 /h/stas> podman version
Error: database libpod root directory (staticdir) "/home/stas/.local/share/containers/storage/libpod" does not match our libpod root directory (staticdir) "/var/home/stas/.local/share/containers/storage/libpod": database configuration mismatch

Just changing the $HOME seems to not be enough. Perhaps there's a config file somewhere that I can edit?

It's also strange it didn't happen after the rebase but just another version update...

elken · 2023-12-08T16:32:58Z

Perhaps there's a config file somewhere that I can edit?

#20872 (comment)

quazar-omega · 2023-12-08T17:04:26Z

Does that workaround have any long-term consequences, or can we safely apply it temporarily until the fix lands?

mheon · 2023-12-08T17:05:37Z

There are no consequences even after the fix lands. The two paths are symlinked together, we're just forcing another version be used when doing database checks.

quazar-omega · 2023-12-08T17:07:38Z

Oh good, thanks for explaining!

Justinzobel · 2023-12-17T12:00:44Z

I rebased from Fedora 39 Kinoite to Rawhide Kinoite to do some testing and this affects my ability to run a toolbox:

toolbox
Error: failed to get the Podman version
...
podman --version
podman version 4.8.1

kisame217 added the kind/bug Categorizes issue or PR as related to a bug. label Dec 1, 2023

Luap99 added the regression label Dec 1, 2023

Luap99 assigned mheon Dec 1, 2023

mheon mentioned this issue Dec 1, 2023

Handle symlinks when checking DB vs runtime configs #20874

Merged

bitestring mentioned this issue Dec 3, 2023

Podman throws database configuration mismatch error fedora-silverblue/issue-tracker#513

Closed

clemak27 added a commit to clemak27/linux_setup that referenced this issue Dec 3, 2023

fest: add target for podman-setup

1150ff9

includes workaround for containers/podman#20872

clemak27 added a commit to clemak27/linux_setup that referenced this issue Dec 3, 2023

fest: add target for podman-setup

c564714

includes workaround for containers/podman#20872

openshift-merge-bot bot closed this as completed in #20874 Dec 3, 2023

This was referenced Dec 4, 2023

Fedora 39 Daily Compose Test- 2023-12-01 virt-s1/rhel-edge#5819

Closed

Fedora 39 Daily Compose Test- 2023-12-03 virt-s1/rhel-edge#5839

Closed

Fedora 39 Daily Compose Test- 2023-12-02 virt-s1/rhel-edge#5828

Closed

walking-octopus mentioned this issue Dec 4, 2023

libpod root directory (staticdir) does not match our libpod root directory (staticdir) on fedora:fedora/39/x86_64/kinoite 39.20231204.0 fedora-silverblue/issue-tracker#514

Closed

srcshelton mentioned this issue Dec 4, 2023

podman-4.8.0: "Rolling back transaction to validate database' #20902

Closed

dustymabe mentioned this issue Dec 4, 2023

various issues with podman 4.8 coreos/fedora-coreos-tracker#1625

Closed

This was referenced Dec 4, 2023

Fedora 39 Daily Compose Test- 2023-12-04 virt-s1/rhel-edge#5858

Closed

Fedora-Rawhide-20231204.n.0 - 2023-12-04 virt-s1/rhel-edge#5859

Merged

Fedora 39 Daily Compose Test- 2023-12-05 virt-s1/rhel-edge#5874

Closed

chrisawi mentioned this issue Dec 9, 2023

No updates since 39.20231205.0 fedora-silverblue/issue-tracker#516

Closed

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Mar 17, 2024

github-actions bot locked as resolved and limited conversation to collaborators Mar 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error: database libpod root directory (staticdir) does not match our libpod root directory (staticdir) #20872

Error: database libpod root directory (staticdir) does not match our libpod root directory (staticdir) #20872

kisame217 commented Dec 1, 2023

yann-soubeyrand commented Dec 1, 2023

Luap99 commented Dec 1, 2023

Luap99 commented Dec 1, 2023

yann-soubeyrand commented Dec 1, 2023

Luap99 commented Dec 1, 2023

Luap99 commented Dec 1, 2023

mheon commented Dec 1, 2023

Luap99 commented Dec 1, 2023

yann-soubeyrand commented Dec 1, 2023

mheon commented Dec 1, 2023

aanno commented Dec 2, 2023

kisame217 commented Dec 2, 2023

mheon commented Dec 2, 2023

elken commented Dec 2, 2023

mheon commented Dec 2, 2023 via email

elken commented Dec 2, 2023 •

edited

Loading

nariba commented Dec 7, 2023

debarshiray commented Dec 7, 2023 •

edited

Loading

walking-octopus commented Dec 8, 2023

elken commented Dec 8, 2023

quazar-omega commented Dec 8, 2023

mheon commented Dec 8, 2023

quazar-omega commented Dec 8, 2023

Justinzobel commented Dec 17, 2023

Error: database libpod root directory (staticdir) does not match our libpod root directory (staticdir) #20872

Error: database libpod root directory (staticdir) does not match our libpod root directory (staticdir) #20872

Comments

kisame217 commented Dec 1, 2023

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

yann-soubeyrand commented Dec 1, 2023

Luap99 commented Dec 1, 2023

Luap99 commented Dec 1, 2023

yann-soubeyrand commented Dec 1, 2023

Luap99 commented Dec 1, 2023

Luap99 commented Dec 1, 2023

mheon commented Dec 1, 2023

Luap99 commented Dec 1, 2023

yann-soubeyrand commented Dec 1, 2023

mheon commented Dec 1, 2023

aanno commented Dec 2, 2023

kisame217 commented Dec 2, 2023

mheon commented Dec 2, 2023

elken commented Dec 2, 2023

mheon commented Dec 2, 2023 via email

elken commented Dec 2, 2023 • edited Loading

nariba commented Dec 7, 2023

debarshiray commented Dec 7, 2023 • edited Loading

walking-octopus commented Dec 8, 2023

elken commented Dec 8, 2023

quazar-omega commented Dec 8, 2023

mheon commented Dec 8, 2023

quazar-omega commented Dec 8, 2023

Justinzobel commented Dec 17, 2023

elken commented Dec 2, 2023 •

edited

Loading

debarshiray commented Dec 7, 2023 •

edited

Loading