One container - multiple zfs datasets - common quota per container

[knutov]

There are cases when per-folder zfs optimization can lead to much better performance.

The simple example: container with some web-related project usually contains mysql/pg, some web server code and logs.

For mysql (/var/lib/mysql) it's better to have recordsize=16k (and sometimes disabled compression)
For logs (/var/logs/) it usually save to have sync=disabled (and may be stronger compression like zstd)
For static files for http server it's better to have recordsize=1M
etc...

LXD allows attach multiple storage volumes for container, but it's possible to set storage quota per volume only.

In case it's used for performance optimization purposes it's logically expected to have only per-container quota.

It's is possible to achieve this with zfs nested datasets with quota on top of them.

Is it possible to manage zfs dataset creation manually on container creation time, or is it possible to implement this, or are any another ways to achieve it?

[stgraber]

Currently, no, your only option is to use storage volumes which I'd likely recommend for the /var/lib/mysql case.
Quotas indeed can't be shared because they're not under the instance's dataset.

In general I don't think we have any interest in implementing this directly in LXD as it would make handling at least migration and import/export significantly more complicated (as we've noticed with all the btrfs issues caused by similar behavior).

I think the right fix for this going forward will be openzfs/zfs#12263 which allows for a container to create and manage datasets directly. Once that's landed in openzfs and available to users, LXD will be able to make the zfs tools work inside of a container at which point you can create and re-shuffle things any way you want.