[Experiment] FFmpeg backend for Qt6 QtMultimedia and support for VAAPI drivers #23089

spiderkeys · 2024-03-14T01:47:53Z

Attempts to add support for building Qt6.4+'s FFMpeg backend for QtMultimedia.

This PR contains several changes to try and work towards functional ffmpeg-based hardware decoding (and other multimedia functions) within Qt6, which shifted focus from their gstreamer-based backend to ffmpeg in Qt6.4, so as to target all platforms with one library.

This is not meant to ever be merged, but to serve as grounds for discussion to figure out the right way to achieve these goals.

The impetus behind these changes come from difficulties we have had when trying to use conan-packaged ffmpeg within our Qt application. We were originally using Qt's official release binaries, but once Qt shifted to ffmpeg and added support for ffmpeg's hardware decoders, we discovered that we had a dependency conflict between the ffmpeg version Qt built and statically linked against (used by their MediaPlayer QML element, among others) and our conan-supplied shared version of ffmpeg+libva. (context, if desired: https://bugreports.qt.io/browse/QTBUG-122757)

We are now trying to shift to using the conan Qt recipe, but there is currently only support for the gstreamer qtmultimedia backend. Upon trying to follow the same process within the recipe as the gstreamer plugin, I ran into some hurdles:

Qt has a handful of its own FindX.cmake files which all differ in behavior, making it hard to get Qt to find conan dependencies. Some get overridden, while others explicitly do not. Some of their cmake modules do not follow the same patterns or implementations as the others, so each needs to be handled on a case-by-case basis.
with_pulseaudio was forbidden by the recipe due to some outstanding bug/build limitation
pulseaudio has a cmake find file called WrapPulseAudio, which I don't seem to properly have accounted for yet
libva also has a Find file, which I think I have properly added support for, but may be wrong

I've included a non-trivial Qt application, which I am trying to use to test the various aspects of the Qt recipe that I ultimately depend on. It isn't complete yet, because I'm stuck on the Qt6 package build with the current conan option set.

My current blocker is a long list of linking errors to pulseaudio symbols:

...
/usr/bin/ld: /home/spiderkeys/.conan2/p/b/pulse864d9de525735/b/src/src/pulse/stream.c:1694: undefined reference to `pa_log_level_meta'
/usr/bin/ld: /home/spiderkeys/.conan2/p/b/pulse864d9de525735/b/src/src/pulse/stream.c:1696: undefined reference to `pa_detect_fork'
/usr/bin/ld: /home/spiderkeys/.conan2/p/b/pulse864d9de525735/b/src/src/pulse/stream.c:1700: undefined reference to `pa_memblockq_get_length'
...
collect2: error: ld returned 1 exit status
[4954/5077] Building CXX object qtmulti...ediaPlugin.dir/qv4l2cameradevices.cpp.o
ninja: build stopped: subcommand failed.

qt/6.6.2: ERROR: 
Package 'dd971a2f5642e628f9fa9524b36fa90ea936552b' build failed
qt/6.6.2: WARN: Build folder /home/spiderkeys/.conan2/p/b/qt098fadc0ba2b7/b/build/Debug
ERROR: qt/6.6.2: Error in build() method, line 813
        cmake.build()
        ConanException: Error 1 while executing

So my next step is to figure out what is wrong with the attempted inclusion of pulseaudio into the recipe, most likely relating to the cmake find file.

Any advice/feedback on this effort is much appreciated.

Some random notes:

There was some issue with either my nvidia-vaapi-driver recipe or the libva recipe which prevented nvidia-vaapi-driver's meson configure process from finding libdrm. I had a custom libva recipe of my own (from before there was one in conan center) which worked, so I replaced the libva recipe content with my own, just to get things working. This required a change in ffmpeg as well, to make sure it found the correct target components.
I added what I think should be necessary to enable cuvid decoding support in the ffmpeg recipe, such that it builds, but I haven't been able to test it yet.

…d on component name changes. Also adds 'with_cuvid' option.

… is super slow for such a large file, even compared to mirrors oceans away)

…end for qtmultimedia. Attempts to re-enable 'with_pulseaudio` because it is required for the ffmpeg plugin. Attempts to override Qt cmake find files to find conan ffmpeg+libva+pulseaudio.

…(still wip)

github-actions · 2024-03-14T01:48:21Z

🤖 Beep Boop! This pull request is making changes to 'recipes/ffmpeg//'.

👋 @MartinDelille you might be interested. 😉

github-actions · 2024-03-14T01:48:50Z

🤖 Beep Boop! This pull request is making changes to 'recipes/qt//'.

👋 @ericLemanissier @jwillikers @MartinDelille @paulharris you might be interested. 😉

…e linking issues

…nctioning) media player example

spiderkeys · 2024-03-14T05:50:11Z

I was able to get the pulseaudio errors resolved by changing it to shared=True. Cleaned up a bit more, though the test project is still quite a mess. That said, it now fully builds with all of my Qt features enabled, and a qml app window launches, but the media player cannot load/play a video yet.

Not sure why it is trying to load vdpau when I've asked for vaapi.

jwillikers · 2024-03-14T14:21:58Z

recipes/intel-media-driver/all/conanfile.py

+        self.cpp_info.set_property("cmake_target_name", "intel-media-driver")
+        self.cpp_info.set_property("pkg_config_name", "intel-media-driver")
+
+        self.cpp_info.components["intel-media-driver"].libs = collect_libs(self)


You should avoid using collect_libs if possible. Since this is a runtime-loadable plugin, I'm not sure that you even need to provide a library in this way.

You'll want to make the VA-API libraries accessible at runtime:

self.runenv_info.prepend_path("LIBVA_DRIVERS_PATH", os.path.join(self.package_folder, "lib", "dri"))

Good point on the libs bit, this seems like it can be removed altogether since libva always runtime loads it and nothing ever depends at link time on it.

Regarding the VAAPI env variables, I thought about this when I originally wrote this recipe, but ultimately ended up setting those variables in my consuming application. We give users the option to set both the path to look for VAAPI drivers and select which one they'd prefer to use, as we found support widely varied across hardware targets and OS versions.

Still might make sense to set it here anyway and the user can just override it if they need. By adding it, there is at least a chance for things to work out of the box.

Yeah, I think it might be worth setting here and letting consumer's figure out whether or not to include it as a dependency as appropriate for the specific system.

One thought just now, how would you expect to handle the case where there are two VAAPI driver packages that both do this?

As an example, in our app we pull in both intel-media-driver and nvidia-vaapi-driver and the correct one can be used by the application based on the hardware it is running on. There can only be one LIBVA_DRIVERS_PATH, so it seems like the value couldn't be the individual package folder itself in that case?

We handle this in our consumer conanfile.py by merging dri/<driver-name>.so for each package into a single thirdparty/dri/ during our consumer's generate() step, then LIBVA_DRIVERS_PATH can be set to that one directory and find any conan-installed VAAPI drivers.

Seriously? That seems like a ridiculous limitation. Do you know if there's a good reason for that?

Hmm, I found this: /~https://github.com/canonical/snapcraft/pull/3591/files which seems to indicate someone at least tried to append a path to that variable.

Aha! It should work with multiple paths: intel/libva#639 (comment)

/~https://github.com/intel/libva/blob/907b2b5405ca1091b4360bf35060e143bd704b62/va/va.c#L393
Great thanks, just validated it myself in the source. Good find!

…ires ffmpeg and puts vaapi drivers in the correct directory

conan-center-bot · 2024-03-14T23:29:49Z

Conan v1 pipeline ❌

Changes not allowed in build 4:

[qt_test_project/CMakeLists.txt, qt_test_project/build.sh, qt_test_project/conanfile.py, qt_test_project/launch.sh, qt_test_project/src/CMakeLists.txt, qt_test_project/src/app/CMakeLists.txt, qt_test_project/src/app/res/qml/main.qml, qt_test_project/src/app/res/svg/lightbulb.svg, qt_test_project/src/app/src/main.cpp, qt_test_project/src/libs/CMakeLists.txt, qt_test_project/src/libs/include/my_lib/my_lib.hpp, qt_test_project/src/libs/src/my_lib.cpp, recipes/ffmpeg/all/conanfile.py, recipes/gmmlib/all/conandata.yml, recipes/gmmlib/all/conanfile.py, recipes/gmmlib/config.yml, recipes/intel-media-driver/23.4.3/conandata.yml, recipes/intel-media-driver/23.4.3/conanfile.py, recipes/intel-media-driver/all/conandata.yml, recipes/intel-media-driver/all/conanfile.py, recipes/intel-media-driver/config.yml, recipes/libsndfile/all/conanfile.py, recipes/libsndio/all/conandata.yml, recipes/libsndio/all/conanfile.py, recipes/libsndio/all/test_package/CMakeLists.txt, recipes/libsndio/all/test_package/conanfile.py, recipes/libsndio/all/test_package/test_package.c, recipes/libsndio/all/test_v1_package/CMakeLists.txt, recipes/libsndio/all/test_v1_package/conanfile.py, recipes/libsndio/config.yml, recipes/libva/all/conanfile.py, recipes/nvidia-vaapi-driver/all/conandata.yml, recipes/nvidia-vaapi-driver/all/conanfile.py, recipes/nvidia-vaapi-driver/config.yml, recipes/qt/6.x.x/conandata.yml, recipes/qt/6.x.x/conanfile.py]

One or more of the folder/files added are not allowed into the repository.

Conan v2 pipeline ❌

Note: Conan v2 builds are now mandatory. Please read our discussion about it.

The v2 pipeline failed. Please, review the errors and note this is required for pull requests to be merged. In case this recipe is still not ported to Conan 2.x, please, ping @conan-io/barbarians on the PR and we will help you.

See details:

Changes not allowed in build 4:

[qt_test_project/CMakeLists.txt, qt_test_project/build.sh, qt_test_project/conanfile.py, qt_test_project/launch.sh, qt_test_project/src/CMakeLists.txt, qt_test_project/src/app/CMakeLists.txt, qt_test_project/src/app/res/qml/main.qml, qt_test_project/src/app/res/svg/lightbulb.svg, qt_test_project/src/app/src/main.cpp, qt_test_project/src/libs/CMakeLists.txt, qt_test_project/src/libs/include/my_lib/my_lib.hpp, qt_test_project/src/libs/src/my_lib.cpp, recipes/ffmpeg/all/conanfile.py, recipes/gmmlib/all/conandata.yml, recipes/gmmlib/all/conanfile.py, recipes/gmmlib/config.yml, recipes/intel-media-driver/23.4.3/conandata.yml, recipes/intel-media-driver/23.4.3/conanfile.py, recipes/intel-media-driver/all/conandata.yml, recipes/intel-media-driver/all/conanfile.py, recipes/intel-media-driver/config.yml, recipes/libsndfile/all/conanfile.py, recipes/libsndio/all/conandata.yml, recipes/libsndio/all/conanfile.py, recipes/libsndio/all/test_package/CMakeLists.txt, recipes/libsndio/all/test_package/conanfile.py, recipes/libsndio/all/test_package/test_package.c, recipes/libsndio/all/test_v1_package/CMakeLists.txt, recipes/libsndio/all/test_v1_package/conanfile.py, recipes/libsndio/config.yml, recipes/libva/all/conanfile.py, recipes/nvidia-vaapi-driver/all/conandata.yml, recipes/nvidia-vaapi-driver/all/conanfile.py, recipes/nvidia-vaapi-driver/config.yml, recipes/qt/6.x.x/conandata.yml, recipes/qt/6.x.x/conanfile.py]

One or more of the folder/files added are not allowed into the repository.

spiderkeys · 2024-03-14T23:35:26Z

As of my last push, I now believe I have everything working together. Qt's MediaPlayer loads a remote MP4 containing h264, loads the FFMpeg plugin, and FFmpeg reports that it has found our conan-provided libva and intel-media-driver and loaded them as the HWaccel decoder:

"FFmpeg log: Trying to use DRM render node for device 0."
"FFmpeg log: libva: VA-API version 1.20.0"
"FFmpeg log: libva: User environment variable requested driver 'iHD'"
"FFmpeg log: libva: Trying to open /home/spiderkeys/z/workspace/conan-center-fork/qt_test_project/build/x86_64/Debug/thirdparty/dri/iHD_drv_video.so"
"FFmpeg log: libva: Found init function __vaDriverInit_1_20"
"FFmpeg log: libva: va_openDriver() returns 0"
"FFmpeg log: Initialised VAAPI connection: version 1.20"
"FFmpeg log: Format 0x41524742 -> bgra."
"FFmpeg log: Format 0x42475241 -> argb."
"FFmpeg log: Format 0x41424752 -> rgba."
"FFmpeg log: Format 0x52474241 -> abgr."
"FFmpeg log: Format 0x58524742 -> bgr0."
"FFmpeg log: Format 0x42475258 -> 0rgb."
"FFmpeg log: Format 0x58424752 -> rgb0."
"FFmpeg log: Format 0x52474258 -> 0bgr."
"FFmpeg log: Format 0x30335241 -> unknown."
"FFmpeg log: Format 0x30334241 -> unknown."
"FFmpeg log: Format 0x30335258 -> x2rgb10le."
"FFmpeg log: Format 0x30334258 -> unknown."
"FFmpeg log: Format 0x36314752 -> unknown."
"FFmpeg log: Format 0x50424752 -> unknown."
"FFmpeg log: Format 0x50524742 -> unknown."
"FFmpeg log: Format 0x56555941 -> unknown."
"FFmpeg log: Format 0x30303859 -> gray."
"FFmpeg log: Format 0x3231564e -> nv12."
"FFmpeg log: Format 0x3132564e -> unknown."
"FFmpeg log: Format 0x32595559 -> yuyv422."
"FFmpeg log: Format 0x59565955 -> uyvy422."
"FFmpeg log: Format 0x32315659 -> yuv420p."
"FFmpeg log: Format 0x30323449 -> yuv420p."
"FFmpeg log: Format 0x50313134 -> yuv411p."
"FFmpeg log: Format 0x48323234 -> yuv422p."
"FFmpeg log: Format 0x56323234 -> yuv440p."
"FFmpeg log: Format 0x50343434 -> yuv444p."
"FFmpeg log: Format 0x33434d49 -> unknown."
"FFmpeg log: Format 0x30313050 -> p010le."
"FFmpeg log: VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 23.4.3 ()."
"FFmpeg log: Driver not found in known nonstandard list, using standard behaviour."
"FFmpeg log: detected 8 logical cores"
"FFmpeg log: nal_unit_type: 7(SPS), nal_ref_idc: 3"
"FFmpeg log: Decoding VUI"
"FFmpeg log: nal_unit_type: 8(PPS), nal_ref_idc: 3"

The "cuda" HWAccel decoder (h264_cuvid in ffmpeg) also works:

[AVHWDeviceContext @ 0x559507c5ef40] Calling cu->cuCtxPushCurrent(hwctx->cuda_ctx)
[AVHWDeviceContext @ 0x559507c5ef40] Calling cu->cuMemcpy2DAsync(&cpy, hwctx->stream)
[AVHWDeviceContext @ 0x559507c5ef40] Calling cu->cuMemcpy2DAsync(&cpy, hwctx->stream)
[AVHWDeviceContext @ 0x559507c5ef40] Calling cu->cuStreamSynchronize(hwctx->stream)
[AVHWDeviceContext @ 0x559507c5ef40] Calling cu->cuCtxPopCurrent(&dummy)
[h264_cuvid @ 0x5595070e03a0] cuvid_output_frame
[h264_cuvid @ 0x5595070e03a0] cuvid_decode_packet
[h264_cuvid @ 0x5595070e03a0] Calling ctx->cudl->cuCtxPushCurrent(cuda_ctx)
[h264_cuvid @ 0x5595070e03a0] pfnDecodePicture
[h264_cuvid @ 0x5595070e03a0] Calling ctx->cvdl->cuvidDecodePicture(ctx->cudecoder, picparams)
[h264_cuvid @ 0x5595070e03a0] Calling ctx->cvdl->cuvidParseVideoData(ctx->cuparser, &cupkt)
[h264_cuvid @ 0x5595070e03a0] Calling ctx->cudl->cuCtxPopCurrent(&dummy)
[h264_cuvid @ 0x5595070e03a0] Calling ctx->cudl->cuCtxPushCurrent(cuda_ctx)
[h264_cuvid @ 0x5595070e03a0] Calling ctx->cvdl->cuvidMapVideoFrame(ctx->cudecoder, parsed_frame.dispinfo.picture_index, &mapped_frame, &pitch, &params)
[h264_cuvid @ 0x5595070e03a0] Calling ctx->cudl->cuMemcpy2DAsync(&cpy, device_hwctx->stream)
[h264_cuvid @ 0x5595070e03a0] Calling ctx->cudl->cuMemcpy2DAsync(&cpy, device_hwctx->stream)
[h264_cuvid @ 0x5595070e03a0] Calling ctx->cudl->cuMemcpy2DAsync(&cpy, device_hwctx->stream)
[h264_cuvid @ 0x5595070e03a0] Calling ctx->cvdl->cuvidUnmapVideoFrame(ctx->cudecoder, mapped_frame)
[h264_cuvid @ 0x5595070e03a0] Calling ctx->cudl->cuCtxPopCurrent(&dummy)
[h264_cuvid @ 0x5595070e03a0] cuvid_output_frame

I also fixed some issues with qtimageformat while I was at it, and brought in my fixes to the whole libsndio->pulseaudio chain, and tested that audio decoding/playback through ffmpeg was also working correctly.

Now that everything works as a proof of concept, if there is any advice/feedback on any element of the approach at a high level, I'd be glad to hear and iterate on it here for a bit longer.

From there, I can split things out into individual PRs per recipe and close this PR.

uilianries · 2024-08-05T12:06:02Z

Closing this PR to avoid new CI actions, as it has multiple recipes.

Please, feel free to open new PRs splitting it. Thank you!

spiderkeys added 10 commits March 13, 2024 18:21

Adds gmmlib recipe

181feab

Adds intel-media-driver recipe

2429b4d

Adds libsndio recipe

875d491

Updates libsndfile recipe to use libsndio

1634e9d

Adds nvidia-vaapi-driver recipe

f07b6cd

Fixes libva recipe so nvidia-vaapi-driver build can find libdrm

76ef391

Adds changes from conan-io#2290, and fixes ffmpeg usage of libva base…

cbbb336

…d on component name changes. Also adds 'with_cuvid' option.

Reorders qt6.6.2 source urls to speed up download (official qt mirror…

0d3af3f

… is super slow for such a large file, even compared to mirrors oceans away)

Initial work to update qt6 recipe to support building the ffmpeg back…

4194fb4

…end for qtmultimedia. Attempts to re-enable 'with_pulseaudio` because it is required for the ffmpeg plugin. Attempts to override Qt cmake find files to find conan ffmpeg+libva+pulseaudio.

Adds a non-trivial qt example project to try and exercise qt6 recipe …

43b89d8

…(still wip)

conan-center-bot added the User-approval pending label Mar 14, 2024

This comment has been minimized.

Sign in to view

spiderkeys added 2 commits March 13, 2024 22:39

Fixes issue with un-provided Libmng in qt imageformats plugin

97c0c91

Enables more qt features and switches pulseaudio to shared, fixing th…

0b068df

…e linking issues

This comment has been minimized.

Sign in to view

Cleans up geojson related stuff and gets to a runnable (albeit not-fu…

7c5b528

…nctioning) media player example

spiderkeys marked this pull request as draft March 14, 2024 05:46

This comment has been minimized.

Sign in to view

ghost mentioned this pull request Mar 14, 2024

[service] Potentially conflicting recipes #3571

Closed

jwillikers suggested changes Mar 14, 2024

View reviewed changes

spiderkeys added 3 commits March 14, 2024 16:18

Turns off webp because -lsharpyuv symbols can't be found currently

74aad49

Cleans up test project and fixes conanfile to ensure it directly requ…

0c86dee

…ires ffmpeg and puts vaapi drivers in the correct directory

Updates libsndio to include fixes from other PR

da04282

conan-center-bot added Failed and removed User-approval pending labels Mar 14, 2024

ErniGH mentioned this pull request Jun 14, 2024

Prioritization of new recipe PRs #24327

Open

uilianries closed this Aug 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Experiment] FFmpeg backend for Qt6 QtMultimedia and support for VAAPI drivers #23089

[Experiment] FFmpeg backend for Qt6 QtMultimedia and support for VAAPI drivers #23089

spiderkeys commented Mar 14, 2024 •

edited

Loading

github-actions bot commented Mar 14, 2024

github-actions bot commented Mar 14, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

spiderkeys commented Mar 14, 2024

jwillikers Mar 14, 2024

spiderkeys Mar 14, 2024

jwillikers Mar 14, 2024

spiderkeys Mar 14, 2024 •

edited

Loading

jwillikers Mar 14, 2024

jwillikers Mar 14, 2024

jwillikers Mar 14, 2024

spiderkeys Mar 14, 2024

conan-center-bot commented Mar 14, 2024

spiderkeys commented Mar 14, 2024 •

edited

Loading

uilianries commented Aug 5, 2024

[Experiment] FFmpeg backend for Qt6 QtMultimedia and support for VAAPI drivers #23089

[Experiment] FFmpeg backend for Qt6 QtMultimedia and support for VAAPI drivers #23089

Conversation

spiderkeys commented Mar 14, 2024 • edited Loading

github-actions bot commented Mar 14, 2024

github-actions bot commented Mar 14, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

spiderkeys commented Mar 14, 2024

jwillikers Mar 14, 2024

Choose a reason for hiding this comment

spiderkeys Mar 14, 2024

Choose a reason for hiding this comment

jwillikers Mar 14, 2024

Choose a reason for hiding this comment

spiderkeys Mar 14, 2024 • edited Loading

Choose a reason for hiding this comment

jwillikers Mar 14, 2024

Choose a reason for hiding this comment

jwillikers Mar 14, 2024

Choose a reason for hiding this comment

jwillikers Mar 14, 2024

Choose a reason for hiding this comment

spiderkeys Mar 14, 2024

Choose a reason for hiding this comment

conan-center-bot commented Mar 14, 2024

Conan v1 pipeline ❌

Conan v2 pipeline ❌

spiderkeys commented Mar 14, 2024 • edited Loading

uilianries commented Aug 5, 2024

spiderkeys commented Mar 14, 2024 •

edited

Loading

spiderkeys Mar 14, 2024 •

edited

Loading

spiderkeys commented Mar 14, 2024 •

edited

Loading