KTOR-6754 Use static libcurl build with conan

[whyoleg]

Subsystem
ktor-client-curl

Motivation
Simplify depending on libcurl.
Similar to #3981 but with different way of building libcurl

Solution
libcurl was built on GitHub actions via Conan and produced libraries are copied.

I have only Mac and linking for all targets was successful. Probably some flags will need to be adopted during libcurl building so that all features are supported.

[whyoleg]

Some notes:

• windows with curl works fine! looks like other failures are unrelated. Also, I see, that Native on Windows x64 is not run be default. Is it expected? Looks like there is some issue with environ there and so ktor-server-core linking error happens
• linux: not sure, why there is an issue with linking there... I don't have linux PC at my side so can investigate only via CI. Also, for linux, it's possible to dynamically link to both curl and openssl, as it's probably will be available all the time, still, because of old GCC used in K/N it will be cumbersome to build final binary when users depend on ktor-client-curl - they will need curl and openssl compatible with old glibc :( That's why I decided to do static linking there
• macOS: no openssl is used at all! Conan allows to build curl with out-of-the-box support, so it's nice!

[whyoleg]

note: libz should be statically linked on mingw, without it, mingw build binaries will not work on Windows. Context: whyoleg/cryptography-kotlin#13

[e5l]

@whyoleg, could you check compilation errors on linux x64?

[e5l]

fails locally for linuxX64

[e5l]

let me check if I can fix it

[e5l]

It looks like the problem is with version of open SSL or build flags. I have found similar issues referring to the version bump:
https://stackoverflow.com/questions/46768071/openssl-linking-undefined-reference-evp-md-ctx-new-and-fre
https://stackoverflow.com/questions/42789278/undefined-reference-to-des-ecb-encrypt-md4-init-md4-update-and-md4-final

@whyoleg, could you tell me what versions and flags are used to build libssl?

[whyoleg]

could you tell me what versions and flags are used to build libssl?

Full information can be found in build log: /~https://github.com/whyoleg/openssl-builds/actions/runs/11719854910/job/32643991499
OpenSSL: 3.3.2
It was configure by running:

perl ./Configure "conan-Release-Linux-x86_64-gcc-8" no-shared --release --prefix=/ --libdir=lib --openssldir="/etc/ssl" threads PERL=perl no-unit-test no-tests -fPIC enable-fips no-md2 zlib --with-zlib-include="/home/runner/.conan2/p/b/zlib459920bf74d77/p/include" --with-zlib-lib="/home/runner/.conan2/p/b/zlib459920bf74d77/p/lib"

so mostly all default flags. Here are the options which are configurable with Conan.

I've also rerun the job to copy all transitive libraries (previously I've taken openssl from other build). Let's see if it will make any sense. UPDATE: no difference, same failure

Though, it's really strange, that it works locally on macOS. May be there is some native cache on CI?

[whyoleg]

@e5l I've disabled native caches via (kotlin.native.cacheKind.linuxX64=none) as suggested in error and now the linking issue is gone. So looks like this could be an issue at K/N side...

[e5l]

thanks, will check soon

[e5l]

Hey @olme04, lgtm! Could you rebase it on 3.1.0-eap?

[whyoleg]

@e5l I've rebased on 3.1.0-eap and also moved kotlin.native.cacheKind.linuxX64=none flag to ktor-client-curl module, so that it will affect only it and looks like it works fine.
I will create an issue on YT and link it in comment message here

[e5l]

Hey @whyoleg, thank you for the nice contribution! btw, it would be great to have an action in the Ktor repo, so we can regenerate dependencies from time to time

[whyoleg]

it would be great to have an action in the Ktor repo

Do you want to do it using GitHub Actions or TC pipeline? I could help to setup both, or you could try to do it on your own based on my setup :)
I think it will be easier to discuss this if you will approach me in Slack.

BTW: this PR fixes https://youtrack.jetbrains.com/issue/KTOR-6754/Support-static-linking-for-curl-on-all-platforms - is it possible to link it somehow after it was merged? :)
Also probably those issues should be marked as obsolete/fixed:

• https://youtrack.jetbrains.com/issue/KTOR-6173/Support-ktor-client-curl-for-linuxArm64 - it was fixed earlier and it's published
• https://youtrack.jetbrains.com/issue/KTOR-7385/Cant-build-ktor-server-curl-on-Apple-Silicon - should be not an issue anymore as we include static libraries
• https://youtrack.jetbrains.com/issue/KTOR-5894/Curl-Document-how-to-install-libcurl-on-Ubuntu-22.04 - no need to install libcurl on linux anymore
• https://youtrack.jetbrains.com/issue/KTOR-6935/cURL-Engine-Update-documentation-on-how-to-install-libcurl-for-different-Linux-distributions - same
• https://youtrack.jetbrains.com/issue/KTOR-6361/Curl-Error-linking-curl-in-linkDebugExecutableLinuxX64-on-macOS - works locally at my side with after PR
• https://youtrack.jetbrains.com/issue/KTOR-4307/Windows-undefined-symbols-in-linker-when-ktor-client-curl-is-used - should not be an issue anymore?
• https://youtrack.jetbrains.com/issue/KTOR-875/Client-with-custom-compiled-libcurl - probably out-dated as we include static lib on windows and there is no need to build it separately
• https://youtrack.jetbrains.com/issue/KTOR-4570/Support-ARM-target-in-Ktor-client-with-Kotlin-Native-and-Curl - fixed?

Probably there could be others, I've checked this YT query: https://youtrack.jetbrains.com/issues/KTOR?q=curl

[89jd]

Hi @e5l @whyoleg .

I have been away the past few weeks. This looks good!

I could have a look at setting up the pipeline to build curl if needed? Would be good to do this, as I started looking at getting curl with web sockets implemented too.

What do you think would be the best way to do it so that when the curl version changes it updates all the static libraries for all of the platforms?

[osipxd]

@whyoleg, @e5l could you check if this failure on CI is connected to this PR?

[whyoleg]

@osipxd yes, this is the same error which was before I disabled the caches there.
And last build for this branch was successful.
Also, probably update to Kotlin 2.1.0 could solve the issue (and cache flag could be removed) as I see no issues with this task in PR with Kotlin update

[dtretyakov]

@whyoleg thanks for your work. Is it possible to enable WebSockets in these curl builds?

[whyoleg]

@dtretyakov, yeah, sure: #4564
Tested, that at least some of the tests on macosArm64 for curl are working on your branch, so built libcurl is definitely supports web sockets:

[dtretyakov]

@whyoleg thank you for such a fast update.