Crash with OpenSSL 3 on arm32

[steveadvanced]

I'm trying to get the Speech SDK working in our environment, a Raspberry PI 4 running a buildroot Linux OS. I had it working a year or so ago before we put the project aside, but since then we've upgraded to OpenSSL 3 (3.3.2 on my test machine) and I now get a ConnectionFailure error.

I've reproduced this in the C# helloworld sample fresh from the git repo (/~https://github.com/Azure-Samples/cognitive-services-speech-sdk/tree/master/quickstart/csharp/dotnetcore/from-microphone), and linked against Microsoft.CognitiveServices.Speech version 1.42.0. Changing only the subscription keys I get the following output when I launch it:

Say something...
CANCELED: Reason=Error
CANCELED: ErrorCode=ConnectionFailure
CANCELED: ErrorDetails=Failed to get HTTP platform singleton instance. Error: 27 SessionId: a21ff6c84d5d405386b5df634616b716
CANCELED: Did you update the subscription info?
Please press <Return> to continue.

The same sample run locally on Windows works exactly as it should, so it isn't the subscription details at fault.

After enabling logging, I see the following in the log:

[784706]: 347ms SPX_DBG_TRACE_SCOPE_EXIT:  audio_stream_session.cpp:1725 DispatchEvent task complete!
[897856]: 347ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:131 Created 'CSpxUspConnection' as '792041254'
[897856]: 347ms SPX_TRACE_VERBOSE:  pal_create_module_object.cpp:98 Trying to load libssl.so.3
[897856]: 348ms SPX_TRACE_VERBOSE:  pal_create_module_object.cpp:112 Creating CSpxDyanmic for libpal_azure_c_shared_openssl3.so.
[897856]: 348ms SPX_TRACE_VERBOSE:  pal_create_module_object.cpp:121 Loaded libpal_azure_c_shared_openssl3.so, getting PAL method.
[897856]: 348ms SPX_TRACE_VERBOSE:  dynamic_module.cpp:53 Loading 'libpal_azure_c_shared_openssl3.so'
[897856]: 348ms SPX_TRACE_VERBOSE:  dynamic_module.cpp:55 dlopen('libpal_azure_c_shared_openssl3.so') returned non-NULL
[897856]: 348ms SPX_TRACE_VERBOSE:  dynamic_module.cpp:61 dlsym('pal_get_value') returned non-NULL
[897856]: 348ms SPX_TRACE_VERBOSE:  pal_create_module_object.cpp:67 Loaded PAL method, calling.
[897856]: 348ms SPX_TRACE_VERBOSE:  pal_create_module_object.cpp:132 Initializing libpal_azure_c_shared_openssl3.so.
[897856]: 348ms SPX_TRACE_ERROR: AZ_LOG_ERROR:  shim_openssl.c:55 libssl could not be loaded
[897856]: 348ms SPX_TRACE_ERROR: AZ_LOG_ERROR:  tlsio_openssl.c:2175 Could not load libssl
[897856]: 349ms SPX_TRACE_ERROR:  exception.cpp:132 About to throw Runtime error: Failed to initialize platform (azure-c-shared). Error: 2176

The full log is attached. It looks like the log is saying it failed to find openssl, even though it successfully determined to use the openssl 3 library.

log.txt

This seems very similar to the closed issue #2420 for arm64 - could the same thing have cropped up in the arm32 build? I don't have a tool on my target system to check exactly what it's trying to link against, but I do see mention in libpal_azure_c_shared_openssl3.so of libssl.so.1.1. The arm64 version of the same library has libssl.so.3 in the same place in the file.

[pankopon]

Hi, this is with Raspberry Pi OS or a custom system based on that? Please try with a supported distribution:
https://learn.microsoft.com/azure/ai-services/speech-service/quickstarts/setup-platform?tabs=linux#platform-requirements
https://ubuntu.com/tutorials/how-to-install-ubuntu-desktop-on-raspberry-pi-4
https://cdimage.ubuntu.com/releases/24.04.1/release/

(The Speech SDK releases for Linux arm32 and arm64 are tested on Ubuntu.)

[steveadvanced]

I've done the same with Ubuntu 22.024 on arm32 and get the same results.

Starting from a clean Ubuntu install, and installing the prerequisites as below, with the config.txt tweaked to enable my microphone.

OS: Ubuntu 22.04.5 LTS

$ uname -a
Linux ubuntu 5.15.0-1061-raspi #64-Ubuntu SMP PREEMPT Wed Aug 7 14:42:42 UTC 2024 armv7l armv7l armv7l GNU/Linux

Hardware: PI CM4

Package: Microsoft.CognitiveServices.Speech version 1.42.0

Sample code: /~https://github.com/Azure-Samples/cognitive-services-speech-sdk/tree/master/quickstart/csharp/dotnetcore/from-microphone

Prerequisites installed:

sudo apt-get update
sudo apt-get install build-essential ca-certificates libasound2-dev libssl-dev wget

Output:

Say something...
CANCELED: Reason=Error
CANCELED: ErrorCode=ConnectionFailure
CANCELED: ErrorDetails=Failed to get HTTP platform singleton instance. Error: 27 SessionId: 6e42094422fd4d2eab76379af6e66c36
CANCELED: Did you update the subscription info?
Please press <Return> to continue.

Log (full log attached):

[862898]: 177ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:131 Created 'CSpxUspConnection' as '792041254'
[862898]: 177ms SPX_TRACE_VERBOSE:  pal_create_module_object.cpp:98 Trying to load libssl.so.3
[862898]: 177ms SPX_TRACE_VERBOSE:  pal_create_module_object.cpp:112 Creating CSpxDyanmic for libpal_azure_c_shared_openssl3.so.
[862898]: 177ms SPX_TRACE_VERBOSE:  pal_create_module_object.cpp:121 Loaded libpal_azure_c_shared_openssl3.so, getting PAL method.
[862898]: 177ms SPX_TRACE_VERBOSE:  dynamic_module.cpp:53 Loading 'libpal_azure_c_shared_openssl3.so'
[862898]: 177ms SPX_TRACE_VERBOSE:  dynamic_module.cpp:55 dlopen('libpal_azure_c_shared_openssl3.so') returned non-NULL
[862898]: 177ms SPX_TRACE_VERBOSE:  dynamic_module.cpp:61 dlsym('pal_get_value') returned non-NULL
[862898]: 177ms SPX_TRACE_VERBOSE:  pal_create_module_object.cpp:67 Loaded PAL method, calling.
[862898]: 177ms SPX_TRACE_VERBOSE:  pal_create_module_object.cpp:132 Initializing libpal_azure_c_shared_openssl3.so.
[862898]: 177ms SPX_TRACE_ERROR: AZ_LOG_ERROR:  shim_openssl.c:55 libssl could not be loaded
[862898]: 178ms SPX_TRACE_ERROR: AZ_LOG_ERROR:  tlsio_openssl.c:2175 Could not load libssl
[862898]: 178ms SPX_TRACE_ERROR:  exception.cpp:132 About to throw Runtime error: Failed to initialize platform (azure-c-shared). Error: 2176 

log-ubuntu.txt

This is on a system with only OpenSSL 3 installed:

$ openssl version
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

Not very scientific, but there is a reference to openssl 1 in the library, where the x64 file references openssl 3:

$ cat speech/runtimes/linux-arm/native/libpal_azure_c_shared_openssl3.so | grep libssl.so.1.1
grep: (standard input): binary file matches
$ cat speech/runtimes/linux-arm/native/libpal_azure_c_shared_openssl3.so | grep libssl.so.3

$ cat speech/runtimes/linux-arm64/native/libpal_azure_c_shared_openssl3.so | grep libssl.so.1.1
$ cat speech/runtimes/linux-arm64/native/libpal_azure_c_shared_openssl3.so | grep libssl.so.3
grep: (standard input): binary file matches

[pankopon]

Thank you for reporting the issue. We have confirmed the bug on Linux arm32, and the fix will be included in the Speech SDK 1.43.0 release. This should be available in early March.