FAT_RUNTIME, shared library only builds, and various other distro related requirements.

[jlinton]

I've been looking at packaging this project on fedora/rhel/epel, and it seems many of the x86 options in place for distro's (hyperscan is part of fedora/etc) don't work on aarch64/ppc. Beyond what others have noted about the signed/unsigned char/colm issues:

For starters the expectation is that something like the FAT_RUNTIME option in the makefile generates binaries with a baseline (say armv-8a, without any optional arch extensions) that runs regardless of the HW, and at runtime selects neon/SVE/SVE2 paths if the HW supports it. Since most distro's also build nativly, the CMake file needs to be aware of this and honor the baseline compiler options (which tend to also include things like -mbranch_protection/etc) only extending them as needed rather than trying to do -march/mtune=native options because the build machine won't be related to the install machine.

Also, most distro's frown on static libraries, but the unit test won't compile without them (-DBUILD_SHARED_LIB:BOOL=ON). There is also a problem on aarch64 platforms where the exception stack size is a TLS variable but won't be fixed at buildtime (this is generic to hyperscan as well AFAIK). There are various other lower priority issues, like the documentation isn't installed, the project tosses a LOT of warnings building on aarch64, the c++17 requirement means it won't build with the system compiler on older distros, etc.

Basically, I've spent a couple days trying to create a generic package on aarch64 and it looks like it needs more work beyond the hacky patches I've applied to force it to build in a single case.

[markos]

first of all, thank you for your packaging effort to include Vectorscan in RH distros. Now to answer your points.
FAT_RUNTIME is almost irrelevant in non-x86 platforms. On x86, there are many SIMD engines available, but there is no guarantee that the newer revision will perform better than the previous, eg AVX512 is not necessarily faster than AVX2 in some cases. So a way to choose the best at runtime had to be implemented. Now Arm does not yet have that problem, NEON is assumed and SVE/SVE2 support is at its infancy. We do plan to offer runtime detection on Arm as well, but I would have to have proper hardware to test this on, which is just becoming available. It is a feature that will be done, just not yet. For the time being, you can just assume FAT_RUNTIME=off on non-x86 platforms. Please check the debian/rules file in the debian branch for how we build the package on other platforms.

Regarding static libraries, I disagree with your assumption about other distros. Traditionally, in Debian at least, the -dev packages almost always include the static libraries, so you do you have to build those at package creation time, and use those to run the unit tests. Again I would suggest you check the debian/rules file to see how it's done there.

Support for older distros is not something we are looking to provide, so the C++17 requirement is not going to go, there were some features that we had to use hence the requirement. C++17 support is at least 5 years old now so even LTS versions support it just fine.

Now regarding TLS and other lower priority issues on aarch64, that is something we're going to work on and quality will improve gradually. We expect to pick up pace soon.

Having said that, I would be more than willing to discuss and merge any fixes that help towards enabling the building of the packages in RH distros.

[jlinton]

The system compiler on RHEL8 is GCC8 (released almost exactly 4 years ago), and it has extended support until at least 2031. That was the most recent RHEL/rocky/alma/centos Linux version until the RHEL9 GA yesterday. So, it is the most common RHEL/etc install on !x86 architectures. For x86, RHEL6 and 7 are still supported "LTS" distros and they utilize even older system compilers, although it is probably safe to ignore them because people won't be rolling out new software on stuff that old.

So, this library doesn't compile against the most popular EPEL version at this time.

Finally, it is great that debian allows a package with static libraries, but that isn't the case for the redhat lineage distros (they aren't the only ones) where static libraries are considered a security/maintence problem because the distro has to then rebuild/etc dependent packages everytime a patch is applied to a library. Which for a distro like RHEL isn't even possible as a large fraction of the software running on it is 3rd party commercial/closed source.

And we are at the cusp of SVE being widely available outside of HPC, which is sorta a big deal for the types of machines one expects RHEL to run on (RHEL8 has been been tested/running on a64fx based machines for a couple years now).

Anyway, the copr builds are here: https://copr.fedorainfracloud.org/coprs/jlinton/vectorscan/ but they are hacked in a way that I wouldn't imagine is appropriate for submission to this repo, and given some of those hacks, probably aren't appropriate for merge into the fedora/epel repos yet either.

[markos]

Ok, I didn't realize RHEL have such huge lifespans. I do understand the importance of having software installable in the most popular release, so what I can do is promise to look about lowering the C++ standard or at least allow fallbacks.

Now revisiting static libraries, you misunderstood me, Debian does not easily allow static libraries to be used as dependencies to other packages. But it does allow -dev packages to include static libraries for development purposes. So, in most packages -with some exceptions- static libraries are built during packaging, and this is how it's possible to run unit tests. Well it is also possible to do that with shared libs only, but requires some tampering with ldconfig.

Regarding SVE/SVE2 support and runtime detection, as I said, I would love to have it, but not before I acquire access to one such system, which I am expecting to happen soon.

[markos]

This will be worked on for the next version, fat runtime for Aarch64 that is.