Hashing audio files using audio data only

[sandreas]

Currently, I'm working on a little Web API + Android App for audio books (similar to audiobookshelf). It uses a file indexer to sync all audio files + metadata in a folder to a database.

One problem is, that if I move an audio file, I lose all stored data (playlists, ratings, collections, playcount, etc.), because I can't identify the file again, even if I only moved it to another folder in my library path.

One possible use case:

• An audio book has a title, but no series (e.g. /audiobooks/J. K. Rowling/Harry Potter and the Philosopher's Stone/Harry Potter and the Philosopher's Stone.m4b)
• Then a second part is released and now the series should be part of the audio book path, as well as of the metadata: (e.g. /audiobooks/J. K. Rowling/Harry Potter/1 - Harry Potter and the Philosopher's Stone.m4b)

The problem is, that if the filename AND the metadata changes, it is not possible to reidentify the file - even a file hash is no longer valid, because the metadata changed and so does the bytes in the file.

My current workaround is to store a specific library identifier as custom metadata field into the files, but if I would share a file with someone else, this may have side effects.

My idea now is to access the raw audio stream part without metadata fields and build a hash of that as identifier. The audiostream should not change in any case and so I would not lose my stored data.

Is it possible to access the audio stream only with atldotnet ignoring all non audio data?

[Zeugma440]

Hashing the raw audio is a good way to solve that indeed, unless your clients manually resample or reprocess their files for some obscure reason. You may use the FNV1a hash function as it is fast and has a reasonably low collision rate. ATL uses an implementation of that algorithm to hash embedded pictures.

As for your request, that's exactly what Track.TechnicalInfo.AudioDataOffset and Track.TechnicalInfo.AudioDataSize have been created for. These should do the trick.

Please keep me informed :)

[sandreas]

You may use the FNV1a hash function as it is fast and has a reasonably low collision rate.

Cool, I check this one... I thought about using blake3 for this, although crypto secure hash is not required. After reading some stuff, I think CrapWow, Crap8 or XXHash could be a better alternative. This XXHash implementation looks promising... or maybe this one or this one

I would use:

• Size of the audio data
• A "partial hash" taking 1MB in the middle of the audio data (start and end are often intros, outros or jingles)
• A "full hash" taking the whole audio data

Then match them one after each other to be sure, the file is the same.

As for your request, that's exactly what Track.TechnicalInfo.AudioDataOffset and Track.TechnicalInfo.AudioDataSize have been created for. These should do the trick.

This is awesome, I think I will implement this is my MetadataTrack accessing internal TechnicalInfo as follows:

• AudioDataSize - property
• ComputeQuickHash(HashAlgorithm)
• ComputeFullHash(HashAlgorithm)

Thank you for helping me out.

[Zeugma440]

After reading some stuff, I think CrapWow, Crap8 or XXHash could be a better alternative. This XXHash implementation looks promising... or maybe this one or this one

Awesome. I was about to say that you shouldn't use a cryptographic hash (blake3) if you don't need the crypto properties, but you apparently came to the same conclusion. Thanks for the links, I might replace FNV1A for XXHash if it passes the unit tests !

[sandreas]

xxHash works for me, see: /~https://github.com/sandreas/tonehub/blob/aa11b18bafc85bc4a26fe69360a4ab53d83cbf66/tonehub/Metadata/FileLoaderBase.cs#L14

        <PackageReference Include="Standart.Hash.xxHash" Version="4.0.4" />