typeerror in postprocessing

[dsm1212]

Headphones seems to have stopped processing for me. I use lidarr to find/download because it uses site specific categories and is way better at finding things. But headphones has been super at postprocessing. Something changed and it stopped working. I get this trace below. It's barfing because the '-' probably needs to be b'-'. But I don't see why it got to this point. The helper extract_data() should have worked before this to parse the filename. But it apparently failed. Maybe due to another byte/string mismatch? The exception is not logged. Did something change recently that caused extract_data to be messed up with byte/string handling?

2024-06-22 22:47:48 | ERROR | Uncaught exception: Traceback (most recent call last):File "/app/headphones/headphones/logger.py", line 215, in new_runold_run(*args, **kwargs)File "/usr/lib/python3.10/threading.py", line 953, in runself._target(*self._args, **self._kwargs)File "/app/headphones/headphones/postprocessor.py", line 1394, in forcePostProcessif '-' not in folder_basename:TypeError: a bytes-like object is required, not 'str'
-- | -- | --
2024-06-22 22:47:48 | DEBUG | Attempt to extract album name by assuming it is the folder name
2024-06-22 22:47:48 | INFO | Counted 0 media files, but only 12 have tags, ignoring.
2024-06-22 22:47:48 | DEBUG | Attempting to extract name, album and year from metadata
2024-06-22 22:47:48 | DEBUG | Attempting to extract name, album and year from folder name
2024-06-22 22:47:48 | DEBUG | Attempting to extract release group from folder name
2024-06-22 22:47:48 | DEBUG | Attempting to find album in the snatched table
2024-06-22 22:47:48 | INFO | Processing: b'BAND - ALBUM (2021)'

[AdeHub]

What indexers are you using where lidarr finds better results?

[dsm1212]

I dug into this a couple years ago, my recollection is fuzzy, but if I recall correctly there were two issues. First that lidarr supports picking categories for each indexer and second that it detects if the indexer supports album and artist as search parameters. I maybe misremembering this, but I'm certain that if I logged what comes back to headphones the match list was really poor, but the same album search in lidarr was all on point. In lidarr I've just got bitsearch, tpb, limetorrents, and solid torrents.

Do you know why extract_data() would be not working when the file name is matching? I put the regex into a python regex tester and it matches on the name fine. I'm thinking there are a couple of byte vs string problems in this code maybe?

[dsm1212]

I think extract_data() maybe has been broken for a while due to byte array and string mismatches. I don't have a setup to easily try a fix now that I'm using docker.

I took a quick look at lidarr. It calls the provider with ?t=caps and gets a response like this below. q for music-search might also indicate the indexer supports sending album and artist for query. This one below doesn't. But I think the thing that makes a big difference is the category selection in lidarr. It prompts the user with the category list to select which to use. You can see this one below has a custom category 104627 for music. Headphones I believe is hardcoding the standard categories. Could we change the jackett indexer config to let the user override the hardcoded list of category numbers for each indexer? Or maybe just to add additional category numbers.

<?xml version="1.0" encoding="UTF-8"?>
<caps>
  <server title="Jackett" />
  <limits default="100" max="100" />
  <searching>
    <search available="yes" supportedParams="q" />
    <tv-search available="yes" supportedParams="q,season,ep" />
    <movie-search available="yes" supportedParams="q" />
    <music-search available="yes" supportedParams="q" />
    <audio-search available="yes" supportedParams="q" />
    <book-search available="yes" supportedParams="q" />
  </searching>
  <categories>
    <category id="1000" name="Console" />
    <category id="2000" name="Movies" />
    <category id="3000" name="Audio" />
    <category id="4000" name="PC">
      <subcat id="4010" name="PC/0day" />
    </category>
    <category id="5000" name="TV">
      <subcat id="5070" name="TV/Anime" />
    </category>
    <category id="7000" name="Books">
      <subcat id="7020" name="Books/EBook" />
    </category>
    <category id="8000" name="Other" />
    <category id="146065" name="Anime" />
    <category id="151062" name="Applications" />
    <category id="121527" name="E-books" />
    <category id="136409" name="Games" />
    <category id="100467" name="Movies" />
    <category id="104627" name="Music" />
    <category id="127246" name="Other" />
    <category id="112972" name="TV shows" />
  </categories>
</caps>

[rembo10]

Do you mind testing the typeerror-fix branch?

[dsm1212]

Unfortunately it was not enough. I tried to keep plugging errors but kept hitting more and more.

postprocessor line 243 errors:
if file.endswith(media_extensions):
So I decoded file and root.

Then in metadata.py this line errored
if b"@HP@" in path:
needed the b added.

Then I got this error and gave up :-)

2024-06-25 22:34:00 | ERROR | Uncaught exception: Traceback (most recent call last):File "/app/headphones/headphones/logger.py", line 215, in new_runold_run(*args, **kwargs)File "/usr/lib/python3.10/threading.py", line 953, in runself._target(*self._args, **self._kwargs)File "/app/headphones/headphones/postprocessor.py", line 1324, in forcePostProcessverify(release['AlbumID'], folder, forced=True, keep_original_folder=keep_original_folder)File "/app/headphones/headphones/postprocessor.py", line 312, in verifydoPostProcessing(albumid, albumpath, release, tracks, downloaded_track_list, Kind,File "/app/headphones/headphones/postprocessor.py", line 482, in doPostProcessingaddAlbumArt(artwork, albumpath, release, metadata_dict)File "/app/headphones/headphones/postprocessor.py", line 703, in addAlbumArtwith open(os.path.join(albumpath, album_art_name), 'wb') as f:File "/usr/lib/python3.10/posixpath.py", line 90, in joingenericpath._check_arg_types('join', a, *p)File "/usr/lib/python3.10/genericpath.py", line 155, in _check_arg_typesraise TypeError("Can't mix strings and bytes in path components") from NoneTypeError: Can't mix strings and bytes in path components
-- | -- | --

[dsm1212]

@rembo10
I took another look and fixed it earlier where the folders list was created in postprocesser.py. The two decodes below added. Sorry I don't have a diff I just hacked my docker image and committed the change to a copy. I removed your change since this preceded it. Everything processed!

steve

               if expand_subfolders and subfolders is not None:  
                    folders.extend(subfolders.decode(headphones.SYS_ENCODING, 'replace'))
                else:
                    folders.append(path_to_folder.decode(headphones.SYS_ENCODING, 'replace'))

[dsm1212]

@rembo10 I submitted a pr #3342

[rembo10]

This is super awesome.... Thank you so much for doing that... Will get it merged in a few hours

[AdeHub]

@dsm1212, made a change to the Torznab search to use the parent3000 category instead of the sub categories and this is in develop if you want to test. Also changed it to allow manual entry of Prowlarr indexers

Using 3000 seems to work pretty weIl across the various providers I tested, e.g. the example you provided, 3000 is interchangeable with 104627, either giving the same results

[dsm1212]

@rembo10 Tried the update to include the 3000 parent category. It's definitely better. I set up the same search engines in both lidarr and headphones. I tried a half dozen searches and I thought I found some differences but it was due to size limits. So is looking good so far. thanks!

[dsm1212]

I'm having a new post processing problem though. Did one of these recent changes alter postprocessing to chmod the files before copying to the destination? It looks like the chmod is happening before the file copy now. This is a bad idea because the incoming files are owned by qbittorrent and the headphones user cannot chmod them. I'm sure it must have done the chmod after the copy previously because I was able to ingest files and now they all fail. I run both headphones and qbittorrent non-root maybe the change wasn't tested this way.

[dsm1212]

I can't for the life of me figure out why I never had this post processing chmod issue. I looked at the code and it appears to have always done a chmod on the incoming files if file_permissions_enabled is true. I looked even in a three year old backup of my config.ini and it had a 1 value. Maybe that was evaluated to false because it was not true? My current file had True. The code which invokes this seems to be 6 years old. It's bizarre. But I set the setting to false and is working again.

There is a bug in helpers.py though where it renames a file but then doesn't update the path variables that use the old file name. It comes up when processing an existing album. I'll create a separate bug and push a fix for it.

[dsm1212]

I believe you merged my fix and this can be closed.