ytdl-raw-options can't pass multiple http headers

[Mikle-Bond]

mpv version and platform

Arch Linux
mpv 0.29.1 Copyright © 2000-2018 mpv/MPlayer/mplayer2 projects
built on Sun Dec 9 16:21:27 CET 2018

Reproduction steps

Side story: I was trying to workaround problems with sibnet.ru videos using this suggestion:
ytdl-org/youtube-dl#18182 (comment)
It works with youtube-dl, so I tried passing those headers to mpv.

Example that sets two headers:

mpv  '--ytdl-raw-options=add-header="'"'"'Foo: bar'"'"'",add-header="'"'"'Hello: world'"'"'"' "$url"

Expected behavior

Multiple headers are passed to ytdl_hook.

Actual behavior

Only one header is passed. I assume it is due to options being stored in dictionary, and multiple values with same key override each other.

Log file

$ mpv --msg-level=ytdl_hook=debug '--ytdl-raw-options=add-header="'"'"'Foo: bar'"'"'",add-header="'"'"'Hello: world'"'"'",verbose=' "$url"

[ytdl_hook] Loading lua script @ytdl_hook.lua...
[ytdl_hook] loading mp.defaults
[ytdl_hook] loading @ytdl_hook.lua
[ytdl_hook] reading options for ytdl_hook 
[ytdl_hook] script-opts/ytdl_hook.conf not found. 
[ytdl_hook] lua-settings/ytdl_hook.conf not found. 
Playing: http://video.sibnet.ru/shell.php?videoid=2850416
[ytdl_hook] ytdl:// hook 
[ytdl_hook] not a ytdl:// url 
[ytdl_hook] full hook 
[ytdl_hook] Running: youtube-dl --no-warnings -J --flat-playlist --sub-format ass/srt/best --format bestvideo[height<=?720][fps<=?30]+bestaudio/best --add-header 'Hello: world' --verbose --all-subs --no-playlist -- http://video.sibnet.ru/shell.php?videoid=2850416 
[ytdl_hook] [debug] System config: []
[ytdl_hook] [debug] User config: ['--cookies', '~/.local/share/youtube-dl/cookies.txt', '--netrc']
[ytdl_hook] [debug] Custom config: []
[ytdl_hook] [debug] Command-line args: ['--no-warnings', '-J', '--flat-playlist', '--sub-format', 'ass/srt/best', '--format', 'bestvideo[height<=?720][fps<=?30]+bestaudio/best', '--add-header', "'Hello: world'", '--verbose', '--all-subs', '--no-playlist', '--', 'http://video.sibnet.ru/shell.php?videoid=2850416']
[ytdl_hook] [debug] Adding header from command line option 'Hello: world'
[ytdl_hook] [debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[ytdl_hook] [debug] youtube-dl version 2019.02.08
[ytdl_hook] [debug] Python version 3.7.2 (CPython) - Linux-4.20.1-arch1-1-ARCH-x86_64-with-arch
[ytdl_hook] [debug] exe versions: ffmpeg 4.1, ffprobe 4.1, phantomjs 2.1.1, rtmpdump 2.4
[ytdl_hook] [debug] Proxy map: {}
[ytdl_hook] youtube-dl succeeded! 
[ytdl_hook] ytdl parsing took 0.002199 seconds 
[ytdl_hook] No fragments to join into EDL 
[ytdl_hook] streamurl: https://video.sibnet.ru/shell.php?videoid=2850416 
[ytdl_hook] script running time: 0.003048 seconds 
Failed to recognize file format.


Exiting... (Errors when loading file)
[ytdl_hook] Exiting...

[garoto]

Try add-header-append for the second one.

Oops, I was actually thinking of --ytdl-raw-options-append, but that doesn't appear to work either.

[Mikle-Bond]

I'm afraid, youtube-dl doesn't have that one. I even bothered re-checking:

...
[ytdl_hook] youtube-dl: error: no such option: --add-header-append
[ytdl_hook] youtube-dl failed: unexpected error ocurred 

MPV has --ytdl-raw-options-append, and I tried it too. Nothing changes.

But even if such options existed, it's not a solution, 'cos I need to pass more or less 5 of them.

mpv '--ytdl-raw-options='`
	`'add-header="'"'"'Host: video.sibnet.ru'"'"'",'`
	`'add-header="'"'"'User-Agent: '"$useragent'"'",'`
	`'add-header="'"'"'Accept: */*'"'"'",'`
	`'add-header="'"'"'Referer: '"$URL'"'",'`
	`'add-header="'"'"'Range: bytes=0-'"'"'",'`
	`'verbose='` ` "$URL"

Seems like ether mpv should treat those options not as key-value-dictionary, or make an exception for add-header option, like it does for ytdl-format. Or maybe, pass headers from --http-header-fields, though it seems like a bad idea, because proxy settings aren't being forwarded, so shouldn't headers be forwarded.

Or at least it should be mentioned in manpages, I think.

[disco0]

You should be able to configure those headers/proxy settings/etc together in a conf file for youtube-dl, right? If you're able to do it in ytdl, just pass that conf file into mpv's raw options instead of everything separate—I had some issues getting my settings working across OSes until I got a config loading and its been smooth since.

Example sourcing the ytdl conf:

# Youtube-dl can locate this path
ytdl-raw-options=config-location="~//.config//mpv//youtube-dl.conf"
# Not on mpv's interpolated paths though
#  ytdl-raw-options=config-location="~~//youtube-dl.conf"

And here's my best guess on how to do the headers:

# idk if the header quotation/backticking was functional or just ASCII art
--add-header Host: video.sibnet.ru
--add-header User-Agent: $useragent
--add-header Accept: */*
--add-header Referer: $URL
--add-header Range: bytes=0-

# You'll still get verbose output to the terminal from youtube-dl
# while running in mpv, might be useful if you're still having 
# issues later
# (also running mpv w/ --msg-level=all=fatal,ytdl_hook=trace )

--verbose 

[KebabLord]

^the above config has problems with parsing, this one worked

--add-header "Host: video.sibnet.ru"
--add-header "User-Agent: Mozilla/5.0 (Linux; Android 4.4.2; Nexus 4 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.114 Mobile Safari/537.36"
--add-header "Accept: */*"
--add-header "Referer: https://video.sibnet.ru/"
--add-header "Range: bytes=0-"

# Usage
#mpv --ytdl-raw-options=config-location="/home/admin/Scripts/ytdl.conf"  <url>
#youtube-dl --config-location ytdl.conf  <url>

[Akemi]

seems like this can be worked around.