Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sibnet.ru does not download #18182

Closed
6 of 9 tasks
LyoSU opened this issue Nov 13, 2018 · 6 comments
Closed
6 of 9 tasks

sibnet.ru does not download #18182

LyoSU opened this issue Nov 13, 2018 · 6 comments

Comments

@LyoSU
Copy link

LyoSU commented Nov 13, 2018

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like this: [x])
  • Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.11.07. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2018.11.07

Before submitting an issue make sure you have:

  • At least skimmed through the README, most notably the FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones
  • Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

youtube-dl https://video.sibnet.ru/rating/video3485877-Kot_i_nevidimaya_pregrada/%D1%8D/ -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://video.sibnet.ru/rating/video3485877-Kot_i_nevidimaya_pregrada/%D1%8D/', u'-v']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.11.07
[debug] Python version 2.7.15rc1 (CPython) - Linux-4.15.0-34-generic-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.4-0ubuntu0.18.04.1, ffprobe 3.4.4-0ubuntu0.18.04.1
[debug] Proxy map: {}
[generic] э: Requesting header
WARNING: Falling back on generic information extractor.
[generic] э: Downloading webpage
[generic] э: Extracting information
[generic] э: Downloading MPD manifest
WARNING: Failed to download MPD manifest: HTTP Error 403: Forbidden
[generic] э: Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://video.sibnet.ru/shell.php?videoid=3485877'
[download] Destination: Кот и невидимая преграда-shell.php
[download] 100% of 9.52KiB in 00:00

Video from the site is loaded as php instead of the video itself. This is clearly visible on the penultimate line. I saw that you were previously addressed with this problem, but the question was supposedly closed by a duplicate. The problem still exists, tested on version 2018.11.07

@LyoSU LyoSU changed the title .sibnet.ru does not download sibnet.ru does not download Nov 13, 2018
@ghost
Copy link

ghost commented Nov 27, 2018

I can confirm this with version 2018.11.23, looks like they updated the player data, now it doesn't include the key and other parameters,
e.g.:

https://dv25.sibnet.ru/32/29/93/3229930.m3u8?st=MXCJ_3JuXXVBtPKw2_Tvbw&e=1543369000&stor=25&noip=1&start=

became:

https://video.sibnet.ru/v/5be6d130afd927ae6241de07cdd78f12/3464673.m3u8

And without these keys it returns 403 - forbidden.

@ghost
Copy link

ghost commented Nov 28, 2018

Using the headers solves it for me, here bash script:

ydl-sib

#!/bin/bash
url="$1"
shift
youtube-dl --add-header "Host: video.sibnet.ru" \
--add-header "User-Agent: Mozilla/5.0 (Linux; Android 4.4.2; Nexus 4 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.114 Mobile Safari/537.36" \
--add-header "Accept: */*" \
--add-header "Referer: $url" \
--add-header "Range: bytes=0-" "$url" "$@"

use it like this:

ydl-sib 'https://video.sibnet.ru/rating/video3485877-Kot_i_nevidimaya_pregrada/%D1%8D/'

Successfully downloads, outputs:

[generic] э: Requesting header
WARNING: Falling back on generic information extractor.
[generic] э: Downloading webpage
[generic] э: Extracting information
[download] Destination: Кот и невидимая преграда - видео ролик смотреть на Video.Sibnet.Ru-э.mp4
[download] 100% of 514.57KiB in 00:02

@ghost
Copy link

ghost commented Feb 25, 2019

I created some bash script which retrieves direct URL and video title from Sibnet video URL: (codename cusi, from CUrl and SIbnet)

cusi

Updated (2019-02-25 14:47:54):

  • Added Video-Id
  • Fixed Video-Title including page title.

Updated (2019-02-25 17:01:38):

  • Added eval mode.
  • Added info.

Updated (2019-02-25 17:19:11):

  • Added safe title to eval mode.
#!/bin/bash
if [[ "$1" == "" ]]; then
  echo "     cusi By BladeMight, v1.0"
  echo "$0 <sibnet-url> <*commands-str>"
  echo "In command str you can use variables:"
  echo " \\\$du = direct url"
  echo " \\\$ti = title"
  echo " \\\$id = id"
  echo " \\\$au = accept url"
  echo "You must escape variables, so they can be processed in code!, Or you can use single quotes ''"
  echo "command str examples:"
  echo "  cusi <url> 'aria2c \"\$du\" -o \"\$ti\"'"
  echo "  cusi <url> 'ffmpeg -i \"\$du\"'"
  exit 0
fi
url="$1"
H1="Accept-Encoding: identity;q=1, *;q=0"
H2="Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7,zh-CN;q=0.6,zh;q=0.5"
H3="User-Agent: Mozilla/5.0 (Linux; Android 4.4.2; Nexus 4 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.114 Mobile Safari/537.36"
H4="chrome-proxy: frfr"
H5="Accept: */*"
H6="Connection: keep-alive"
H7="Range: bytes=0-"
H8="Referer: $url"
resp=$(curl -s -H "$H1" -H "$H2" -H "$H3" -H "$H4" -H "$H5" -H "$H6" -H "$H7" --compressed "$url" | iconv -f cp1251 -t utf-8 )
accepturl=$(echo "$resp" | grep -iP '\d+\.mp4' | sed -re 's/.*(\/v\/.*?mp4)", type.*/https:\/\/video.sibnet.ru\1/g')
id=$(echo "$resp" | grep -i DOCTYPE | sed -re 's/.*php\?videoid=([0-9]*).*/\1/g')
echo "Video-Id: $id"
title=$(echo "$resp" | grep -iP 'videoname' | sed -re 's/.*?videoName.>(.*?)<\/h1>.*/\1/g')
echo "Video-Title: $title"
echo "Accept-URL: $accepturl"
directurl=$(curl -I "$accepturl" -s -H "$H1" -H "$H2" -H "$H3" -H "$H4" -H "$H5" -H "$H6" -H "$H7" -H "$H8" --compressed | awk 'BEGIN{IGNORECASE=1} match($0, /location: (.*)/, z) {printf "https:%s", z[1]}')
echo "Direct-URL: $directurl"
if [[ "$2" != "" ]]; then
  echo "Eval mode."
  au="$accepturl"
  du="$directurl"
  ti=$(echo "$title" | sed -re 's/\\|\/|\*|:|\"|<|>|\?|\|//g') # safe title
  eval $2
fi

Use it like this:

$ cusi 'https://video.sibnet.ru/rating/video3485877-Kot_i_nevidimaya_pregrada/%D1%8D/'
Video-Id: 3485877
Video-Title: Кот и невидимая преграда
Accept-URL: https://video.sibnet.ru/v/b191e90609fb5399adb254d0b25d84e5/3485877.mp4
Direct-URL: https://dv98.sibnet.ru/32/46/57/3246579.mp4?st=4OqCJPY1IMlq4gxP4VzYrA&e=1551128000&stor=26&noip=1

The trick is to use Android user agent, then in the Sibnet video player there is Mp4 Direct Url, then you just need to GET it with correct headers.

And you can use it with aria2c(or just modify the script):

cusi 'https://video.sibnet.ru/rating/video3485877-Kot_i_nevidimaya_pregrada/%D1%8D/'  | awk '/Title:/{ $1=""; TITLE=substr($0, 2, length($0))}/Direct-URL:/{ $1=""; URL=substr($0,2,length($0)) } END { gsub(/\\|\/|\*|:|\"|<|>|\?|\|/, "", TITLE); printf "aria2c \"%s\" -o \"%s.mp4\"", URL, TITLE}' | sh

With command str mode, you can use even it simpler:

cusi 'https://video.sibnet.ru/rating/video3485877-Kot_i_nevidimaya_pregrada/%D1%8D/' 'aria2c "$du" -o "$ti"'

or even directly to player, e.g. ffplay:

cusi 'https://video.sibnet.ru/rating/video3485877-Kot_i_nevidimaya_pregrada/%D1%8D/' 'ffplay -i "$du"'

run cusi without arguments to see help.

@bato3
Copy link
Contributor

bato3 commented Mar 2, 2019

one header what you need is "Referer

youtube-dl https://video.sibnet.ru/shell.php?videoid=3422904 --add-header "Referer: https://video.sibnet.ru/shell.php?videoid=3422904" 

btw @BladeMight in current cusi you remove 2 first lines and I have error:
awk: line 1: syntax error at or near ,

@ghost
Copy link

ghost commented Mar 2, 2019

@bato3 Somehow two lines were out in cusi...(i edited the post) Btw, I created repository: BladeMight/cusi, and added some other scripts.
I measured the Sibnet download speed of youtube-dl's (MPD/m3u8) and cusi's (direct mp4) and seems like cusi + aria2c is 6-8 faster:

  • youtube-dl - 100~500KB
  • cusi+aria2c- ~2000KB(my maximum speed)
Tests
  • time youtube-dl https://video.sibnet.ru/shell.php?videoid=3422904 --add-header "Referer: https://video.sibnet.ru/shell.php?videoid=3422904"
[generic] shell: Requesting header
WARNING: Falling back on generic information extractor.
[generic] shell: Downloading webpage
[generic] shell: Extracting information
[generic] shell: Downloading MPD manifest
[generic] shell: Downloading m3u8 information
[dashsegments] Total fragments: 9
[download] Destination: Доставщик пиццы спросил разрешения сыграть на фортепиано-shell.fdash-v1-x3.mp4
[download] 100% of 5.21MiB in 00:10
[dashsegments] Total fragments: 9
[download] Destination: Доставщик пиццы спросил разрешения сыграть на фортепиано-shell.fdash-a1-x3.m4a
[download] 100% of 862.29KiB in 00:07
[ffmpeg] Merging formats into "Доставщик пиццы спросил разрешения сыграть на фортепиано-shell.mp4"
Deleting original file Доставщик пиццы спросил разрешения сыграть на фортепиано-shell.fdash-v1-x3.mp4 (pass -k to keep)
Deleting original file Доставщик пиццы спросил разрешения сыграть на фортепиано-shell.fdash-a1-x3.m4a (pass -k to keep)

real    0m27.466s
user    0m0.031s
sys     0m0.015s
  • time cusi 'https://video.sibnet.ru/shell.php?videoid=3422904' "#aria"
Video-Id: 3422904
Video-Title: Доставщик пиццы спросил разрешения сыграть на фортепиано
Accept-URL: https://video.sibnet.ru/v/ac3b4b2ebbab7610e5fee5a4b43be09f/3422904.mp4
Direct-URL: https://dv98.sibnet.ru/31/99/63/3199632.mp4?st=1RP51i0cM7pf2iByV3X6xg&e=1551548000&stor=9&noip=1
Eval mode: [#aria]

03/02 11:29:50 [NOTICE] Downloading 1 item(s)
[#cfd09d 5.6MiB/6.0MiB(92%) CN:4 DL:1.9MiB]
03/02 11:29:54 [NOTICE] Download complete: C:/msu/Доставщик пиццы спросил разрешения сыграть на фортепиано.mp4

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
cfd09d|OK  |   1.9MiB/s|C:/msu/Доставщик пиццы спросил разрешения сыграть на фортепиано.mp4

Status Legend:
(OK):download completed.

real    0m7.273s
user    0m0.242s
sys     0m0.394s

About the headers, to get mp4 (not m3u8/mpd) you need the header of UA of Android.

@professeurjt

This comment was marked as spam.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants