Here is the task: several users need to watch a video on YouTube simultaneously, with latency as low as possible.

Video as a stream

Apparently, if each of spectators just starts playing a video, the stated goal cannot be achieved, because one user will receive the video faster, will the other will get it slower. And this gap is hardly controllable.

In order to eliminate this divergence, the video should be delivered to all spectators simultaneously. This can be achieved by enveloping the video to a Live-stream. Below is the description of how you can do this using a tandem of this library with ffmpeg.

Youtube video as stream

We need to implement the diagram presented above. Specifically, ydl connects to YouTube and starts downloading the video. ffmpeg grabs the video being downloaded, envelops it to an RTMP stream and sends to the server. The server broadcasts the received stream as WebRTC in real time.

 

Installing youtube-dl

First of all, we install youtube-dl. The installation process on Linux is extremely simple and is described thoroughly in Readme for Linux and for Win.

1. Download.

curl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl

2. Modify permissions to allow execution

chmod a+rx /usr/local/bin/youtube-dl

That’s it. YouTube downloader is ready for work.

Now, take a video from YouTube and see its metadata:

youtube-dl --list-formats https://www.youtube.com/watch?v=9cQT4urTlXM

The result is:

[youtube] 9cQT4urTlXM: Downloading webpage
[youtube] 9cQT4urTlXM: Downloading video info webpage
[youtube] 9cQT4urTlXM: Extracting video information
[youtube] 9cQT4urTlXM: Downloading MPD manifest
[info] Available formats for 9cQT4urTlXM:
format code  extension  resolution note
171          webm       audio only DASH audio    8k , vorbis@128k, 540.24KiB
249          webm       audio only DASH audio   10k , opus @ 50k, 797.30KiB
250          webm       audio only DASH audio   10k , opus @ 70k, 797.30KiB
251          webm       audio only DASH audio   10k , opus @160k, 797.30KiB
139          m4a        audio only DASH audio   53k , m4a_dash container, mp4a.40.5@ 48k (22050Hz), 10.36MiB
140          m4a        audio only DASH audio  137k , m4a_dash container, mp4a.40.2@128k (44100Hz), 27.56MiB
278          webm       256x144    144p   41k , webm container, vp9, 30fps, video only, 6.54MiB
242          webm       426x240    240p   70k , vp9, 30fps, video only, 13.42MiB
243          webm       640x360    360p  101k , vp9, 30fps, video only, 20.55MiB
160          mp4        256x144    DASH video  123k , avc1.4d400c, 15fps, video only, 24.83MiB
134          mp4        640x360    DASH video  138k , avc1.4d401e, 30fps, video only, 28.07MiB
244          webm       854x480    480p  149k , vp9, 30fps, video only, 30.55MiB
135          mp4        854x480    DASH video  209k , avc1.4d401f, 30fps, video only, 42.42MiB
133          mp4        426x240    DASH video  274k , avc1.4d4015, 30fps, video only, 57.63MiB
247          webm       1280x720   720p  298k , vp9, 30fps, video only, 59.25MiB
136          mp4        1280x720   DASH video  307k , avc1.4d401f, 30fps, video only, 62.58MiB
17           3gp        176x144    small , mp4v.20.3, mp4a.40.2@ 24k
36           3gp        320x180    small , mp4v.20.3, mp4a.40.2
43           webm       640x360    medium , vp8.0, vorbis@128k
18           mp4        640x360    medium , avc1.42001E, mp4a.40.2@ 96k
22           mp4        1280x720   hd720 , avc1.64001F, mp4a.40.2@192k (best)

Installing ffmpeg

Then, we install ffmpeg using typical spells:

wget http://ffmpeg.org/releases/ffmpeg-3.3.4.tar.bz2
tar -xvjf ffmpeg-3.3.4.tar.bz2
cd  ffmpeg-3.3.4
./configure --enable-shared --disable-logging --enable-gpl --enable-pthreads --enable-libx264 --enable-librtmp
make
make install

Make sure everything is ok:

ffmpeg -v

Now, the most interesting part. The youtube-dl library is intended for downloading. That’s where the name comes from. This means you can download a YouTube video completely and then stream it via ffmpeg as a file.

But imagine this use case first. A web conference with three participants: a marketer, a manager and a programmer. The marketer decides to play a video from Youtube in real time to other conference participants. The video is 300 Mbs, and that’s a little bit embarrassing.

  1. The marketer says: “Now, buys let’s watch this pussycat video as it precisely displays our marketing strategy” and clicks “Share the video”.
  2. A preloader appears on the screen saying something like “The pussycat video is being downloaded now. This will take less than 10 minutes”.
  3. The manager takes a coffee break, and the programmer goes to reddit.

 

Apparently, asking people to wait is bad for business, so we need real-time. Specifically, we need to grab the video directly while playing, encapsulate it to a stream on the fly and broadcast it in real time. Below is how this can be done.

 

Transferring data from youtube-dl to ffmpeg

The youtube-dl grabber saves the stream in the file system. We need to connect to this stream and read from the file using ffmpeg while youtube-dl keeps downloading new chunks.

To merge these two processes – downloading and ffmpeg streaming – together, we need a simple script.

#!/usr/bin/python

import subprocess
import sys

def show_help():
    print 'Usage: '
    print './streamer.py url streamName destination'
    print './streamer.py https://www.youtube.com/watch?v=9cQT4urTlXM streamName rtmp://192.168.88.59:1935/live'
    return

def streamer() :
    url = sys.argv[1]
    if not url :
        print 'Error: url is empty'
        return
    stream_id = sys.argv[2]
    if not stream_id:
        print 'Error: stream name is empty'
        return
    destination = sys.argv[3]
    if not destination:
        print 'Error: destination is empty'
        return

    _youtube_process = subprocess.Popen(('youtube-dl','-f','','--prefer-ffmpeg', '--no-color', '--no-cache-dir', '--no-progress','-o', '-', '-f', '22/18', url, '--reject-title', stream_id),stdout=subprocess.PIPE)
    _ffmpeg_process = subprocess.Popen(('ffmpeg','-re','-i', '-','-preset', 'ultrafast','-vcodec', 'copy', '-acodec', 'copy','-threads','1', '-f', 'flv',destination + "/" + stream_id), stdin=_youtube_process.stdout)
    return

if len(sys.argv) < 4:
    show_help()
else:
    streamer()

This Python script does the following:

  1. Creates a subprocess named _youtube_process that reads the video using the youtube-dl library.
  2. Creates a second subprocess named _ffmpeg_process that receives data from the first one via pipe. This process creates an RTMP stream and sends it to the server at the specified address.

 

Testing the script

To run the script we need to install python. Download Python here.
We used version 2.6.6 for testing. Most likely any version will do, because the script is simple enough with just one goal – to send data from one process to another.

Running the script:

python streamer.py https://www.youtube.com/watch?v=9cQT4urTlXM stream1 rtmp://192.168.88.59:1935/live

As you see, three arguments are passed:

  1. The youtube address of a video: https://www.youtube.com/watch?v=9cQT4urTlXM
  2. The name of the stream the RTMP broadcast should go with: stream1
  3. The address of the RTMP-server: rtmp://192.168.88.59:1935/live

 

For testing, we use Web Call Server. It can receive RTMP streams and broadcast them via WebRTC. Here you can download and install WCS5 on your own VPS or local testing Linux server.

The diagram of tests using Web Call Server looks as follows:

The scheme of testing from Web Call Server

Below we use one of demo servers for the test:

rtmp://wcs5-eu.flashphoner.com:1935/live

This is the RTMP address we need to pass to the streamer.py script to quickly test broadcasting using the demo server.

So, launching looks like this:

python streamer.py https://www.youtube.com/watch?v=9cQT4urTlXM stream1 rtmp://wcs5-eu.flashphoner.com:1935/live

In stdout we can see the following output:

# python streamer.py https://www.youtube.com/watch?v=9cQT4urTlXM stream1 rtmp://wcs5-eu.flashphoner.com:1935/live
ffmpeg version 3.2.3 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 4.4.7 (GCC) 20120313 (Red Hat 4.4.7-11)
  configuration: --enable-shared --disable-logging --enable-gpl --enable-pthreads --enable-libx264 --enable-librtmp --disable-yasm
  libavutil      55. 34.101 / 55. 34.101
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.101 / 57. 56.101
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
]# [youtube] 9cQT4urTlXM: Downloading webpage
[youtube] 9cQT4urTlXM: Downloading video info webpage
[youtube] 9cQT4urTlXM: Extracting video information
[youtube] 9cQT4urTlXM: Downloading MPD manifest
[download] Destination: -
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'pipe:':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2016-08-23T12:21:06.000000Z
  Duration: 00:29:59.99, start: 0.000000, bitrate: N/A
    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 288 kb/s, 30 fps, 30 tbr, 90k tbn, 60 tbc (default)
    Metadata:
      creation_time   : 2016-08-23T12:21:06.000000Z
      handler_name    : ISO Media file produced by Google Inc.
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
    Metadata:
      creation_time   : 2016-08-23T12:21:06.000000Z
      handler_name    : ISO Media file produced by Google Inc.
Output #0, flv, to 'rtmp://192.168.88.59:1935/live/stream1':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    encoder         : Lavf57.56.101
    Stream #0:0(und): Video: h264 (Main) ([7][0][0][0] / 0x0007), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 288 kb/s, 30 fps, 30 tbr, 1k tbn, 90k tbc (default)
    Metadata:
      creation_time   : 2016-08-23T12:21:06.000000Z
      handler_name    : ISO Media file produced by Google Inc.
    Stream #0:1(und): Audio: aac (LC) ([10][0][0][0] / 0x000A), 44100 Hz, stereo, 125 kb/s (default)
    Metadata:
      creation_time   : 2016-08-23T12:21:06.000000Z
      handler_name    : ISO Media file produced by Google Inc.
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
frame=  383 fps= 30 q=-1.0 size=     654kB time=00:00:12.70 bitrate= 421.8kbits/s speed=   1x

From brief studying of the log we can see what happens:

  1. A page with the video is opened.
  2. Video format data is extracted.
  3. The mp4 video is downloaded, 1280x720, H.264+AAC
  4. ffmpeg runs, grabs the downloaded data and starts RTMP streaming at 421 kbps. Such low bitrate is explained by the chosen video – a simple timer. A more typical video would produce much higher bitrate.

 

After streaming starts, we try to play the stream in WebRTC-player. The name of the stream is specified in the Stream field, and the address of the server is in the Server field. Connection to the server is established via Websocket (wss), and the stream is received by the player as WebRTC (UDP).

Playing the stream in WebRTC player

We used this specific video from YouTube intentionally, to demonstrate the real-time nature of the stream. Indeed, our goal was to deliver a YouTube video to all spectators simultaneously with minimum latency and time spread. This millisecond timer video supremely well fits for this test.

The test itself is simple. We open two tabs in a browser (a simulation of two spectators), and play this timer stream as described. Then we take several screenshots to capture the time difference in video delivery. Finally, we compare milliseconds and see who’s got the video earlier, and who’s – later, and the actual difference

The results are:

Test 1

Testing the WebRTC stream in the player

Test 2

Testing the WebRTC stream in the player

Test 3

Testing the WebRTC stream in the player

As you can see, each spectator watches the same video with time divergence of no more than 130 milliseconds.

Hence, the goal of real-time broadcasting of a YouTube video as WebRTC is solved successfully. Spectators receive the video almost simultaneously. The manager didn’t have to go for coffee, the programmer had no time to read reddit, and the marketer managed to display the pussycat video to everyone.

Good streaming to you!

 

References

youtube-dl - the library to download video from YouTube
ffmpeg - RTMP encoder
Web Call Server – the server that shares an RTMP stream via WebRTC
streamer.py – the script to integrate youtube-dl and ffmpeg followed by RTMP stream sending.

 

Related articles

Stream RTMP video stream from the Live Encoder on WebRTC

 

Related features

Broadcasting of a streaming video from a professional video capturing device (Live Encoder) via the RTMP protocol