Today, the words “virtual reality” will not surprise anyone. Futuristic equipment, which we could only see in science fiction movies a couple of dozen years ago, is now available to regular users. Even those who cannot afford proper equipment can build a VR headset of available materials following a variety of DIY-tips with YouTube.

In addition to purely entertainment areas, such as games and movies, VR can be useful in many other areas of our lives. For example, one day, having received an inheritance from a prematurely deceased aunt, you decide to purchase a villa on the Cote d’Azur. But the trouble is, such a busy person like you has absolutely no time to visit each object offered by a real estate agent. This is where VR may come in handy. The real estate agent installs a special camera in the house, and you can view it without getting up from the couch leaving your office.

After viewing the property, you can ask what your children are doing after school and if your favorite cat is taking a nap on the windowsill. And virtual reality will come to the rescue once again – with the help of a pre-installed camera, you can inspect all the corners of your sweet home with the effect of “being there.”

Let’s figure out how to make everything work, so that the children do not run away and the cat does not eat your pot flowers.

Key difference between VR image

In fact, the VR media stream is no different from the usual one: a video track plus an audio track. But there is a feature.

A VR stream assumes a stereo image. For this, a special camera with two lenses is used, each of which has a viewing angle of 180 degrees

 

VR-streaming-issues-robot-head_4K-WebRTC-WCS

Images from each of the lenses are collected in one frame,

 

vr-stream-play-flat-pic-only_WebRTC-HLS-4K-TCP

therefore, the player on the viewer’s device must make two images one. As you can see, the aspect ratio of the frame is 2:1, which is a FullHD image for each lens.

The nuances of VR broadcast

So, we are talking about broadcasting 4K streams with a high bit rate. What do we have? We have RTMP and WebRTC.

RTMP is cheap, reliable and practical. It uses TCP, which is useful in case of not-so-good channels. There are various software solutions for the publishing client, both paid and free. However, despite all its merits, RTMP has high latency. In one of the previous articles, we noted a delay of 2-3 seconds in a 720p stream. In some cases, delays are acceptable, but VR will lag behind reality, and the children will surely scatter.

WebRTC is really cool. With good channels, delays are measured in milliseconds, and reality will remain reality. But there are nuances.

First, by default WebRTC runs on UDP, which will lead to losses at the slightest deterioration in channel quality. For a 4K stream, any little thing on the channel is already a deterioration. This can be dealt with by switching to TCP transport, but in this case…

Second, WebRTC is broadcast from the browser, and we both know which browser is currently the most popular on the planet (spoiler – it is not IE6). In this popular browser, the maximum publication bitrate at the engine level is 2500 kbps. As Google thinks, this is enough for FullHD but not enough for 4K. If the bitrate is not accelerated to 5 – 10 Mbps, viewers will see moving watercolor spots instead of virtual reality. Fortunately, there are special settings for overclocking the bitrate that work in pretty much all browsers on the Chromium engine. Then, overcoming this obstacle, we will run into…

Third, we will run into channel bandwidth. To publish and play a VR picture with the bitrate mentioned above, the client will need channels of 20+ Mbit, or better 50, and even better 100+ Mbit, both for download and upload. In this case, these should not be the parameters that the provider talks about in advertising booklets, but the actual bandwidth from the client to the server. By the way, read how to measure it without iperf and command line. Correspondingly, on the server side, it is desirable to have 10 Gb if a large number of streams is envisioned.

Fourth is what follows from the requirements for channels. Transcoding a 4K stream will take up too much processor resources on the server. Therefore, you must either choose a king-size server, which is expensive, or use encoding on video cards, which is also very expensive, or avoid transcoding. That is, never change the resolution, frame rate, and bit rate of the streams during broadcasting.

Let’s try to do this

Let’s take:

 

Mount the camera on a table or tripod

vr-stream-camera_WebRTC-4K-VR

Connect it to the PC via USB 3.0 with 1A power, as for external drives

vr-stream-camera-connect-usb3_WebRTC-4K-VR

Note that the camera is very sensitive to equipment, and if the laptop has 2 USB 3.0 ports, it may well refuse to work with one of them. Make sure not to use a cheap Chinese connection cable; all pins provided for by the standard should be in place. Even with a perfect connection, the camera can produce glitch, like an old TV in a thunderstorm. The only positive thing is that no drivers are required for Windows 10; it’s an honest Plug-n-Play type connection.

We will set the necessary bitrate limits on the server side

webrtc_cc_min_bitrate=5000000
webrtc_cc_max_bitrate=10000000

In Chrome, open the page of an example, allowing to set the necessary parameters to capture the stream from the camera, and set

  • resolution of 3072×1536
  • FPS 24 (the camera supports only this FPS when broadcasting from the browser)
  • TCP transport

 

and configure the SDP settings so that the browser does not cut the bitrate on the client side

x-google-max-bitrate=10000;x-google-min-bitrate=5000

or the server side (in this case, the settings will be valid for all publishing clients)

webrtc_sdp_min_bitrate_bps=5000000
webrtc_sdp_max_bitrate_bps=10000000

vr-stream-parameters_WebRTC-4K-Chrome-bitrate-TCP

Let’s publish the stream from the camera

vr-stream-publish_WebRTC-4K-Chrome-bitrate-TCP

Now let’s try to play the stream with a regular WebRTC player

vr-stream-play-flat_WebRTC-HLS-4K-TCP

All is fine… Almost. The virtuality is not really virtual. Now let’s play the same stream in a special player embedded on the page

vr-stream-play-vr-WebRTC-HLS-4K-TCP-Delight-Player-embedding

Now that’s better (and we have cats too). If you play the stream on a mobile device or in VR glasses, you can turn your head, and we will move the mouse in the browser on the PC. There is a kid hiding somewhere in this room. Let’s look to the left

vr-stream-play-vr-left_WebRTC-HLS-4K-TCP-Delight-Player-embedding

And now to the right

vr-stream-play-vr-right_WebRTC-HLS-4K-TCP-Delight-Player-embedding

And now up

vr-stream-play-vr-up_WebRTC-HLS-4K-TCP-Delight-Player-embedding

Where’s the kid? There he is, hiding behind the curtain. But the cats sees him!

We need more code!

There is little code on the client’s side.

Let’s publish the stream from the browser page

session.createStream({
     name: streamName,
     display: localVideo,
     cacheLocalResources: true,
     transport: "TCP",
     sdpHook: rewriteSdp, 
     constraints: {
         audio:true, 
         video: { 
             width: 3072,
             height: 1536,
             frameRate: 24
        }
     }
}).publish();

SDP browser override feature

function rewriteSdp(sdp) {
    var sdpStringFind = "a=fmtp:(.*) (.*)";
    var sdpStringReplace = "a=fmtp:$1 $2;x-google-max-bitrate=10000;x-google-min-bitrate=5000";
    var newSDP = sdp.sdpString.toString();
    newSDP = newSDP.replace(new RegExp(sdpStringFind,"g"), sdpStringReplace);
    return newSDP;
}

An example of a page with a VR player

<!DOCTYPE html>
<html>
    <head>
        <title>WebRTC Delight</title>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
           <script type="text/javascript" src="../../../../flashphoner.js"></script>
           <script type="text/javascript" src="../../dependencies/jquery/jquery-1.12.0.js"></script>
           <script type="text/javascript" src="../../dependencies/js/utils.js"></script>
           <script src="dl8-player.js" async></script>
        <meta name="dl8-custom-format" content='{"name": "STEREO_TERPON","base":"STEREO_MESH","params":{"uri": "03198702.json"}}'>
    </head>
    <body>       
        <div style="width: 50%;" id="display">
            <dl8-live-video id="remoteVideo" format="STEREO_TERPON">
                <source>
            </dl8-live-video>
        </div>
        <input class="form-control" type="text" id="playStream" placeholder="Stream Name">
        <button id="playBtn" type="button" class="btn btn-default" disabled>Play</button>
        <button id="stopBtn" type="button" class="btn btn-default" disabled>Stop</button>
        <script>
            Flashphoner.init({flashMediaProviderSwfLocation: '../../../../media-provider.swf'});
            var SESSION_STATUS = Flashphoner.constants.SESSION_STATUS;
            var STREAM_STATUS = Flashphoner.constants.STREAM_STATUS;
            var STREAM_STATUS_INFO = Flashphoner.constants.STREAM_STATUS_INFO;
            var playBtn = document.getElementById('playBtn');
            var display = document.getElementById('display');
            var dl8video = null;
            var url = setURL();
            document.addEventListener('x-dl8-evt-ready', function () {
                dl8video = document.getElementById('remoteVideo');
                $('#playBtn').prop('disabled', false).click(function() {
                    playStream();
                });
            });
            function playStream() {
            $('#playBtn').prop('disabled', true);
            $('#stopBtn').prop('disabled', false);
            var video = dl8video.contentElement;
            Flashphoner.createSession({urlServer: url}).on(SESSION_STATUS.ESTABLISHED, function (session) {
                var session = Flashphoner.getSessions()[0];
                session.createStream({
                    name: document.getElementById('playStream').value,
                    display: display,
                    remoteVideo: video,
                    transport: "TCP"
                }).on(STREAM_STATUS.PLAYING, function (stream) {
                    dl8video.start();
                    $('#stopBtn').prop('disabled', false).click(function() {
                         $('#playBtn').prop('disabled', false);
                         $('#stopBtn').prop('disabled', true);
                         stream.stop();
                         dl8video.exit();
                    });
                }).play();
            }) 
            }
        </script>
    </body>
</html>

Usually, when creating a stream for a player (query session.createStream()), we pass the div element, into which the video element will be mounted to play the stream via WebRTC. But the VR player uses its own video element, and we need to somehow forward it into the code of the API used. To do this, we directly pass the video element of the third-party player to the session.createStream() with the remoteVideo parameter

demo.flashphoner.com is used as the server side; examples of web applications are available on the links below.

Good luck in your VR endeavors! Happy streaming!

 

Links

An example for publishing a stream with the necessary parameters

An example of integrating a VR player into WebRTC streaming

 

Related Articles

Quality control of channels from client to server

 

Related features

WebRTC streaming from browser

 

Documentation

Using WCS with a VR player