WebRTC Back and Forth

In our blog, we have been recently looking at all kinds of stress tests. In one of the previous articles we tested one of the common server operation scenarios — “One stream, many viewers.” In this scenario, one WebRTC video stream is published on the server and a large number of viewers connect to view this stream. This is how, for example, the competition broadcast works: the broadcast is on, and the users are watching. But quite often you need to implement another approach — “Many streams, many viewers” — when many WebRTC streams are published on the server at the same time and each published stream has one viewer. This option can be used to organize a simple one-on-one video chat without using a mixer or MCU.

In this article, we will analyze the method in detail and conduct a stress test to simulate the connection of 200 users, each of whom simultaneously publishes and views WebRTC streams.

Testing Plan

Publish a stream of 1280×720, 30 fps, bitrate 2,500 kbps, on the testing WCS#1 server
Publish the source stream using the REST API /pull/push request in the required number of instances to the tested WCS#2 server
Using the second REST API /pull/push request, take each instance of a previously published stream from the tested WCS#2 server to the WCS#1 server
Play a random stream on the tested WCS#2 server and visually evaluate its quality. Estimate the server load using metrics received from the Prometheus monitoring system.

scheme-200-streams-720p-WCS-WebRTC-stream-WebSocket-publishing-testing-RESTApi

Preparing for Testing

For testing, we need

two WCS servers;
standard example “Media Devices” for publishing a stream with specified parameters;
test script;

We will assume that you already have your WCS servers installed and configured. If not, install them according to this instruction, make tweaks to boost performance and connect to monitoring.

We used two servers for testing:

WCS#1 (testing) with characteristics:

24 cores, 48 streams;
128 Gb RAM;
WCS 5.2.946;
OpenJDK version 14.0.1

and

WCS#2 (tested) with characteristics:

12 cores, 24 streams;
80 Gb RAM;
WCS 5.2.946;
OpenJDK version 14.0.1

In one of the previous tests, we have already found out empirically that the load on the testing server is quite large, therefore, for the tests to succeed, the testing server must be at least twice as powerful as the tested one, or several servers must act as the testing server.

Our server settings:

File flashphoner.properties (w/o default settings):

#webrtc ports range
media_port_from = 20001
media_port_to = 40000

wcs_activity_timer_timeout=86400000

global_bandwidth_check_enabled=true
zgc_log_parser_enable=true
zgc_log_time_format=yyyy-MM-dd'T'HH:mm:ss.SSSZ

Here we increased the range of ports and the time for passing the test, as well as specified the settings for collecting additional metrics for network congestion and pauses in ZGC.

File wcs-core.properties (w/o default settings):

-XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log

# ZGC
-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xms24g -Xmx24g -XX:+UseLargePages -XX:ZPath=/hugepages

-Xlog:gc*:/usr/local/FlashphonerWebCallServer/logs/gc-core-:time

Here we changed the log file naming settings, enabled the use of ZGC, adjusted the Heap size, and enabled the use of memory pages.

Full versions of the configuration files can be downloaded from the Useful Files at the bottom of the article. There you will also find a panel for Grafana and a script file to start testing.

Testing Script

#!/bin/bash

sourceStreamName=$1
sourceServer=$2
serverToTest=$3
testStreamsCount=$4

if [[ -z $sourceStreamName ]] || [[ -z $sourceServer ]] || [[ -z $serverToTest ]]; then
  echo "Usage: $(basename $0) stream_name source_server server_to_test test_streams_count"
  echo -e "   stream_name\t\tstream published on source server name"
  echo -e "   source_server\t\tsource server ip address"
  echo -e "   server_to_test\t\tserver to test ip address"
  echo -e "   test_streams_count\t\tstreams count to push (1 by default)"
  exit 1
fi

[[ -z $testStreamsCount ]] && testStreamsCount=1

# Check if source stream is published
result=`curl -s -i -o /dev/null -w '%{http_code}' -H "Content-Type: application/json" -X POST http://$sourceServer:8081/rest-api/stream/find --data "{\"name\":\"$sourceStreamName\",\"published\":true}" | grep 200`
if [[ $result != "200" ]]; then
  echo "Stream $sourceStreamName is not published on server $sourceServer!"
  exit 1
fi

# Push streams to the server to test
echo "Publishing $testStreamsCount copies of stream $sourceStreamName to the server $serverToTest"
for ((i = 1; i <= $testStreamsCount; i++)); do
  curl -s -H "Content-Type: application/json" -X POST http://$sourceServer:8081/rest-api/pull/push --data "{\"uri\":\"ws://$serverToTest:8080/websocket\",\"localStreamName\":\"$sourceStreamName\",\"remoteStreamName\":\"$sourceStreamName$i\"}"
done

# Wait for 5 seconds
echo "Waiting for 5 seconds..."
sleep 5

# Push streams back to source server
echo "Playing $testStreamsCount copies of stream $sourceStreamName from the server $serverToTest"
for ((i = 1; i <= $testStreamsCount; i++)); do
  curl -s -H "Content-Type: application/json" -X POST http://$serverToTest:8081/rest-api/pull/push --data "{\"uri\":\"ws://$sourceServer:8080/websocket\",\"localStreamName\":\"$sourceStreamName$i\",\"remoteStreamName\":\"$sourceStreamName$i\"}"
done

echo "To stop the test, just stop publishing stream $sourceStreamName"

where:

sourceStreamName — name of the source stream published on the testing WCS#1 server;

sourceServer — name or IP address of the testing WCS#1 server;

serverToTest — name or IP address of the tested WCS#2 server;

testStreamsCount — number of streams for the test.

Testing

On the testing WCS#1 server, using the example “Media Devices” publish a stream under the name “test” with parameters of 1280×720, 30 fps, bit rate 2,500 kbps.

media-devices-720p-WCS-WebRTC-stream-WebSocket-publishing-testing-RESTApi

start-publish-media-devices-720p-WCS-WebRTC-stream-WebSocket-publishing-testing-RESTApi

Then, in the testing server console, run the test script with the following parameters:

./pull_push_test.sh test 172.16.40.3 172.16.40.2 200

where
test — name of the stream published on WCS#1
172.16.40.3 — IP address of the testing WCS#1 server
172.16.40.2 — IP address of the tested WCS#2 server
200 — number of streams

The result for testing 200 streams is shown in the screenshot below:

The test was successful. No stream degradation was recorded and the control stream was played with acceptable quality – without friezes, artifacts, or a decrease in sound quality:

control-stream-720p-WCS-WebRTC-stream-WebSocket-publishing-testing-RESTApi

The test was completed successfully on our hardware. Please note that the test results on your hardware can be better or worse depending on the environment, geographical location of servers and users – both subscribers and streamers.

Therefore, we recommend that, before making the final decision on the choice of servers, you conduct all possible tests that may be relevant to your task, and make a decision based on their results.

Have a good streaming!