Developing WebRTC videochat between iOS, Android and browser

Let’s implement a three-way video chat on the following platforms: desktop Google Chrome, an Android application on a tablet, and an iOS application for Apple iPhone.

To recap, the two main principles of developing a video chat are:

1. Each connected user can send (publish) his or her video stream to the server.
2. Users know their names of each other’s video streams and can play them.

Therefore, in a video chat with three participants, each participant has to play two video streams.

The exact communication sequence consists of nine actions shown below. At first, participants of the video chat publish their video streams, then they play each other’s streams.

Publishing video streams

1. Alice: session.createStream({name:”stream_alice”}).publish();
2. Boris: session.createStream({name:”stream_boris”}).publish();
3. Anna: session.createStream({name:”stream_anna”}).publish();

Playing video streams

4. Alice: session.createStream({name:”stream_anna”}).play();
5. Alice: session.createStream({name:”stream_boris”}).play();
6. Boris: session.createStream({name:”stream_alice”}).play();
7. Boris session.createStream({name:”stream_anna”}).play(); 8. Anna: session.createStream({name:”stream_alice”}).play();
9. Anna: session.createStream({name:”stream_boris”}).play();

What the developer needs here is to organize transferring of names and statuses of video streams between subscribers of the video chat. You can use any suitable technology for this: PHP, websockets, Node.js and so on, since sending the name of the video stream is absolutely identical to sending any other text message from one user to another.

A stream can have three main statuses: PUBLISHING, PLAYING, STOPPED For successful playback, the requested stream must be in the active state PUBLISHING.

This diagram illustrates how you can exchange names and statuses of streams in a simplified scenario where Alice shows her video stream to Boris and Anna. Such procedure takes 8 steps and can be called signalling, because the result of this procedure is the below negotiation:

1. Alice sends a video stream to the WCS server.
2. Alice receives the confirmation status PUBLISHING from the server
3. Alice sends a message to Boris that her stream is ready for playback.
4. Alice sends a message to Anna that her stream is ready for playback.
5. Anna plays the video stream from the WCS server.
6. Anna receives the confirmation status PLAYING from the server.
7. Boris plays the video stream from the WCS server.
8. Boris receives the confirmation status PLAYING from the server.

As a result, an arbitrary interaction scenario can be implemented: with two or more users connected to the chat, with multiple viewers connected to the chat and so on.

Rooms

Technically, it is not that hard to implement exchange of names of video streams. But this still requires some effort and coding.
On the other hand, such a task begs some versatile solution that could help users to quickly synchronize streams and start the chat. Such solution is called Rooms or Room API.

Indeed, if two or more subscribers interact in the same context, this looks similar to a physical room. Inside such a room, users can see each other’s video streams, know who else is in the room and can exchange messages, public or private alike.

Therefore, we’ve got four objects that completely cover the functionality of a room:

• Room
• Stream
• Participant
• Message

Room API allows for cross-platform usage of the above abstractions: Room, Stream, Participant и Message to implement the following functions:

Connection

• The user can connect to the room (join).
• The user specifies a room name upon connection.
• If the room with this name already exists, the user joins this room.
• If there’s no room with such a name, a new room is created and the user joins the newly created room.
• The user receives notifications about connected / disconnected users.
• The user can receive a list of participants.

Streaming

• The user can publish a video stream in the room.
• The user receives updates about statuses of video streams of other participants.
• The user can play a video stream in the room.

Messages

• The users can exchange messages in the room.
• The users can exchange images or other content as long as it is packed into the text format.
• A message can be sent to one or more participants.

Room API

For the Web platform rooms are implemented as a JavaScript module with the following functionality:

1. Connect to the server.

var connection = Flashphoner.roomApi.connect({urlServer: "wss://host:8443", username: "Alice"});

2. Enter the room.

connection.join({name: "room1"});

3. Get the list of room participants.

var participants = room.getParticipants();

4. Publish our video stream to the room.

room.publish({
	display: document.getElementById("localDisplay"),
        constraints: constraints,
        record: false,
        receiveVideo: false,
        receiveAudio: false
});

5. Play the stream of a participant.

participant.getStreams()[0].play(document.getElementById(pDisplay))

6. Monitor other participants of the room:

connection.on(ROOM_EVENT.JOINED, function(participant){...});
connection.on(ROOM_EVENT.LEFT, function(participant){...});
connection.on(ROOM_EVENT.PUBLISHED, function(participant){...});

JOINED – a new participant has joined the room
LEFT – a participant has left the room
PUBLISHED – a participant has published his video stream

7. Receive messages from other participants.

connection.on(ROOM_EVENT.MESSAGE, function(participant){...});

8. Send a private message to the specified participant.

participants[i].sendMessage(message);

9. Send a message to all participants at once.

var participants = room.getParticipants();
        for (var i = 0; i < participants.length; i++) {
            participants[i].sendMessage(message);
        }
}

Therefore, this room implementation allows for simple exchange of messages and statuses between all connected participants of the room.

Room API limitation

All room operation logic is managed by the client. The server control only the basic functionality of the room, specifically:

• room participant connect/disconnect notification
• notifications of creating/closing of video stream by participants of the room
• routing of messages to the given participants

Consequently, authentication, permissions, user roles (moderator, spectator, anchorman) and all other logic is implemented on the client side and / or on the third-party back-end. Rooms simply simplify quick exchange of information on video streams and their statuses.

Rooms for Web, Android, iOS

Each of these server SDK (Web, Android, iOS) offers the room API.

Examples of joining a room:

Web SDK

connection.join({name: "room1"});

Android SDK

Room room = roomManager.join(roomOptions);

iOS SDK

room = [roomManager join:options];

Therefore, three participants from three different platforms can join the same room:

• Web browser
• Mobile application for Android
• Mobile application for iOS

Test applications to work with rooms

Below are three test applications to work with rooms. Each of them is open source and can be build from the sources. Each of the below applications allows exchanging video streams and messages between three subscribers:

• Conference for Web
• Conference for Android
• Conference for iOS

Conference for Web

The source code of this applications is available for download here. There are two div elements in the HTML code:

These elements display video of participants. The div element localDisplay displays video captured from the camera.

Using the Join / Leave buttons a participant can join or leave the room respectively. Stop / Start publish a video in the room and stop publishing respectively. The Login field must be unique because it identifies a participant.

The next element of the interface is a text chat. This chat displays messages received or sent by other users and shows information about events in the room.

The last element is the link to invitation. If a user joined first, a new room is created. In this case roomName=room-fb41b7. If a second user specifies the same roomName, he joins the same room. In the Conference for Web application invitations are implemented as join links with an additional parameter that contains the given roomName. In Android and iOS versions of the room, the name of the room is specified directly in the interface.

Therefore, the test app Conference for Web implements all functions described above, and allows several users to join the same room and exchange video streams and text messages. The below screenshot shows how a room with three participants functions. We opened three tabs in the Google Chrome browser and initiated connection to the room on each tab.

Conference for Android

The source code of this applications is available for download here. In the app, you can enter the name of the room directly in the UI. Apart from that, the application is very similar to Conference for Web described above. The interface elements are the same:

• Leave / Join buttons to enter and exit the room.
• Two video windows to play video of participants.
• One video window to display the video captured from the camera.
• A text chat with system information and messages of participants.

Conference for iOS

The source code of this application is available for download here. The interface of the iOS application is similar to that of the Android app down to buttons and controls. Summarizing the above, we prepared a test and gathered all three platforms in one room with id 3119d6.

Google Chrome is where the hare who comes out of the hole:

Android 5.1.1 on the Asus tablet displays a flower pot.

iOS 10.1.1 on iPhone 6 shows a ficus on the window.

Below is a full length screenshot from the iOS device:

So, we finished testing and reviewing all three applications build on the Room API, and now we can discuss the source code and building.

Building the Conference for iOS application from the source codes

We are going to explain how to build the Conference for iOS example, and will comment the most important parts of the source code. First of all, we need some Mac hardware to install the latest Xcode. The build requires installing Cocoapods, downloading source codes of the examples and the SDK for iOS. We will build the project in the shell, then we will open the project in Xcode.

1. Install Cocoapods.

sudo gem install cocoapods

2. Download all examples from github.

git clone https://github.com/flashphoner/wcs-ios-sdk-samples.git

3. Download iOS SDK.

wget https://flashphoner.com/downloads/builds/flashphoner_client/wcs-ios-sdk/WCS-iOS-SDK-2.3.0.tar.gz

4. Unpack the archive. tar -xvzf WCS-iOS-SDK-2.3.0.tar.gz

tar -xvzf WCS-iOS-SDK-2.3.0.tar.gz

5. Copy the FPWCSApi2.framework folder to examples.

cp -R FPWCSApi2.framework wcs-ios-sdk-samples

6. Build.

./build_example.sh

If the build is successful, the terminal displays **ARCHIVE SUCCEDED**

After the examples are built from the console using Cocoapods, all dependencies are linked and configured, all further builds of examples can be made directly in Xcode.

7. Open WCSExample.xcworkspace in Xcode

8. Select Generic iOS Device in the build target. Run building of the Conference example from the Product / Build menu and wait for completion.

9. Connect the iPhone via USB and run the built application, Conference. The console displays debug information.

10. The screen of the iPhone displays the Conference application.

So, we successfully built the Conference mobile application for iOS using iOS SDK version 2.3.0 + Cocoapods for this build. As a result, we managed to run this application on iPhone 6 connected via USB to the Mac we built the app on.

Source code for Web, Android, iOS video chats

Above, are examples of applications for three platforms that use the Room API and exchange video streams within the created room. Below are a brief overview of the main parts of code that are responsible for video chat operation on each of these platforms:

Overall, we’ve got 10 main fragments for each platform. You can find each of the above constructions in the corresponding source file. All three applications – Web, iOS and Android – were tested with the latest build of Web Call Server 5 – a server for video chats and low latency broadcasts with the support for WebRTC technology.

References

Web Call Server – a server for WebRTC video chats

EC2 launch – start a pre-configured image on Amazon EC2

Install – install the server to VPS or Dedicated host

Cocoapods – build dependency manager