Tizen WASM Player Features

This section provides an overview of WASM Player, a Tizen TV WebAssembly extension allowing for low-level elementary media stream playback. Essential information & guidelines on WASM Player APIs for WebAssembly are included.

Related Info

Samples

Overview

Tizen WASM Player is a Samsung Web extension API which grants a WebAssembly application low level access to Tizen platform's media player. The WASM Player operates on Elementary Media Stream Packet level, accepting encoded packets (audio/video frames) that are decoded and rendered by the multimedia pipeline. Depending on operation mode, WASM Player is fit to be used as a video on-demand player (normal latency mode, suitable for adaptive streaming scenarios), as a streaming player (low latency mode) or as a game streaming player (ultra low latency mode).

Audio/Video Frame Based Playback

Tizen WASM Player is a low level API that accepts Encoded Audio/Video Frames (or Elementary Media Packets) to render media.

When used in this documentation, Audio/Video Frames are defined as raw, encoded audio or video data that can be rendered after decoding in the WASM Player. The complementary term (often used interchangeably) is Elementary Media Packet, which is defined as an Audio/Video Frame with added metadata.

Tizen WASM Player is labelled as a low level media API, because it works at the Elementary Media Packet level: Tizen WASM Player is responsible for decoding and rendering media content, while acquiring media content and splitting it into Audio/Video Frames (Elementary Media Packets) is entirely dependent on the application itself.

Rendering of Audio/Video Frames is configurable in Tizen WASM Player:

The application can select the latency mode to be used by the player (normal latency or one of the low latency modes).
Output can be either displayed by the WASM Player itself on an associated Media Element or rendered to an OpenGL video texture for the application to display.

The configurable rendering allows for great flexibility: the application has full control over downloading and demuxing of data, and implementation of either low latency or adaptive streaming protocol. When the WASM Player is used, a major part of the multimedia pipeline can be implemented as a WebAssembly module and is therefore platform-independent, allowing for a wide variety of multimedia applications.

This document describes both latency modes in detail and covers all API components required to use the WASM Player in a WebAssembly application.

Typical Usage Scenarios

Flexible configuration options allow for Tizen WASM Player usage in a wide range of applications. However, 2 typical usage scenarios can be highlighted:

Applications that use media container-based multimedia playback (such as DASH, HLS).
Applications that use streaming protocol-based multimedia playback (such as RTSP).

These typical scenarios are described in detail below.

NoteThe WASM Player's latency mode is orthogonal to the scenarios listed above. The latency mode is decoupled from an application's media data processing method (container- or streaming-based playback) and display mode (the WASM Player renders content, the application renders content using OpenGL). While the low latency modes are usually used with streaming protocols, they can be also used with container-based applications (and vice versa).

Usage Scenario: Media Container-based Multimedia Playback Application

In this usage scenario:

The application manages the downloading of media containers (such as DASH and HLS).
The application incorporates a demuxer implementation which splits containers into Audio/Video Frames (such as using FFmpeg).
Audio/Video Frames are sent to the WASM Player to be rendered.
The UI is typically implemented in HTML5. However, it is also possible to implement the UI (or parts of it) using OpenGL directly in the WebAssembly module.

Figure 1. High-Level Architecture of a Media Container-Based Playback Application

Usage Scenario: Streaming Protocol-based Multimedia Playback Application

In this usage scenario:

The application incorporates a streaming protocol implementation that controls and downloads media stream (such as RTSP).
Audio/Video Frames are sent to the WASM Player to be rendered.
The UI is typically implemented in HTML5. However, it is also possible to implement the UI (or parts of it) using OpenGL directly in the WebAssembly module.

Figure 2. High-Level Architecture of a Streaming Protocol-Based Multimedia Playback Application

Tizen WASM Player Comparison to MSE

Tizen WASM Player is a more low-level alternative to Media Stream Extensions (MSE):

MSE operates on media containers, whereas the WASM Player operates directly on encoded Audio/Video Frames (Elementary Media Packets).
Tizen WASM Player offers a wide range of configuration options that allows tailoring the media pipeline to specific needs of any given application (such as latency modes and an option to render video to a video texture).
When Tizen WASM Player is used, it's the application's responsibility to split the media into Elementary Media Packets. This approach offers greater flexibility compared to MSE, at the cost of increased application-side code complexity.

Refer to the following table for more key differences between Tizen WASM Player and MSE.

Feature	Tizen WASM Player	MSE
Accepted media	Encoded Audio/Video Frames (Elementary Media Packets)	Media Containers
Rendering targets	HTMLMediaElement, OpenGL texture	HTMLMediaElement
Latency modes	Normal latency, low latency, ultra low latency (Game mode)	Normal latency
Demuxer included in	Application	Platform

Table 1. Tizen WASM Player vs. MSE

Latency Modes

Tizen WASM player has three distinct latency modes: normal latency, low latency, and ultra low latency.

Normal Latency
In normal latency mode, the WASM Player can be used to playback typical on-demand media. In this mode the platform maintains both the multimedia pipeline clock and the synchronization of all running streams (audio and video), whereas the WebAssembly application is required to fill the WASM Player packet buffer with ES packets to be decoded and rendered by the platform. The normal latency mode supports all playback operations defined by WASM Player components, as well as DRM-protected content playback (see the Latency mode comparison table).

The normal latency mode is the best choice when playback is not optimized for latency.

Pipeline clock and stream sync: When the normal latency mode is used, the platform runs the multimedia pipeline according to a steady, monotonic internal clock. Rendering of packets sent to the WASM Player is synchronized with the clock according to their timestamps. Pipeline time advances only when the player's internal buffer allows continuous playback of all streams; if buffer underflow is detected, the pipeline pauses.
Low Latency
In low latency mode, the WASM Player can be used to playback streams which require low latency, such as media originating from online streams (such as a football match). In this mode, both the pipeline clock and the synchronization of the multimedia pipeline are controlled entirely by the WebAssembly application. The platform renders ES packets immediately after they are passed to the WASM Player. The low latency mode allows playback with low latency, however access to some operations and DRM is limited (see the Latency mode comparison table). Due to the nature of low latency playback, B-frames cannot be used.

The low latency mode is the best choice for playing media from online streams that can benefit from the picture enhancement functionality of device, with only little overhead on latency.

Pipeline clock and stream sync: When the low latency mode is used, the platform advances the multimedia pipeline running time according to the timestamps of packets passed to EMSS (ElementaryMediaStreamSource): whenever a packet is sent, the pipeline clock is updated immediately based on its timestamp.

Since the platform renders incoming Elementary Media Packets as they are sent and updates of the pipeline clock are directly dependent on the application's actions, the application is also responsible for stream synchronization.
Ultra Low Latency
The WASM Player set to the ultra low latency mode can be used to playback streams which require minimal latency, such as game streaming applications. In comparison to the low latency mode, picture enhancements are disabled, providing further latency reduction.

The ultra low latency mode is the best choice for playing media from live sources that require the lowest possible latency

Pipeline clock and stream sync: The pipeline clock and stream synchronization of the WASM Player set to the ultra low latency mode operate the same as in the low latency mode.

Latency Mode Comparison

The following table shows how the 3 latency modes differ from each other with their handling of specific features.

Feature		Normal Latency	Low Latency	Ultra Low Latency
Pipeline clock and stream sync controlling		By platform	By application	By application
Runtime configuration change	Resolution	Yes	Yes	Yes
Runtime configuration change	Frame rate	Yes	N/A	N/A
Seek		Yes	N/A	N/A
DRM		Yes	No	No
Example usage		VOD solutions (DASH and HLS)	Streaming solutions and RTSP	Game streaming

Table 2. Latency Mode Comparison

Note

Frame rate does not apply in low latency mode, as the rendering rate is dependent entirely on packet sending frequency (therefore, frame rate is freely and entirely controlled by the application).
The seek operation does not apply in low latency modes, as it implies working with a live, non-seekable source.
DRM support is available only in conjunction with the Media Element rendering mode.

Rendering Modes

Tizen WASM Player has 2 rendering modes: Media Element and Video texture.

Media Element
In the Media Element mode, the WASM Player presents decoded video frames directly on an attached HTMLMediaElement.
This is the default rendering mode, and provides support for both normal and encrypted playback.
Video Texture
In the Video Texture mode, the WASM Player fills a provided texture with pictures decoded from the video stream. This WebGL texture can then be rendered on the screen on a canvas element.
Encrypted playback is not supported in this mode.

Rendering Mode Comparison

Feature	Media Element	Video Texture
DRM Support	Yes	No
Example usage	VOD solutions	Video thumbnails

Table 3. Rendering Mode Comparison

NoteDRM support is available only in conjunction with the normal latency mode.

API Guide

The following sections contain a detailed look into the WASM Player architecture and its components.

Architecture Overview

The WASM Player is meant to be used in Tizen Web widget applications that employ WebAssembly modules. Tizen Web widget applications work in a HTML5 environment: the WASM Player provides an ElementaryMediaStreamSource component that is designed to be used as a source object of HTMLMediaElement.

Figure 3. WASM Player Architecture Overview

In order to use Media Player with the WASM Player, the application must use the provided WASM Player classes (which act as a source of data for playback) and the HTMLMediaElement class (to control Media Player). The following table defines the class responsibilities and functions:

Class	Description
`ElementaryMediaStreamSource`	This is the main class of Tizen WASM Player. It acts as a source object for `HTMLMediaElement`.`ElementaryMediaStreamSource` manages a set of `ElementaryMediaTrack` objects. Together, their purpose is essentially feeding the platform with media data required to play content; EMSS classes are acting as a source of media data.
`ElementaryMediaTrack`	Each instance of this class represents a single elementary media track (either audio or video). The track object allows sending Elementary Stream Packet data to the platform for playback.
`HTMLMediaElement`	`HTMLMediaElement` is used alongside WASM Player classes as a playback control element, providing operations such as play, pause, and seek. That is, Media Element controls Media Player.`HTMLMediaElement` is associated with either the `<audio>` or `<video>` tags in HTML, allowing for the positioning of the playback area.The relevant part of the HTMLMediaElement API is wrapped by a C++ API available alongside the WebAssembly EMSS C++ API.`HTMLMediaElement` is not covered in this document in detail. For more information, see HTML Media Element Specification.

Table 4. WASM Player and Media Player classes

Relationship Between ElementaryMediaStreamSource and HTMLMediaElement

ElementaryMediaStreamSource and HTMLMediaElement serve different purposes in Media Player. As shown above, Elementary Media Stream Source objects act as the means to deliver media data to HTMLMediaElement for playback. Controlling playback (such as issuing Play, Pause, and Seeking commands) on Media Player is done through the HTMLMediaElement API. Elementary Media Stream Source allows you to deliver media data in form of Elementary Media Packets.
When operating Media Player, the application must rely on HTMLMediaElement events and methods. For example, ElementaryMediaStreamSource signals when Media Player requires media data. However, the moment Media Player starts reading media data is not always the exact moment that playback can be started. Readiness for playback is signalized by HTMLMediaElement through the HTMLMediaElementListener::OnCanPlay() event.

ElementaryMediaStreamSource States

The following figure and table describe the ElementaryMediaStreamSource states and the transitions between them.

Figure 4. ElementaryMediaStreamSource State Diagram

Ready State	Description
`kDetached`	`ElementaryMediaStreamSource` is not assigned to `HTMLMediaElement`.This is the initial state of an `ElementaryMediaStreamSource` object. It also enters this state after disconnecting from `HTMLMediaElement`.
`kClosed`	Tracks are not configured and the player is not initialized. Media cannot be played in this state.`ElementaryMediaTrack` objects can be added to and removed from `ElementaryMediaStreamSource`. Track layout can be changed only in this state.This state is entered after `ElementaryMediaStreamSource` is attached to `HTMLMediaElement`, when unrecoverable playback error occurs, or on App request.
`kOpen`	Media Player is fully initialized and Elementary Media Stream Source is ready to accept Elementary Packet Data from the application through `ElementaryMediaTrack` objects.When the application finishes configuring Tracks, it can request entering this state. The `kOpen` state is entered when possible. Some operations (like Seek) trigger a temporary transition to `kOpenPending` state instead.
`kOpenPending`	Opening `ElementaryMediaStreamSource` was requested by the application, but the current state of the multimedia pipeline prevents immediately entering the `kOpen` state. The `kOpen` state will be entered as soon as possible.This is a temporary state, entered on several occasions: When the application requests opening `ElementaryMediaStreamSource`, the `kOpenPending` state is set for the duration of multimedia pipeline initialization, During playback, when the application must cease sending packets temporarily (such as during Seek).
`kEnded`	Stream has ended but multimedia pipeline remains initialized.This state is entered when the application marks all active `ElementaryMediaTrack` objects as ended. The `kEnded` state reverts to the `kOpen` state when the multimedia pipeline resumes playback (such as due to Seek).

Table 5. ElementaryMediaStreamSource States

ImportantElementaryMediaStreamSource's and HTMLMediaElement's states must not be confused. ElementaryMediaStreamSource represents the state of the source of data, while HTMLMediaElement represents the state of the multimedia player. As such, ElementaryMediaStreamSource signals the application whether the multimedia pipeline needs data and is able to accept Elementary Media Packets. This is not in sync with the multimedia player's state.
For example, the kEnded state can be associated with a still-playing Media Element. This occurs in normal latency mode when the application signals the end of all tracks, but the multimedia pipeline has buffered Elementary Media Packets remaining to play.

The kOpen State and Latency Modes

Depending on the latency mode, the ElementaryMediaStreamSource's preconditions for entering the kOpen state are slightly different.

In normal latency mode, requesting ElementaryMediaStreamSource to open initializes Media Player and immediately triggers its data buffering mechanism. Therefore, Elementary Media Stream Source enters the kOpen state shortly after open is requested (briefly entering the kOpenPending state during initialization). As soon as the kOpen state is entered, the application is expected to send media data to EMSS.
In low latency or ultra low latency mode, requesting ElementaryMediaStreamSource to open puts it in the kOpenPending state and initializes Media Player. The kOpen state (where the application is expected to send media data to EMSS) is entered when Play is issued on HTMLMediaElement. The source remains open as long as the Media Element is in the playing state. Media Player renders any Elementary Media Packets immediately after they are sent, with no intermediate buffering.

NoteThere are numerous reasons for EMSS to switch between states. The WASM Player Usage Guide provides detailed information on when such a change happens.

ElementaryMediaTrack States

The state of a ElementaryMediaTrack object reflects the state of an associated ElementaryMediaStreamSource, however in a simplified manner. A track can be either open or closed, represented by a single boolean attribute.

IsOpen	Description
true	The track can accept Elementary Media Packets.The track opens just before the source enters the `kOpen` state.
false	The track cannot accept Elementary Media Packets.The track closes just after the source leaves the `kOpen` state.

Table 6. ElementaryMediaTrack States

NoteWhen a track closes, an event is generated carrying a reason why the closure occurred.

HTMLMediaElement States

The C++ HTMLMediaElement class supplied with Elementary Media Stream Source is a wrapper for a HTML5's HTMLMediaElement. For information on HTMLMediaElement states, see HTML Media Element Specification.

WASM Player and Application Multitasking

Applications that target devices work in a multitasking environment and must respond to multitasking appropriately. For a multimedia application, this often means having to suspend media playback when the application becomes invisible (such as goes into background when the user launches another application) and resuming playback when the application becomes visible (such as goes into foreground when the user resumes the application).

The WASM Player tracks an application's multitasking state and generates appropriate events for the application to react to. Specifics of the event sequence depends on a selected EMSS mode of operation.