Overview
This section provides an overview of WASM Player, a Tizen TV WebAssembly extension allowing for low-level elementary media stream playback. Essential information & guidelines on WASM Player APIs for WebAssembly are included.
Related Info
- Tizen WASM Player API Documentation
- Tizen WASM Player Sample (Normal Latency)
- Moonlight port to Tizen WASM (Low Latency)
- Samsung SmartTV Media Specification
- Samsung SmartTV Widget Multitasking Guide
- WebAssembly for Tizen TV Download
- W3C HTML Media Element Specification
Samples
Overview
Tizen WASM Player is a Samsung TV extension API which grants a WebAssembly application low level access to a Platform's media player. WASM Player operates on Elementary Media Stream Packet level, accepting encoded packets (audio/video frames) that are decoded and rendered by the TV multimedia pipeline. Depending on operation mode, WASM Player is fit to be used as a video on-demand player (normal Latency Mode, suitable for adaptive streaming scenarios), as a streaming player (low Latency Mode) or as a game streaming player (ultra low Latency Mode).
Audio/Video Frame based playback
Tizen WASM Player is a low level API that accepts Encoded Audio/Video Frames (or Elementary Media Packets) to render media.
When used in this documentation Audio/Video Frames are defined as raw, encoded audio or video data that can be rendered after decoding in WASM Player. The complementary term, often used interchangeably, is Elementary Media Packet: defined as an Audio/Video Frame with added metadata.
Tizen WASM Player is labelled as a low level media API, because it works at Elementary Media Packet level: Tizen WASM Player responsibility is decoding and rendering media content. Acquiring media content and splitting it into Audio/Video Frames (Elementary Media Packets) is entirely dependent on App.
Rendering of Audio/Video Frames is configurable in Tizen WASM Player:
- App can select Latency Mode that will be used by the player (Normal Latency or one of Low Latency Modes).
- Output can be either displayed by WASM Player itself on an associated Media Element or rendered to an OpenGL video texture for App to display.
This allows for a great flexibility: the application has full control over downloading and demuxing data, implementation of either low latency or adaptive streaming protocol, etc. When WASM Player is used major part of multimedia pipeline can be implemented as a WebAssembly module and is platform-independent, allowing for a wide variety of multimedia applications.
This article describes both modes in detail and describes all API components required to use WASM Player in a WebAssembly App.
Typical usage scenarios
Flexible configuration options allow for Tizen WASM Player usage in a wide range of applications. However, two typical usage scenarios can be highlighted:
- Apps that use media container-based multimedia playback (e.g. DASH, HLS).
- Apps that use streaming protocol-based multimedia playback (e.g. RTSP).
Those two typical scenarios are described in detail below.
Please note that WASM Player's Latency Mode is orthogonal to the scenarios listed above. Latency Mode is decoupled from app's media data processing method (container- or streaming-based playback) and display mode (WASM Player renders content, app renders content using OpenGL). While low Latency Modes will usually be used with streaming protocols, they can be also used with container-based apps (and vice versa).
Usage scenario: media container-based multimedia playback app
In this approach:
- App manages downloading media containers (e.g. DASH, HLS).
- App incorporates a demuxer implementation which splits containers into Audio/Video Frames (e.g. using FFmpeg).
- Audio/Video Frames are sent to WASM Player to be rendered.
- Typically UI will be implemented in HTML5. However, it's also possible to implement UI (or parts of UI) using OpenGL directly in WebAssembly module instead.
Usage scenario: streaming protocol-based multimedia playback app
In this approach:
- App incorporates a streaming protocol implementation that controls and downloads media stream (e.g. RTSP).
- Audio/Video Frames are sent to WASM Player to be rendered.
- Typically UI will be implemented in HTML5. However, it's also possible to implement UI (or parts of UI) using OpenGL directly in WebAssembly module instead.
Tizen WASM Player comparison to MSE
Tizen WASM Player is a more low level alternative to MSE:
- Where MSE operates on media containers, WASM Player operates directly on encoded Audio/Video Frames (Elementary Media Packets).
- Tizen WASM Player offers a wide range of configuration options that allows tailoring media pipeline to specific needs of any given application (Latency Modes, an option to render video to video texture).
- It's app responsibility to split media into Elementary Media Packets when Tizen WASM Player is used. This approach offers greater flexibility compared to MSE, at the cost of increased App-side code complexity.
Please refer to the following table for Tizen WASM Player/MSE key difference highlights.
Feature | Tizen WASM Player | MSE |
---|---|---|
Accepted Media | Encoded Audio/Video Frames (Elementary Media Packets) | Media Containers |
Rendering Targets | HTMLMediaElement, OpenGL texture | HTMLMediaElement |
Latency Modes | Normal Latency, Low Latency, Ultra Low Latency (Game Mode) | Normal Latency |
Who ships demuxer? | App | Platform |
Latency Modes
Normal Latency
WASM Player in normal latency mode can be used to playback typical on-demand media. In this mode Platform maintains both multimedia pipeline clock and synchronization of all running streams (audio and/or video), whereas the WebAssembly application is required to fill WASM Player packet buffer with ES packets to be decoded and rendered by Platform. Normal latency mode supports all playback operations defined by WASM Player components, as well as DRM-protected content playback (see table comparing modes below).
Normal latency mode is the best choice when playback is not optimized for latency.
Pipeline clock & stream sync
When Normal Latency mode is used, Platform runs the multimedia pipeline according to a steady, monotonic internal clock. Rendering of packets sent to WASM Player is synchronized with the clock according to their timestamps. Pipeline time advances only when player's internal buffer allows continuous playback of all streams; if buffer underflow is detected, pipeline pauses.
Low Latency
WASM Player set to Low Latency mode can be used to playback streams which require low latency, such as media originating from online streams (e.g. football match). In this mode both pipeline clock and synchronization of multimedia pipeline is controlled entirely by the WebAssembly application. Platform renders ES Packets immediately after they are passed to WASM Player. Low latency mode allows playback with low latency, however access to some operations and DRMs is limited (see table comparing modes below). Due to a nature of low latency playback, B-frames cannot be used.
Low latency mode is the best choice for playing media from online streams, that should benefit from picture enhancements functionality of Smart TV, with only little overhead on latency.
Pipeline clock & stream sync
When Low Latency mode is used, Platform advances multimedia pipeline running time according to timestamps of packets passed to EMSS: whenever a packet is sent the pipeline clock is updated immediately based on it's timestamp.
Since Platform renders incoming Elementary Media Packets as they are sent and updates of the pipeline clock are directly dependent on App's actions, track synchronization is App's responsibility.
Ultra Low Latency
WASM Player set to Ultra Low Latency mode can be used to playback streams which require minimal latency, such as game streaming applications. In comparison to Low Latency mode, picture enhancements are disabled, providing further latency reduction.
Ultra low latency mode is the best choice for playing media from live sources that require lowest possible latency.
Pipeline clock & stream sync
Pipeline clock and stream synchronization of WASM Player set to Ultra Low Latency mode is the same as in Low Latency mode.
Latency modes comparison
Feature | Normal Latency | Low Latency | Ultra Low Latency |
---|---|---|---|
Pipeline clock & stream sync | controlled by Platform | controlled by App | controlled by App |
Runtime configuration change | resolution: yes fps: yes |
resolution: yes fps: N/A* |
resolution: yes fps: N/A* |
Seek | yes | N/A** | N/A** |
DRM Support | yes*** | no | no |
Example usage | VOD solutions; DASH, HLS, etc. | streaming solutions; RTSP, etc. | game streaming |
* - FPS rate doesn't apply in low latency mode, as rendering rate is dependent entirely on packet sending frequency (therefore fps is freely and entirely controlled by App).
** - Seek operation doesn't apply in low latency mode, as it implies working with a live, non-seekable source.
*** - DRM support is available only in conjunction with Media Element rendering mode.
Rendering modes
Media Element rendering mode
WASM Player set to Media Element rendering mode presents decoded video frames directly on the attached HTMLMediaElement.
This is the default rendering mode that provides support for both normal and encrypted playback.
Video texture rendering mode
WASM Player set to Video Texture rendering mode fills a provided texture with pictures decoded from the video stream. That WebGL texture can be rendered on a screen on canvas element.
Encrypted playback is not supported in this mode.
Rendering modes comparison
Feature | Media Element | Video Texture |
---|---|---|
DRM Support | yes* | no |
Example usage | VOD solutions | Video thumbnails |
* - DRM support is available only in conjunction with Normal Latency Mode.
API Guide
Architecture overview
WASM Player is meant to be used in Tizen TV widget applications that employ WebAssembly modules. Tizen TV widget applications work in HTML5 environment: WASM Player provides an ElementaryMediaStreamSource
component that is designed to be used as a source object of HTMLMediaElement.
In order to use Media Player with WASM Player, App should use classes it provides (which act as a source of data for playback) and the HTMLMediaElement
class (to control Media Player). Class responsibilities and functions are as follow:
Class | Description |
---|---|
ElementaryMediaStreamSource
|
This is the main class of Tizen WASM Player. It acts as a source object of HTMLMediaElement .ElementaryMediaStreamSource manages a set of ElementaryMediaTrack objects. Together, their purpose is essentially feeding Platform with media data required to play content; EMSS classes are acting as a source of media data.
|
ElementaryMediaTrack
|
Each instance of this class represents a single elementary media track (either audio or video). The track object allows sending Elementary Stream Packet data to Platform for a playback. |
HTMLMediaElement
|
HTMLMediaElement is used alongside WASM Player classes as a playback control element, providing operations like play, pause, seek, etc. That is, Media Element controls Media Player.HTMLMediaElement is associated with either
|
Relation between ElementaryMediaStreamSource and HTMLMediaElement
Please note that ElementaryMediaStreamSource
and HTMLMediaElement
serve different purpose in Media Player. As shown above, Elementary Media Stream Source objects act as a mean to deliver media data to HTMLMediaElement
for playback. Controlling playback (e.g. issuing Play, Pause, Seeking, etc.) on Media Player is done via HTMLMediaElement
API. Elementary Media Stream Source allows to deliver media data in form of Elementary Media Packets.
When operating Media Player, App should rely on HTMLMediaElement
events and methods. For example, ElementaryMediaStreamSource
signals when Media Player requires media data. However, the moment Media Player starts reading media data is not always the very same moment a playback can be started. Readiness for playback is signalized by HTMLMediaElement
by the means of HTMLMediaElementListener::OnCanPlay()
event.
ElementaryMediaStreamSource's states
Ready State | Description |
---|---|
kDetached
|
ElementaryMediaStreamSource is not assigned to HTMLMediaElement .This is an initial state of ElementaryMediaStreamSource object. It is also entered after disconnecting from HTMLMediaElement .
|
kClosed
|
Tracks are not configured and player is not initialized. Can't play in this state.ElementaryMediaTrack objects can be added to and removed from ElementaryMediaStreamSource . Track layout can be changed only in this state.This state is entered after ElementaryMediaStreamSource is attached to HTMLMediaElement , when unrecoverable playback error occurs or on App request.
|
kOpen
|
Media Player is fully initialized and Elementary Media Stream Source is ready to accept Elementary Packet Data from App via ElementaryMediaTrack objects.When App finishes configuring Tracks, it can request entering this state. Open state is entered when possible. Some operations (like Seek) trigger a temporary transition to kOpenPending state.
|
kOpenPending
|
Opening ElementaryMediaStreamSource was requested by App, however current state of multimedia pipeline prevents entering the kOpen state. The kOpen state will be entered as soon as possible.This is a temporary state, entered on several occasions:
|
kEnded
|
Stream has ended but multimedia pipeline remains initialized. This state is entered when App marks active ElementaryMediaTrack objects as ended. The kEnded state reverts to the kOpen state when multimedia pipeline resumes playback (e.g. due to Seek).
|
ElementaryMediaStreamSource
's and HTMLMediaElement
's states should not be confused. ElementaryMediaStreamSource
represents state of a source of data, while HTMLMediaElement
represents state of a multimedia player. As such, ElementaryMediaStreamSource
signals App whether or not multimedia pipeline needs data and can accept Elementary Media Packets. This is not in sync with multimedia player's state.
For example, the kEnded
state can be associated with a still-playing Media Element. This occurs in Normal Latency mode when App signals end of all tracks but multimedia pipeline has buffered Elementary Media Packets remaining to play.
The kOpen state and Latency Modes
Depending on the Latency Mode the ElementaryMediaStreamSource
's preconditions for entering the kOpen
state are slightly different.
- In Normal Latency mode, requesting
ElementaryMediaStreamSource
to open initializes Media Player and immediately triggers it's data buffering mechanism. Therefore, Elementary Media Stream Source enters thekOpen
state shortly after open is requested (briefly entering thekOpenPending
state during initialization). As soon as thekOpen
state is entered the App is expected to send media data to EMSS. - In Low Latency or Ultra Low Latency mode, requesting
ElementaryMediaStreamSource
to open puts it in thekOpenPending
state and initializes Media Player. ThekOpen
state (where App is expected to send media data to EMSS) is entered when Play is issued onHTMLMediaElement
. Source remains open as long as Media Element is in playing state. Media Player renders any Elementary Media Packets immediately after they are sent, with no intermediate buffering.
There is a number of reasons for EMSS to switch between states. Usage Guide provides detailed information on when such a change happens.
ElementaryMediaTrack's states
ElementaryMediaTrack
state reflects that of an associated ElementaryMediaStreamSource
, however in a simplified manner. Track can be either open or closed, represented by a single boolean attribute.
IsOpen | Description |
---|---|
true | Track can accept Elementary Media Packets. Track opens just before Source enters the kOpen state.
|
false | Track can't accept Elementary Media Packets. Track closes just after Source leaves the kOpen state.
|
When track closes, an event is generated carrying a reason why close occurred. Possible close reasons can be checked here.
HTMLMediaElement's states
C++ HTMLMediaElement
class supplied with Elementary Media Stream Source is a wrapper for a HTML5's HTMLMediaElement
. Please refer to HTML Media Element Specification for information on HTMLMediaElement
's states.
WASM Player and App Multitasking
App that targets Tizen Smart TV work in a multitasking environment and must respond to multitasking appropriately. For a multimedia application this often means it have to suspend media playback when the application becomes invisible (e.g. goes into background when user launches another application) and resume playback when the application becomes visible (e.g. goes into foreground when user resumes App).
WASM Player tracks an application's multitasking state and generates appropriate events for App to react to. Specifics of event sequence depends on a selected EMSS mode of operation.
- For more details on supporting Multitasking see Usage Guide.
- For more details on Multitasking on Tizen TV see SmartTV Widget Multitasking Guide.
Supported media formats
Please refer to SmartTV Media Specifications for information on supported codecs and configurations.