WASM Player Usage Guide
This section provides a detailed Tizen WASM Player API usage guide.
Related Info
- Tizen WASM Player API Documentation
- Samsung SmartTV Media Specification
- Samsung SmartTV Widget Multitasking Guide
- WebAssembly for Tizen TV Download
- W3C HTML Media Element Specification
- Tizen WASM Player Sample (Normal Latency)
- Moonlight port to Tizen WASM (Low Latency)
Interacting with Tizen WASM Player
Tizen WASM Player provides an ElementaryMediaStreamSource
component that acts as a source object for HTMLMediaElement
and provides a low level multimedia API targeting WebAssembly that enables App to work with Media Player on elementary media packets level.
A figure below shows general idea of interaction between App and Media Player when using WASM Player:
App manages Media Player (controlled by HTMLMediaElement
) and sends Elementary Media Packet data via ElementaryMediaTrack
objects.
The way App interacts with WASM Player is partially dependent on the selected mode of operation, however most of the operations are common to all modes. Paragraphs below describe available operations in detail, highlighting areas where the selected mode influences behavior of the source.
Initializing Media Player to work with WASM Player
Setting up HTMLMediaElement with ElementaryMediaStreamSource as a source object
In WASM Player, the ElementaryMediaStreamSource
class acts as a source object for HTMLMediaElement
. In order to set up ElementaryMediaStreamSource
and HTMLMediaElement
to work together:
- Create
HTMLMediaElement
object:auto html_media_element = std::make_unique<samsung::html::HTMLMediaElement>(element_id);
Note :HTMLMediaElement
class connects to either a<video>
or an<audio>
element in a HTML document hosting WebAssembly module. The HTML element is chosen by theelement_id
specified during construction.
- Create an object implementing
HTMLMediaElementListener
interface and associate it with the media element. This step is optional, however events generated by the media element are essential for creating a functional media player.-
First, define a custom implementation of
HTMLMediaElementListener
:class MyMediaElementListener : public samsung::html::HTMLMediaElementListener { // override virtual methods that App uses };
-
Instantiate the custom listener and associate it with the
HTMLMediaElement
object:auto my_media_element_listener = std::make_unique<MyMediaElementListener>(); html_media_element->SetListener(my_media_element_listener.get());
Note :Please note
HTMLMediaElement::SetListener()
method does not take ownership of the listener. Make sure that the listener outlives the media element object!
-
- Create
ElementaryMediaStreamSource
object:using LatencyMode = samsung::wasm::ElementaryMediaStreamSource::LatencyMode; using RenderingMode = samsung::wasm::ElementaryMediaStreamSource::RenderingMode; auto elementary_media_stream_source = std::make_unique<samsung::wasm::ElementaryMediaStreamSource>(LatencyMode::kNormal, RenderingMode::kMediaElement);
Note :Choose
LatencyMode::kNormal
to set up WASM Player to work in Normal Latency mode,LatencyMode::kLow
for Low Latency mode andLatencyMode::kUltraLow
for Ultra Low Latency mode.
Choose RenderingMode::kMediaElement
to set up WASM Player to work in Media Element rendering mode or RenderingMode::kVideoTexture
for Video Texture rendering mode.
- Create an object implementing
ElementaryMediaStreamSourceListener
interface and associate it with the source. This step is optional, however events generated by the source are essential for creating a functional media player.-
First, define a custom implementation of
ElementaryMediaStreamSourceListener
:class MySourceListener : public samsung::wasm::ElementaryMediaStreamSourceListener { // override virtual methods that App uses };
-
Instantiate the custom listener and associate it with the
ElementaryMediaStreamSource
object:auto my_source_listener = std::make_unique<MySourceListener>(); elementary_media_stream_source->SetListener(my_source_listener.get());
Note :Please note
ElementaryMediaStreamSource::SetListener()
method does not take ownership of the listener. Make sure it outlives the source object!
-
-
Attach
ElementaryMediaStreamSource
toHTMLMediaElement
:html_media_element->SetSrc(elementary_media_stream_source.get());
Note :When created,
ElementaryMediaStreamSource
is set to theReadyState::kDetached
state, preventing any operations on the source. Attaching it toHTMLMediaElement
will change source's state to theReadyState::kClosed
state, allowing configuration of media tracks.
Adding ElementaryMediaTracks to ElementaryMediaStreamSource
Once ElementaryMediaStreamSource
is created and attached to HTMLMediaElement
, App should configure tracks that takes part in the media playback. Depending on its needs App can add up to one video track and up to one audio track to the source, as described below.
Preparing video track config
To prepare an ElementaryVideoStreamTrackConfig
structure, fill it with an initial configuration of a video track:
samsung::wasm::ElementaryVideoStreamTrackConfig conf;
conf.mimeType = "video/mp4; codecs=\"hev1.1.6.L93.B0\""; // h265
conf.extradata = { /* codec extradata */ };
conf.width = 1920;
conf.height = 1080;
conf.framerateNum = 60;
conf.framerateDen = 1;
conf.decodingMode = samsung::wasm::DecodingMode::kHardware;
The configuration provided when a video track is added is an initial configuration of the track. If the configuration changes during the stream playback (e.g. due to an adaptive streaming), existing track can be reconfigured.
Please refer to Samsung Smart TV Media Specifications for information on supported codecs and configurations.
Preparing audio track config
To prepare an ElementaryAudioStreamTrackConfig
structure, fill it with a configuration of an audio track:
samsung::wasm::ElementaryAudioStreamTrackConfig conf;
conf.mimeType = "audio/mp4; codecs=\"mp4a.40.2\""; // AAC
conf.extradata = { /* codec extradata */ };
conf.sampleFormat = samsung::wasm::SampleFormat::kPlanarF32;
conf.channelLayout = samsung::wasm::ChannelLayout::kStereo;
conf.samplesPerSecond = 48000;
conf.decodingMode = samsung::wasm::DecodingMode::kHardware;
The configuration provided when an audio track is added is the final configuration of the track. Changing configuration during the stream runtime is not possible.
Adding a track to the source
As soon as a track's configuration is ready, the track can be added to the source. This procedure is the same for both audio and video tracks.
-
Add a track to the source:
auto track = elementary_media_stream_source->AddTrack(conf);
-
Create an object implementing
ElementaryMediaTrackListener
interface and associate it with the track. This step is optional, however events generated by the track are essential for implementing certain media player functionalities.-
First, define a custom implementation of
ElementaryMediaTrackListener
:class MyTrackListener : public samsung::wasm::ElementaryMediaTrackListener { // override virtual methods that App uses };
-
Instantiate the custom listener and associate it with the
ElementaryMediaTrack
object:auto my_track_listener = std::make_unique<MyTrackListener>(); track->SetListener(my_track_listener.get());
Note :Please note that
ElementaryMediaTrack::SetListener()
method does not take ownership of the listener. Make sure that the listener outlives the track object!
-
Setting media duration
Media duration cannot be set in Low Latency modes, where duration is automatically set to infinite.
Duration of media should be set on the ElementaryMediaStreamSource
instance during initialization:
auto content_duration = samsung::wasm::Seconds{42};
elementary_media_stream_source->SetDuration(content_duration);
Once set, the duration is not fixed. It can also be changed in runtime if playback scenario requires it.
Completing Media Player initialization
When App configures track layout, ElementaryMediaStreamSource
can be initialized by putting it in ReadyState::kOpen
and thus enabling playback.
- Request
ElementaryMediaStreamSource
to open:
Running Open operation on the source immediately puts it in theusing OperationResult = samsung::wasm::OperationResult; elementary_media_stream_source->Open([](OperationResult result) { if (result != OperationResult::kSuccess) { // handle error } });
ReadyState::kOpenPending
state. That means the source will enter theReadyState::kOpen
state as soon as possible, however this is not neccesarily the moment when Open operation finishes! For details see Starting playback after initialization paragraph below.
Starting Playback
Normal Latency mode
Normal Latency playback in WASM Player is meant for VOD scenarios. In this mode, Platform's Media Player buffers Elementary Media Packet data sent by App and ensures it plays smoothly.
Starting playback
Once ElementaryMediaStreamSource
enters ReadyState::kOpen
(thus opening ElementaryMediaTrack
), App can start buffering media data. In Normal Latency mode, a number of packets needs to be buffered and processed by Platform before the playback can be started. App should observe HTMLMediaElement
events (HTMLMediaElementListener::OnCanPlay()
) to detect this moment or HTMLMediaElement
autoplay feature.
In Normal Latency mode opening and closing of ElementaryMediaStreamSource
and ElementaryMediaTrack
objects are not in sync with HTMLMediaElementListener::OnCanPlay()
. WASM Player objects open when Platform is ready to receive media data and leave open state when packets cannot be received anymore. Elementary Media Packets that are sent affect both the ready state and events emitted by HTMLMediaElement
, depending on Platform's internal media data buffer level. Therefore App should observe HTMLMediaElement
via HTMLMediaElementListener
for entire Media Player's state.
To start playback in Normal Latency mode:
-
Configure Media Player as described in Initializing Media Player to work with WASM Player paragraph above.
-
Source enters
ReadyState::kOpen
and each track emitsElementaryMediaTrackListener::OnTrackOpen()
event to notify that it can accept elementary media data.class MyTrackListener : public samsung::wasm::ElementaryMediaTrackListener { // ... void OnTrackOpen() override { // Start App's packet sending mechanism for the associated track. } // ... };
Note :Alternatively, App can wait on
ElementaryMediaStreamSourceListener::OnSourceOpen()
event emitted by the source itself.OnTrackOpen
events are generated beforeOnSourceOpen
, so it's safe to trigger packet sending mechanism in this event handler too.
OnTrackOpen
and OnSourceOpen
events are generated whenever a track opens: starting playback after initialization is the most prominent case, however it is also the case during Seek, when application is Multitasking, etc. No matter the reason, the first packet sent to a newly opened ElementaryMediaTrack
must be a keyframe.
-
Build up internal Media Player data buffer by sending a number of packets. This is basically the same mechanism as described below in Sending Elementary Media Packets paragraph below.
-
When
HTMLMediaElement
is ready to start playback, it emitsHTMLMediaElementListener::OnCanPlay()
event. Handle the event and callHTMLMediaElement::Play()
.class MyMediaElementListener : public samsung::html::HTMLMediaElementListener { // ... void OnCanPlay() override { html_media_element->Play([](OperationResult result) { if (result != OperationResult::kSuccess) { // handle error } }); } // ... };
See Sending Elementary Media Packets paragraph below for information regarding sending media data to Media Player.
Low Latency modes
Low Latency playback in WASM Player is meant for live streaming scenarios. In this mode Platform renders Elementary Media Packets immediately after App sends them. Therefore it is up to App to maintain pipeline clock and stream synchronization.
Starting playback
In Low Latency modes no buffering is required to start media playback. Media Player is ready to start playback at any moment, however WASM Player can accept packets only when HTMLMediaElement
is in playing state.
When ElementaryMediaStreamSource::Open()
is called, ElementaryMediaStreamSource
is immediately set in ReadyState::kOpenPending
state. Once App calls HTMLMediaElement::Play()
, the source enters ReadyState::kOpen
state and ElementaryMediaTrack
s open. From that point on Elementary Media Packets will be accepted and rendered immediately.
To start playback in Low Latency modes:
-
Configure Media Player as described in Initializing Media Player to work with WASM Player paragraph above.
-
When App is ready to start playback, call
HTMLMediaElement::Play()
:html_media_element->Play([](OperationResult result) { if (result != OperationResult::kSuccess) { // handle error } });
-
Source enters
ReadyState::kOpen
and each track emitsElementaryMediaTrackListener::OnTrackOpen()
event to notify it can accept elementary media data.class MyTrackListener : public samsung::wasm::ElementaryMediaTrackListener { // ... void OnTrackOpen() override { // Start App's packet sending mechanism for the associated track. } // ... };
Note :Alternatively, App can wait on
ElementaryMediaStreamSourceListener::OnSourceOpen()
event emitted by the source itself.OnTrackOpen
events are generated beforeOnSourceOpen
, so it's safe to trigger packet sending mechanism in this event handler too.
OnTrackOpen
and OnSourceOpen
events are generated whenever a track opens: starting playback after initialization is the most prominent case, however it is also the case when playback is Paused and then Unpaused, application is Multitasking, etc. No matter the reason, the first packet sent to a newly opened ElementaryMediaTrack
must be a keyframe.
Sending Elementary Media Packets
Elementary media stream playback with WASM Player relies on App sending Elementary Media Packets to the Platform. This is done via ElementaryMediaTrack
objects, each representing a single media track.
Pipeline, clock and Mode
- In Normal Latency mode Platform buffers elementary media data, guaranteeing a smooth playback as long as enough data is buffered.Note :
If Platform's internal media data buffer level is too low to maintain a steady playback,
HTMLMediaElement
pauses the playback. Media element resumes the playback automatically when enough data is buffered.
- In Low Latency modes Platform renders elementary media data immediately, allowing App to implement live streaming scenarios.Note :
Stream running time represented by
HTMLMediaElement::GetCurrentTime()
advances according to timestamps of Elementary Media Packets sent by App.
AppendPacket() operation
App uses an ElementaryMediaTrack::AppendPacket()
operation to send media data to Media Player. Remarks:
- Elementary Media Packets should be appended to each individual track in decoding order (i.e. sorted by decoding timestamp),
- appends do not need to maintain decoding order between tracks, however it is recommended they do (this does not apply in Low Latency mode, where rendering speed is entirely dependent on App and no internal buffering occurs),
- B-Frames should not be used in Low Latency mode.
A mechanism that sends Elementary Media Packets to WASM Player should follow the scheme below:
-
Prepare
ElementaryMediaPacket
structure for the next append. The packet to append is a packet with the lowest structure dts among packets that were not sent to WASM Player yet.samsung::wasm::ElementaryMediaPacket packet; packet.pts = /* presentation timestamp in samsung::wasm::Seconds */; packet.dts = /* decoding timestamp in samsung::wasm::Seconds */; packet.duration = /* packet duration in samsung::wasm::Seconds */; packet.is_key_frame = /* bool value indicating if packet is a keyframe */; packet.data = /* pointer to packet data payload */; packet.data_size = /* size of the data payload in bytes */; packet.session_id = /* current session id */
Note :Usually all audio packets are keyframes.
Multithreaded Apps should use session_id. For details, please see Session id paragraph in Threading section.
App can set additional parameters if the packet in question is a video frame:
// Set either both or none; can be 0 unless resolution changed.
packet.width = /* 0 or width */;
packet.height = /* 0 or height */;
// Set either both or none; can be 0 unless resolution changed.
packet.framerate_num = /* 0 or value */;
packet.framerate_den = /* 0 or value */;
Video-only parameters come in pairs: .width
and .height
, .framerate_num
and .framerate_den
. Each pair is optional and doesn't have to be specified unless track parameter they describe changes from this packet onwards. See Runtime configuration change paragraph for more details on reconfiguring ElementaryMediaTrack
.
Video-only packet parameters must be zeroed for audio packets!
- Send the packet by appending it to the corresponding
ElementaryMediaTrack
:auto result = track->AppendPacket(packet); if (!result) { // handle error }
AppendPacketAsync()
can be used as an alternative operation to AppendPacket()
. It is functionally equivalent to sync method - it will validate the packet and return a result synchronously. However, unlike AppendPacket()
, AppendPacketAsync()
operation can be called from the main thread.
Calls to both methods for the same track can be mixed.
Managing amount of buffered data
This paragraph does not apply in Low Latency modes, as it does not buffer data on Platform's side.
In order to maintain a smooth playback, App should maintain a steady flow of Elementary Media Packets that will prevent underrun of Platform's media data buffer. As a rule of thumb, it's good to buffer a couple of seconds worth of packets for each track, with the following limitations:
- for clear (non-encrypted) content playback at most 64 MiB of data can be buffered,
- for DRM-protected content playback at most 10 MiB of data can be buffered,
- for audio track at most 768 KiB of data can be buffered,
- App shouldn't buffer more than 3 seconds of content in advance.
If Platform's media data buffer underrun occurs, HTMLMediaElement
pauses the playback. Playback is resumed automatically when enough packets are buffered.
In order to keep track of amount of data that should be buffered, App should use ElementaryMediaStreamSourceListener::OnPlaybackPositionChanged()
event. This event is emitted periodically to update current playback time and can be used to calculate amount of data to buffer:
// Controls how many packets should be buffered ahead of a current playback
// position.
constexpr samsung::wasm::Seconds kBufferAhead = Seconds{3.};
class MySourceListener : public samsung::wasm::ElementaryMediaStreamSourceListener {
using Seconds = samsung::wasm::Seconds;
// ...
void OnPlaybackPositionChanged(Seconds new_time) override {
// Buffer packets up to timestamp (new_time + kBufferAhead).
}
// ...
};
:::
ElementaryMediaStreamSourceListener::OnPlaybackPositionChanged()
event is similar to HTMLMediaElementListener::OnTimeUpdate()
event, however the one associated with HTMLMediaElement
should not be used to control a data source.
Events associated with each of those two components are associated with a state of the respective component. States of ElementaryMediaStreamSource
(data source) and HTMLMediaPlayer
(Media Player control element) are not in sync. An example would be Seek operation: data source can be done seeking when Media Player is still in the middle of the Seek. Therefore HTMLMediaElementListener::OnTimeUpdate()
is not reliable in the context of a data source.
:::
Threading considerations
Usually App will be sending Elementary Media Packets on a worker thread (see Threading paragraph below). However, ElementaryMediaStreamSourceListener
events are delivered on the main thread (via main JS message loop). If triggering packet buffering involves thread synchronization, it's advised to limit amount of inter-thread communication that involves locking the main thread to a minimum.
Most of the events delivered from both ElementaryMediaStreamSource
and HTMLMediaElement
are scarce. The only exception are time updates, which are delivered on a regular basis, potentially with a high frequency. App should fine-tune how often such events are handled. Usually updating buffering mechanism once or twice per second is sufficient.
When to stop sending packets
App can send packets as long as ElementaryMediaTrack
s remain open and ElementaryMediaStreamSource
is in ReadyState::kOpen
state:
class MyTrackListener : public samsung::wasm::ElementaryMediaTrackListener {
// ...
void OnTrackClosed(ElementaryMediaTrack::CloseReason close_reason) override {
// Stop App's packet sending mechanism for the associated track.
}
// ...
};
Alternatively, App can detect when ElementaryMediaStreamSource
leaves ReadyState::kOpen
state using ElementaryMediaStreamSourceListener
. OnTrackClosed
events are always generated after ElementaryMediaStreamSource
state change handlers are executed, so stopping packet sending mechanism there is safe. However, Source can enter multiple states from the ReadyState::kOpen
state, so using a track listener is preferred.
Platform can close tracks due to both App actions and the user interaction with the device. App should properly handle close event and expect tracks to close at any moment. The most notable occurrence that causes tracks to close is the user switching active application (see Multitasking section). Another common example would be the user performing Seek operation (see Seek section).
End of stream
This paragraph does not apply in Low Latency modes, as stream duration is set to infinite there.
When playback is coming to an end and all remaining Elementary Media Packets were buffered, App should mark each track as ended. It can be done by using one of 2 possible methods: AppendEndOfTrack()
or AppendEndOfTrackAsync()
. AppendEndOfTrack()
is working synchronously - it returns a result when operation is finished (track was closed or an error occurred):
auto result = track->AppendEndOfTrack(session_id);
if (!result) {
// handle error
}
AppendEndOfTrackAsync()
method returns a result synchronously when end of track was appended and track starts closing (or an error occurred before that happened). All errors that occur when track is closing will be signalled via ElementaryMediaTrackListener::OnAppendError()
event. For details, please see Handling async append errors section.
Multithreaded Apps should use session_id. For details, please see Session id paragraph in Threading section.
While each track is marked as ended separately, all the tracks should end on a timestamp close to the stream duration set with ElementaryMediaStreamSource::SetDuration()
.
Seek
This paragraph does not apply in Low Latency modes, as it does not support Seeking.
Seek operation allows the user to jump to a new playback position. It is performed either via HTMLMediaElement
's interface when controls
are enabled or programmatically by calling HTMLMediaElement::SetCurrentTime()
.
Seek algorithm
Whenever a new current time value is set:
-
If
ElementaryMediaStreamSource
isReadyState::kOpen
:ElementaryMediaStreamSource
state is changed toReadyState::kOpenPending
,ElementaryMediaStreamSourceListener::OnSourceOpenPending()
is fired,
ElementaryMediaTrack
s are closed,ElementaryMediaTrackListener::OnTrackClosed()
withCloseReason::kTrackSeeking
is fired for each track,
ElementaryMediaStreamSourceListner::OnSessionIdChanged()
is fired for each track.
-
ElementaryMediaStreamSourceListner::OnSeek()
is fired wit a new current time. -
If
ElementaryMediaStreamSource
wasReadyState::kOpen
prior Seek:ElementaryMediaTrack
s are open,ElementaryMediaTrackListener::OnTrackOpen()
is fired for each track,
ElementaryMediaStreamSource
state is changed toReadyState::kOpen
,ElementaryMediaStreamSourceListener::OnSourceOpen()
is fired,
When multiple Seeks happen in a close succession, it's possible that step 2 will be repeated multiple times. That is, multiple ElementaryMediaStreamSourceListner::OnSeek()
events will be delivered between Source open and Source close.
Loop
This paragraph does not apply in Low Latency modes, as low latency streams have infinite duration.
If HTMLMediaElement
has the loop
property set to true, WASM Player automatically generates a Seek to 0s when playback reaches end of stream.
Multitasking
Multitasking happens when App is either hidden or shown on the device's screen (e.g. when the user switches active application). App should always properly handle Multitasking.
Suspend algorithm
When App is suspended (i.e. visibilitychange
reports document.hidden
change to true):
- If
ElementaryMediaStreamSource
isReadyState::kOpen
:ElementaryMediaStreamSource
state is changed toReadyState::kOpenPending
,ElementaryMediaStreamSourceListener::OnSourceOpenPending()
is fired,
ElementaryMediaTrack
s are closed,ElementaryMediaTrackListener::OnTrackClosed()
withCloseReason::kSourceSuspended
is fired for each track.
- If
HTMLMediaElement::IsPaused()
attribute is false:HTMLMediaElement
is paused.
HTMLMediaElement::GetCurrentTime()
is stored internally by WASM Player.
(this step is omitted in Low Latency modes)
Resume algorithm
When App is resumed (i.e. visibilitychange
reports document.hidden
change to false):
- If
ElementaryMediaStreamSource
wasReadyState::kOpen
prior to suspend, an automatic Seek is performed to the storedHTMLMediaElement::GetCurrentTime()
.
As a result,ElementaryMediaStreamSource
is set toReadyState::kOpen
and eachElementaryMediaTrack
is opened.
(In Low Latency mode Seek is omitted; instead Source and Tracks are opened immediately) - If
HTMLMediaElement::IsPaused()
was false prior suspend:HTMLMediaElement
is unpaused.
Supporting DRMs
This paragraph does not apply in Low Latency modes, as it does not support DRMs.
WASM Player supports DRM-protected content playback. Supported formats are equivalent to those specified in EME section in Samsung Smart TV Media Specifications.
In order to set up encrypted media playback App should do as follow:
-
Prepare a
DRMConfig
structure:samsung::wasm::DRMConfig drm_config; drm_config.cdm = /* a samsung::wasm::ContentDecryptionModule value */; drm_config.encryption_mode = /* a samsung::wasm::EncryptionMode value */; drm_config.license_server /* an URL to a license server */; drm_config.init_data = /* DRM System-specific initialization data. */; // If video track is encrypted: drm_config.audio_mime_type = /* a mime of encrypted audio or empty string if track is clear */; drm_config.audio_robustness = /* a samsung::wasm::Robustness parameter for audio track */; // If video track is encrypted: drm_config.video_mime_type = /* a mime of encrypted video or empty string if track is clear */; drm_config.video_robustness = /* a samsung::wasm::Robustness parameter for video track */;
-
Create a
MediaKey
using the config created in the previous step and assign it to a correspondingElementaryMediaTrack
s:using OperationResult = samsung::wasm::OperationResult; using MediaKey = samsung::wasm::MediaKey; // ... // Created MediaKey must be stored by App: MediaKey decryption_key; // ... auto result = samsung::wasm::MediaKey::SetupEncryption(drm_config, [](OperationResult result, MediaKey media_key) { if (result != OperationResult::kSuccess) { // handle error return; } // Store the key. Make sure it outlives tracks it is assigned to below! decryption_key = std::move(media_key); // if audio is encrypted... audio_track->SetMediaKey(&decryption_key) // if video is encrypted... video_track->SetMediaKey(&decryption_key) })
Note :When creating a
MediaKey
instance, implementation automatically contacts the license server specified in thedrm_config.license_server
parameter.
- When multimedia content is DRM-protected, App should follow normal usage guidelines for Elementary Media Stream source except for one difference: encrypted Elementary Media Packets are sent to
ElementaryMediaTrack
s with anAppendEncryptedPacketAsync
method:samsung::wasm::EncryptedElementaryMediaPacket encrypted_packet; // ... // Set all clear packet information as usual. // ... encrypted_packet.subsamples = /* subsample information: a vector of samsung::wasm::EncryptedSubsampleDescription */; encrypted_packet.key_id = /* id of a key that should be used to decrypt this packet */; encrypted_packet.initialization_vector = /* IV required to decrypt this packet */; encrypted_packet.encryption_mode = /* samsung::wasm::EncryptionMode for this packet */; auto result = track->AppendEncryptedPacketAsync(encrypted_packet); if (!result) { // handle error }
AppendEncryptedPacketAsync()
will validate the packet and return a result synchronously. However, packet decryption will be executed asynchronously. If a decryption error occurs, it will be signalled via ElementaryMediaTrackListener::OnAppendError()
event. For details, please see Handling async append errors section.
In order to receive decryption result synchronously, it is possible to use AppendEncryptedPacket()
operation. It is an alternative method to AppendEncryptedPacketAsync()
. However, decrypting packets synchronously is not recommended - decryption is potentially a long running operation and blocking append thread for extended periods of time can reduce video playback performance.
Calling sync and async appends for the same track can be mixed.
Handling async append errors
If AppendPacketAsync()
, AppendEncryptedPacketAsync()
or AppendEndOfTrackAsync()
are used, App should detect async append errors using ElementaryMediaTrackListener
. The OnAppendError()
event should be handled in a custom implementation of ElementaryMediaTrackListener
:
class MyMediaElementListener : public samsung::wasm::ElementaryMediaTrackListener {
using OperationResult = samsung::wasm::OperationResult;
// ...
// override virtual OnAppendError method
void OnAppendError(OperationResult result) override {
// handle append error
}
// ...
};
Runtime configuration change
A playing stream can be reconfigured in the runtime, to an extent. Changes to the following parameters are supported in runtime:
Track type | Parameter |
---|---|
Video | Resolution |
Video | framerate |
Configuration change must always happen on a keyframe. Depending on a codec in use, there may be additional requirements regarding a packet that triggers runtime reconfiguration (e.g. a h264 packet may be required to contain PPS NAL).
Changing resolution
In order to change stream resolution, send a keyframe packet that has the new resolution set:
ElementaryMediaPacket packet;
// ...
packet.width = /* new width */;
packet.height = /* new height */;
// ...
Data payload of the packet must contain appropriate initialization data for the new configuration, depending on a codec that is currently in use.
Changing framerate
In order to change stream framerate, send a keyframe packet that has the new framerate set:
ElementaryMediaPacket packet;
// ...
packet.framerate_num = /* new framerate_num */;
packet.framerate_den = /* new framerate_den */;
// ...
Data payload of the packet must contain appropriate initialization data for the new configuration, depending on a codec that is currently in use.
To avoid video glitches, it is recommended that the new FPS is a multiple of the old one. The 30 <-> 60 FPS transitions are safe, as well as 29.97 <-> 59.94.
Handling pipeline errors
A multimedia pipeline in Platform runs asynchronously. Elementary media packets are moved via pipeline and processed after AppendPacket()
operation is finished. WASM Player verifies as much data as it's feasible when it's API is called, however some errors can still happen asynchronously (e.g. decoding errors).
App should detect asynchronous errors using HTMLMediaElementListener
. The OnError()
event should be handled in a custom implementation of HTMLMediaElementListener
:
class MyMediaElementListener : public samsung::html::HTMLMediaElementListener {
using MediaError = samsung::html::MediaError;
// ...
// override virtual OnError method
void OnError(MediaError error_code, const char* error_msg) override {
// handle media error
}
// ...
};
An ownership of error_msg
is not transferred and the pointer is valid only for the duration of the OnError()
call. Do not free this pointer manually!
Threading
Media applications tend to make use of threads, to both speed up multimedia processing and free the main thread that is responsible for handling UI. While JavaScript lacks a convenient thread support, WebAssembly offers a full thread support: using threads with WASM Player is encouraged.
Common thread structure
Main application thread is a JS main thread that runs JS event loop. This is the thread that runs all JS events (this includes an UI/user interaction and App's JS code not running in WebWorkers) and can also execute WASM code. However, App shouldn't execute long running operations in WASM module on a main thread.
App should offload work to a number of worker threads to increase App's performance and ensure UI responsiveness. For a multimedia application, worker threads can be used to, for example:
- download data (e.g. when sockets are used),
- process content (e.g. demux containers),
- send and process Elementary Media Packets.
Thread affinity of listeners
All events emitted by listeners related to WASM Player (i.e. ElementaryMediaStreamSourceListener
, ElementaryMediaTrackListener
, HTMLMediaElementListener
) are delivered via main JS thread.
Please note:
- event handlers shouldn't include any long-running operations; if any is necessary, it should be dispatched to a worker thread,
- excessive locking of a main JS thread is not advised and should be avoided whenever possible.
Session id
A session id is an ElementaryMediaTrack
parameter associated with all ElementaryMediaPacket
s sent to a track between OnTrackOpen
and OnTrackClose
.
A session starts when track opens (OnTrackOpen
) and lasts until it closes (OnTrackClose
). All packets sent between those two events belong to a single session. When appending either a packet or an end of track to ElementaryMediaTrack
, App should mark them with the current session_id
value.
When session id changes, an ElementaryMediaTrackListener::OnSessionIdChanged()
event is fired:
class MyTrackListener : public samsung::wasm::ElementaryMediaTrackListener {
// ...
void OnSessionIdChanged(uint32_t session_id) override {
// Update stored session_id: propagate it to components managing
// ElementaryMediaTracks.
}
// ...
};
ElementaryMediaPacket
s marked with non-current session ids are dropped by Platform. This causes AppendPacket()
to return OperationResult::kAppendIgnored
error, however it's not a fatal condition and doesn't impair playback in a properly written App.
Detecting available WASM Player features
Features supported by WASM Player can vary depending on Platform version. Availability of the features (and presence of the ElementaryMediaStreamSource
class implementation itself) can be determined in runtime by checking the EmssVersionInfo
structure entries:
// Query platform for WASM Player features.
auto features = samsung::wasm::EmssVersionInfo::Create();
if (features.has_emss |
features.has_legacy_emss) {
// Above condition evaluates to true if ElementaryMediaStreamSource data
// source is available on the Platform.
std::cout << "Platform supports WASM Player!" << std::endl;
}
Additionally, if WASM Player is not available on the Platform but source object is created, it's ElementaryMediaStreamSource::IsValid()
method will always return false
.
Getting detailed WASM Player version information
Using the EmssVersionInfo
structure is the recommended way to check features supported by WASM Player on the current device. However, this method doesn't provide WASM Player version information directly. Detailed version information can be checked with Tizen TV WASM versioning APIs:
#include <algorithm>
#include <samsung/wasm/tizen_tv_api_info.h>
// ...
using TizenTVApiInfo = samsung::wasm::TizenTVApiInfo;
auto apis = samsung::wasm::GetAvailableApis();
auto api_iterator =
std::find_if(apis.begin(), apis.end(), [](const TizenTVApiInfo& api) {
return api.name == "ElementaryMediaStreamSource";
});
if (api_iterator != apis.end()) {
// WASM Player is available on the current device! api_iterator entries can
// be checked for detailed version information.
}
The TizenTVApiInfo
structure has following entries:
name
, as seen above. WASM Player is identified by it's main class name:ElementaryMediaStreamSource
,version
, which is a string in format:API_LEVEL.DETAILED_VERSION
(e.g.1.0
),api_levels
, astd::vector<uint32_t>
of all API levels supported on the current device.
API Level identifies a single WASM Player API revision, which maps to EmssVersionInfo
entries. Detailed version is internal implementation revision id.
Feature availability per Tizen version
WASM Player features available on the current device can be conveniently checked by looking up the EmssVersionInfo
structure entries:
EmssVersionInfo.
|
Description | API Level |
---|---|---|
has_ultra_low_latency
|
WASM Player LatencyMode::kUltraLow mode is available on this device
|
5 |
has_low_latency_video_texture
|
WASM Player LatencyMode::kLow in conjunction with RenderingMode::kVideoTexture is available on this device
|
4 |
has_decoding_mode
|
DecodingMode::kSoftware and DecodingMode::kHardwareWithFallback are available on this device
|
3 |
has_video_texture
|
The device supports WASM Player Video Texture mode. | 2 |
has_emss
|
WASM Player is available on the device. | 1 |
has_legacy_emss
|
WASM Player is available on the device in the legacy version; please see below for notes on compatibility. | 0 |
Note on compatibility
Presence of ElementaryMediaStreamSource
can be determined by checking both has_emss
and has_legacy_emss
flags. Only one of those two flags can be true
at a time.
Depending on the playback scenario App can be obliged to distinguish those two WASM Player versions, as has_legacy_emss
one comes with certain limitations. When has_legacy_emss
is true
:
- seeking requires extra care, please see guidelines in note,Note :
For
has_legacy_emss
, seeks should follow extra guidelines:
- seek bursts (i.e. multiple consecutive seeks occurring in close succession) should be avoided. Essentially, no new seek should start while
HTMLMediaElement.seeking
istrue
. - App shouldn't call any
ElementaryMediaTrack::AppendPacket()
s afterHTMLMediaElement::SetCurrentTime()
is called. No packets should be sent to WASM Player untilElementaryMediaTrackListener::OnTrackOpen()
event is fired after seek.
HTMLMediaElement
loop will be executed before playback position reaches duration.
Above affect only Normal Latency mode. Low Latency mode works the same on both versions.