This topic covers how to manage media playback in Tizen WASM Player, including sending elementary media packets, using the Seek operation, and looping playback.
Elementary media stream playback with the WASM Player relies on the application sending Elementary Media Packets to the platform. This is done through ElementaryMediaTrack objects, each representing a single media track.
ElementaryMediaTrack
The sending packets operation differs based on the selected latency mode:
In normal latency mode, the platform buffers elementary media data, guaranteeing a smooth playback as long as enough data is buffered.
Note If the platform's internal media data buffer level is too low to maintain a steady playback, HTMLMediaElement pauses the playback. The media element resumes the playback automatically when enough data is buffered.
If the platform's internal media data buffer level is too low to maintain a steady playback, HTMLMediaElement pauses the playback. The media element resumes the playback automatically when enough data is buffered.
HTMLMediaElement
In low latency modes, the platform renders elementary media data immediately, allowing the application to implement live streaming scenarios.
Note Stream running time represented by HTMLMediaElement::GetCurrentTime() advances according to timestamps of Elementary Media Packets sent by the application.
Stream running time represented by HTMLMediaElement::GetCurrentTime() advances according to timestamps of Elementary Media Packets sent by the application.
HTMLMediaElement::GetCurrentTime()
The application uses an ElementaryMediaTrack::AppendPacket() operation to send media data to Media Player.
ElementaryMediaTrack::AppendPacket()
A mechanism that sends Elementary Media Packets to the WASM Player should follow the scheme below:
Prepare ElementaryMediaPacket structure for the next append. The packet to append is a packet with the lowest decoding time stamp (DTS) among packets that have already not been sent to the WASM Player.
ElementaryMediaPacket
samsung::wasm::ElementaryMediaPacket packet; packet.pts = /* Presentation timestamp in samsung::wasm::Seconds */; packet.dts = /* Decoding timestamp in samsung::wasm::Seconds */; packet.duration = /* Packet duration in samsung::wasm::Seconds */; packet.is_key_frame = /* Boolean value indicating if packet is a keyframe */; packet.data = /* Pointer to packet data payload */; packet.data_size = /* Size of the data payload in bytes */; packet.session_id = /* Current session ID */
Note Audio packets are usually keyframes.
Audio packets are usually keyframes.
Multithreaded applications must use session ID.
The application can set additional parameters if the packet in question is a video frame:
// Set either both or none; can be 0 unless resolution changed packet.width = /* 0 or width */; packet.height = /* 0 or height */; // Set either both or none; can be 0 unless resolution changed packet.framerate_num = /* 0 or value */; packet.framerate_den = /* 0 or value */;
Video-only parameters come in pairs: .width and .height, .framerate_num and .framerate_den. Each pair is optional and doesn't have to be specified unless the track parameter they describe changes from this packet onwards. For more details on reconfiguring ElementaryMediaTrack, see Changing Runtime Configuration.
.width
.height
.framerate_num
.framerate_den
Important Video-only packet parameters must be zeroed for audio packets.
Video-only packet parameters must be zeroed for audio packets.
Send the packet by appending it to the corresponding ElementaryMediaTrack:
auto result = track->AppendPacket(packet); if (!result) { // Handle errors }
Note AppendPacketAsync() can be used as an alternative operation to AppendPacket(). It is functionally equivalent to sync method: it validates the packet and returns a result synchronously. However, unlike AppendPacket(), AppendPacketAsync() operation can be called from the main thread. Calls to both methods for the same track can be mixed.
AppendPacketAsync() can be used as an alternative operation to AppendPacket(). It is functionally equivalent to sync method: it validates the packet and returns a result synchronously. However, unlike AppendPacket(), AppendPacketAsync() operation can be called from the main thread. Calls to both methods for the same track can be mixed.
AppendPacketAsync()
AppendPacket()
Note This section does not apply in low latency modes, which do not buffer data on the platform's side.
This section does not apply in low latency modes, which do not buffer data on the platform's side.
To maintain a smooth playback, the application needs to maintain a steady flow of Elementary Media Packets to prevent an underrun of the platform's media data buffer. As a rule of thumb, buffer a couple of seconds worth of packets for each track, with the following limitations:
Note If a media data buffer underrun occurs, HTMLMediaElement pauses playback. Playback is resumed automatically when enough packets are buffered.
If a media data buffer underrun occurs, HTMLMediaElement pauses playback. Playback is resumed automatically when enough packets are buffered.
In order to keep track of the amount of data that must be buffered, use the ElementaryMediaStreamSourceListener::OnPlaybackPositionChanged() event. This event is emitted periodically to update the current playback time, and can be used to calculate amount of data to buffer:
ElementaryMediaStreamSourceListener::OnPlaybackPositionChanged()
// Controls how many packets must be buffered ahead of a current playback // position constexpr samsung::wasm::Seconds kBufferAhead = Seconds{3.}; class MySourceListener : public samsung::wasm::ElementaryMediaStreamSourceListener { using Seconds = samsung::wasm::Seconds; // ... void OnPlaybackPositionChanged(Seconds new_time) override { // Buffer packets up to timestamp (new_time + kBufferAhead) } // ... };
Note The ElementaryMediaStreamSourceListener::OnPlaybackPositionChanged() event is similar to the HTMLMediaElementListener::OnTimeUpdate() event, however the one associated with HTMLMediaElement must not be used to control a data source. Events associated with each of these 2 components are associated with the state of their respective component. States of ElementaryMediaStreamSource (data source) and HTMLMediaPlayer (Media Player control element) are not in sync. For example, during a Seek operation the data source can be done seeking while Media Player is still in the middle of the Seek. Therefore, HTMLMediaElementListener::OnTimeUpdate() is not reliable in the context of a data source.
The ElementaryMediaStreamSourceListener::OnPlaybackPositionChanged() event is similar to the HTMLMediaElementListener::OnTimeUpdate() event, however the one associated with HTMLMediaElement must not be used to control a data source.
HTMLMediaElementListener::OnTimeUpdate()
Events associated with each of these 2 components are associated with the state of their respective component. States of ElementaryMediaStreamSource (data source) and HTMLMediaPlayer (Media Player control element) are not in sync. For example, during a Seek operation the data source can be done seeking while Media Player is still in the middle of the Seek. Therefore, HTMLMediaElementListener::OnTimeUpdate() is not reliable in the context of a data source.
ElementaryMediaStreamSource
HTMLMediaPlayer
Threading Considerations
Usually, the application is sending Elementary Media Packets on a worker thread (see the Threading section for more). However, ElementaryMediaStreamSourceListener events are delivered on the main thread (through the main JS message loop). If triggering packet buffering involves thread synchronization, keep to a minimum the amount of inter-thread communication that involves locking the main thread.
ElementaryMediaStreamSourceListener
Most of the events delivered from both ElementaryMediaStreamSource and HTMLMediaElement are scarce. The only exception are time updates, which are delivered on a regular basis, potentially with high frequency. The application must fine-tune how often such events are handled. Usually, updating the buffering mechanism once or twice per second is sufficient.
The application can send packets as long as ElementaryMediaTrack objects remain open and ElementaryMediaStreamSource is in the ReadyState::kOpen state. When the track closes and the OnTrackClosed event is triggered, you must stop sending packets:
ReadyState::kOpen
OnTrackClosed
class MyTrackListener : public samsung::wasm::ElementaryMediaTrackListener { // ... void OnTrackClosed(ElementaryMediaTrack::CloseReason close_reason) override { // Stop the application's packet sending mechanism for the associated track } // ... };
Note Alternatively, the application can detect when ElementaryMediaStreamSource leaves the ReadyState::kOpen state, using ElementaryMediaStreamSourceListener. OnTrackClosed events are always generated after ElementaryMediaStreamSource state change handlers are executed, so stopping packet sending mechanism there is safe. However, the source can enter multiple states from the ReadyState::kOpen state, so using a track listener is preferred.
Alternatively, the application can detect when ElementaryMediaStreamSource leaves the ReadyState::kOpen state, using ElementaryMediaStreamSourceListener. OnTrackClosed events are always generated after ElementaryMediaStreamSource state change handlers are executed, so stopping packet sending mechanism there is safe. However, the source can enter multiple states from the ReadyState::kOpen state, so using a track listener is preferred.
The platform can close tracks due to both the application actions and the user interaction with the device. The application must properly handle close events and expect tracks to close at any moment. The most common occurrence that causes tracks to close is the user switching active applications (see Multitasking). Another common example would be the user performing a Seek operation.
Note This section does not apply in low latency modes, as stream duration is set to infinite there.
This section does not apply in low latency modes, as stream duration is set to infinite there.
When playback is coming to an end and all remaining Elementary Media Packets are buffered, the application must mark each track as ended. It can be done by using one of 2 possible methods: AppendEndOfTrack() or AppendEndOfTrackAsync(). AppendEndOfTrack() works synchronously: it returns a result when operation is finished (either the track was closed or an error occurred):
AppendEndOfTrack()
AppendEndOfTrackAsync()
auto result = track->AppendEndOfTrack(session_id); if (!result) { // Handle error }
The AppendEndOfTrackAsync() method returns a result synchronously when end of track has been appended and the track starts closing (or an error occurred before that happened). All errors that occur when track is closing will be signalled through the ElementaryMediaTrackListener::OnAppendError() event. For details, see Handling Async Append Errors.
ElementaryMediaTrackListener::OnAppendError()
Multithreaded applications should use session ID.
Note While each track is marked as ended separately, all the tracks should end on a timestamp close to the stream duration set with ElementaryMediaStreamSource::SetDuration().
While each track is marked as ended separately, all the tracks should end on a timestamp close to the stream duration set with ElementaryMediaStreamSource::SetDuration().
ElementaryMediaStreamSource::SetDuration()
Note This section does not apply in low latency modes, as they do not support the Seek operation.
This section does not apply in low latency modes, as they do not support the Seek operation.
The Seek operation allows the user to jump to a new playback position. It is performed either through the HTMLMediaElement interface when controls are enabled, or programmatically by calling HTMLMediaElement::SetCurrentTime().
controls
HTMLMediaElement::SetCurrentTime()
Whenever a new current time value is set:
ReadyState::kOpenPending
ElementaryMediaStreamSourceListener::OnSourceOpenPending()
ElementaryMediaTrackListener::OnTrackClosed()
CloseReason::kTrackSeeking
ElementaryMediaStreamSourceListener::OnSessionIdChanged()
ElementaryMediaStreamSourceListener::OnSeek()
ElementaryMediaTrackListener::OnTrackOpen()
ElementaryMediaStreamSourceListener::OnSourceOpen()
Note When multiple Seeks happen in a close succession, step 2 can be repeated multiple times, with multiple ElementaryMediaStreamSourceListener::OnSeek() events delivered between the source opening and closing.
When multiple Seeks happen in a close succession, step 2 can be repeated multiple times, with multiple ElementaryMediaStreamSourceListener::OnSeek() events delivered between the source opening and closing.
Note This section does not apply in low latency modes, as low latency streams have infinite duration.
This section does not apply in low latency modes, as low latency streams have infinite duration.
If an HTMLMediaElement has the loop property set to true, the WASM Player automatically generates a Seek back to the beginning (0s) when playback reaches the end of the stream.
loop