Managing Playback
This topic covers how to manage media playback in Tizen WASM Player, including sending elementary media packets, using the Seek operation, and looping playback.
Sending Elementary Media Packets
Elementary media stream playback with the WASM Player relies on the application sending Elementary Media Packets to the platform. This is done through ElementaryMediaTrack
objects, each representing a single media track.
The sending packets operation differs based on the selected latency mode:
-
In normal latency mode, the platform buffers elementary media data, guaranteeing a smooth playback as long as enough data is buffered.
NoteIf the platform's internal media data buffer level is too low to maintain a steady playback,HTMLMediaElement
pauses the playback. The media element resumes the playback automatically when enough data is buffered. -
In low latency modes, the platform renders elementary media data immediately, allowing the application to implement live streaming scenarios.
NoteStream running time represented byHTMLMediaElement::GetCurrentTime()
advances according to timestamps of Elementary Media Packets sent by the application.
1. Appending Packets
The application uses an ElementaryMediaTrack::AppendPacket()
operation to send media data to Media Player.
- Elementary Media Packets must be appended to each individual track in decoding order (sorted by decoding timestamp).
- Appends do not need to maintain decoding order between tracks. However it is recommended they do (this does not apply in low latency mode, where the rendering speed is entirely dependent on the application and no internal buffering occurs).
- B-Frames must not be used in low latency mode.
A mechanism that sends Elementary Media Packets to the WASM Player should follow the scheme below:
-
Prepare
ElementaryMediaPacket
structure for the next append. The packet to append is a packet with the lowest decoding time stamp (DTS) among packets that have already not been sent to the WASM Player.samsung::wasm::ElementaryMediaPacket packet; packet.pts = /* Presentation timestamp in samsung::wasm::Seconds */; packet.dts = /* Decoding timestamp in samsung::wasm::Seconds */; packet.duration = /* Packet duration in samsung::wasm::Seconds */; packet.is_key_frame = /* Boolean value indicating if packet is a keyframe */; packet.data = /* Pointer to packet data payload */; packet.data_size = /* Size of the data payload in bytes */; packet.session_id = /* Current session ID */
NoteAudio packets are usually keyframes.Multithreaded applications must use session ID.
The application can set additional parameters if the packet in question is a video frame:// Set either both or none; can be 0 unless resolution changed packet.width = /* 0 or width */; packet.height = /* 0 or height */; // Set either both or none; can be 0 unless resolution changed packet.framerate_num = /* 0 or value */; packet.framerate_den = /* 0 or value */;
Video-only parameters come in pairs:
.width
and.height
,.framerate_num
and.framerate_den
. Each pair is optional and doesn't have to be specified unless the track parameter they describe changes from this packet onwards. For more details on reconfiguringElementaryMediaTrack
, see Changing Runtime Configuration.ImportantVideo-only packet parameters must be zeroed for audio packets. -
Send the packet by appending it to the corresponding
ElementaryMediaTrack
:auto result = track->AppendPacket(packet); if (!result) { // Handle errors }
NoteAppendPacketAsync()
can be used as an alternative operation toAppendPacket()
. It is functionally equivalent to sync method: it validates the packet and returns a result synchronously. However, unlikeAppendPacket()
,AppendPacketAsync()
operation can be called from the main thread.
Calls to both methods for the same track can be mixed.
2. Managing Buffered Data Amount
To maintain a smooth playback, the application needs to maintain a steady flow of Elementary Media Packets to prevent an underrun of the platform's media data buffer. As a rule of thumb, buffer a couple of seconds worth of packets for each track, with the following limitations:
- For clear (non-encrypted) content playback, at most 64 MiB of data can be buffered
- For DRM-protected content playback, at most 10 MiB of data can be buffered
- For an audio track, at most 768 KiB of data can be buffered
- The application must not buffer more than 3 seconds of content in advance
HTMLMediaElement
pauses playback. Playback is resumed automatically when enough packets are buffered.In order to keep track of the amount of data that must be buffered, use the ElementaryMediaStreamSourceListener::OnPlaybackPositionChanged()
event. This event is emitted periodically to update the current playback time, and can be used to calculate amount of data to buffer:
// Controls how many packets must be buffered ahead of a current playback
// position
constexpr samsung::wasm::Seconds kBufferAhead = Seconds{3.};
class MySourceListener : public samsung::wasm::ElementaryMediaStreamSourceListener {
using Seconds = samsung::wasm::Seconds;
// ...
void OnPlaybackPositionChanged(Seconds new_time) override {
// Buffer packets up to timestamp (new_time + kBufferAhead)
}
// ...
};
ElementaryMediaStreamSourceListener::OnPlaybackPositionChanged()
event is similar to the HTMLMediaElementListener::OnTimeUpdate()
event, however the one associated with HTMLMediaElement
must not be used to control a data source.Events associated with each of these 2 components are associated with the state of their respective component. States of
ElementaryMediaStreamSource
(data source) and HTMLMediaPlayer
(Media Player control element) are not in sync. For example, during a Seek operation the data source can be done seeking while Media Player is still in the middle of the Seek. Therefore, HTMLMediaElementListener::OnTimeUpdate()
is not reliable in the context of a data source.Threading Considerations
Usually, the application is sending Elementary Media Packets on a worker thread (see the Threading section for more). However, ElementaryMediaStreamSourceListener
events are delivered on the main thread (through the main JS message loop). If triggering packet buffering involves thread synchronization, keep to a minimum the amount of inter-thread communication that involves locking the main thread.
Most of the events delivered from both ElementaryMediaStreamSource
and HTMLMediaElement
are scarce. The only exception are time updates, which are delivered on a regular basis, potentially with high frequency. The application must fine-tune how often such events are handled. Usually, updating the buffering mechanism once or twice per second is sufficient.
3. Stopping the Packet Sending
The application can send packets as long as ElementaryMediaTrack
objects remain open and ElementaryMediaStreamSource
is in the ReadyState::kOpen
state. When the track closes and the OnTrackClosed
event is triggered, you must stop sending packets:
class MyTrackListener : public samsung::wasm::ElementaryMediaTrackListener {
// ...
void OnTrackClosed(ElementaryMediaTrack::CloseReason close_reason) override {
// Stop the application's packet sending mechanism for the associated track
}
// ...
};
ElementaryMediaStreamSource
leaves the ReadyState::kOpen
state, using ElementaryMediaStreamSourceListener
. OnTrackClosed
events are always generated after ElementaryMediaStreamSource
state change handlers are executed, so stopping packet sending mechanism there is safe. However, the source can enter multiple states from the ReadyState::kOpen
state, so using a track listener is preferred.The platform can close tracks due to both the application actions and the user interaction with the device. The application must properly handle close events and expect tracks to close at any moment. The most common occurrence that causes tracks to close is the user switching active applications (see Multitasking). Another common example would be the user performing a Seek operation.
4. Ending the Stream
When playback is coming to an end and all remaining Elementary Media Packets are buffered, the application must mark each track as ended. It can be done by using one of 2 possible methods: AppendEndOfTrack()
or AppendEndOfTrackAsync()
. AppendEndOfTrack()
works synchronously: it returns a result when operation is finished (either the track was closed or an error occurred):
auto result = track->AppendEndOfTrack(session_id);
if (!result) {
// Handle error
}
The AppendEndOfTrackAsync()
method returns a result synchronously when end of track has been appended and the track starts closing (or an error occurred before that happened). All errors that occur when track is closing will be signalled through the ElementaryMediaTrackListener::OnAppendError()
event. For details, see Handling Async Append Errors.
Multithreaded applications should use session ID.
ElementaryMediaStreamSource::SetDuration()
.Seeking
The Seek operation allows the user to jump to a new playback position. It is performed either through the HTMLMediaElement
interface when controls
are enabled, or programmatically by calling HTMLMediaElement::SetCurrentTime()
.
Whenever a new current time value is set:
-
If
ElementaryMediaStreamSource
isReadyState::kOpen
:ElementaryMediaStreamSource
state is changed toReadyState::kOpenPending
andElementaryMediaStreamSourceListener::OnSourceOpenPending()
is fired.ElementaryMediaTrack
s are closed andElementaryMediaTrackListener::OnTrackClosed()
withCloseReason::kTrackSeeking
is fired for each track.ElementaryMediaStreamSourceListener::OnSessionIdChanged()
is fired for each track.ElementaryMediaStreamSourceListener::OnSeek()
is fired with a new current time value.
-
If
ElementaryMediaStreamSource
wasReadyState::kOpen
prior Seek:ElementaryMediaTrack
s are opened andElementaryMediaTrackListener::OnTrackOpen()
is fired for each track.ElementaryMediaStreamSource
state is changed toReadyState::kOpen
andElementaryMediaStreamSourceListener::OnSourceOpen()
is fired.
ElementaryMediaStreamSourceListener::OnSeek()
events delivered between the source opening and closing.Looping
If an HTMLMediaElement
has the loop
property set to true, the WASM Player automatically generates a Seek back to the beginning (0s) when playback reaches the end of the stream.