Sending Elementary Media Packets

Elementary media stream playback with the WASM Player relies on the application sending Elementary Media Packets to the platform. This is done through ElementaryMediaTrack objects, each representing a single media track.

The sending packets operation differs based on the selected latency mode:

• In normal latency mode, the platform buffers elementary media data, guaranteeing a smooth playback as long as enough data is buffered.

  Note

  If the platform's internal media data buffer level is too low to maintain a steady playback, HTMLMediaElement pauses the playback. The media element resumes the playback automatically when enough data is buffered.

• In low latency modes, the platform renders elementary media data immediately, allowing the application to implement live streaming scenarios.

  Note

  Stream running time represented by HTMLMediaElement::GetCurrentTime() advances according to timestamps of Elementary Media Packets sent by the application.

1. Appending Packets

The application uses an ElementaryMediaTrack::AppendPacket() operation to send media data to Media Player.

• Elementary Media Packets must be appended to each individual track in decoding order (sorted by decoding timestamp).

• Appends do not need to maintain decoding order between tracks. However it is recommended they do (this does not apply in low latency mode, where the rendering speed is entirely dependent on the application and no internal buffering occurs).

• B-Frames must not be used in low latency mode.

A mechanism that sends Elementary Media Packets to the WASM Player should follow the scheme below:

1. Prepare ElementaryMediaPacket structure for the next append. The packet to append is a packet with the lowest decoding time stamp (DTS) among packets that have already not been sent to the WASM Player.

   samsung::wasm::ElementaryMediaPacket packet;
   packet.pts = /* Presentation timestamp in samsung::wasm::Seconds */;
   packet.dts = /* Decoding timestamp in samsung::wasm::Seconds */;
   packet.duration = /* Packet duration in samsung::wasm::Seconds */;
   packet.is_key_frame = /* Boolean value indicating if packet is a keyframe */;
   packet.data = /* Pointer to packet data payload */;
   packet.data_size = /* Size of the data payload in bytes */;
   packet.session_id = /* Current session ID */
   

   Note

   Audio packets are usually keyframes.

   Multithreaded applications must use session ID.

   The application can set additional parameters if the packet in question is a video frame:

   // Set either both or none; can be 0 unless resolution changed
   packet.width = /* 0 or width */;
   packet.height = /* 0 or height */;
   
   // Set either both or none; can be 0 unless resolution changed
   packet.framerate_num = /* 0 or value */;
   packet.framerate_den = /* 0 or value */;
   

   Video-only parameters come in pairs: .width and .height, .framerate_num and .framerate_den. Each pair is optional and doesn't have to be specified unless the track parameter they describe changes from this packet onwards. For more details on reconfiguring ElementaryMediaTrack, see Changing Runtime Configuration.

   Important

   Video-only packet parameters must be zeroed for audio packets.

2. Send the packet by appending it to the corresponding ElementaryMediaTrack:

   auto result = track->AppendPacket(packet);
   
   if (!result) {
     // Handle errors
   }
   

Note

AppendPacketAsync() can be used as an alternative operation to AppendPacket(). It is functionally equivalent to sync method: it validates the packet and returns a result synchronously. However, unlike AppendPacket(), AppendPacketAsync() operation can be called from the main thread.
Calls to both methods for the same track can be mixed.

2. Managing Buffered Data Amount

Note

This section does not apply in low latency modes, which do not buffer data on the platform's side.

To maintain a smooth playback, the application needs to maintain a steady flow of Elementary Media Packets to prevent an underrun of the platform's media data buffer. As a rule of thumb, buffer a couple of seconds worth of packets for each track, with the following limitations:

• For clear (non-encrypted) content playback, at most 64 MiB of data can be buffered
• For DRM-protected content playback, at most 10 MiB of data can be buffered
• For an audio track, at most 768 KiB of data can be buffered
• The application must not buffer more than 3 seconds of content in advance

Note

If a media data buffer underrun occurs, HTMLMediaElement pauses playback. Playback is resumed automatically when enough packets are buffered.

In order to keep track of the amount of data that must be buffered, use the ElementaryMediaStreamSourceListener::OnPlaybackPositionChanged() event. This event is emitted periodically to update the current playback time, and can be used to calculate amount of data to buffer:

// Controls how many packets must be buffered ahead of a current playback
// position
constexpr samsung::wasm::Seconds kBufferAhead = Seconds{3.};

class MySourceListener : public samsung::wasm::ElementaryMediaStreamSourceListener {
  using Seconds = samsung::wasm::Seconds;
  // ...
  void OnPlaybackPositionChanged(Seconds new_time) override {
    // Buffer packets up to timestamp (new_time + kBufferAhead)
  }
  // ...
};

Note

The ElementaryMediaStreamSourceListener::OnPlaybackPositionChanged() event is similar to the HTMLMediaElementListener::OnTimeUpdate() event, however the one associated with HTMLMediaElement must not be used to control a data source.

Events associated with each of these 2 components are associated with the state of their respective component. States of ElementaryMediaStreamSource (data source) and HTMLMediaPlayer (Media Player control element) are not in sync. For example, during a Seek operation the data source can be done seeking while Media Player is still in the middle of the Seek. Therefore, HTMLMediaElementListener::OnTimeUpdate() is not reliable in the context of a data source.

Threading Considerations

Usually, the application is sending Elementary Media Packets on a worker thread (see the Threading section for more). However, ElementaryMediaStreamSourceListener events are delivered on the main thread (through the main JS message loop). If triggering packet buffering involves thread synchronization, keep to a minimum the amount of inter-thread communication that involves locking the main thread.

Most of the events delivered from both ElementaryMediaStreamSource and HTMLMediaElement are scarce. The only exception are time updates, which are delivered on a regular basis, potentially with high frequency. The application must fine-tune how often such events are handled. Usually, updating the buffering mechanism once or twice per second is sufficient.

3. Stopping the Packet Sending

The application can send packets as long as ElementaryMediaTrack objects remain open and ElementaryMediaStreamSource is in the ReadyState::kOpen state. When the track closes and the OnTrackClosed event is triggered, you must stop sending packets:

class MyTrackListener : public samsung::wasm::ElementaryMediaTrackListener {
   // ...
   void OnTrackClosed(ElementaryMediaTrack::CloseReason close_reason) override {
      // Stop the application's packet sending mechanism for the associated track
   }
   // ...
};

Note

Alternatively, the application can detect when ElementaryMediaStreamSource leaves the ReadyState::kOpen state, using ElementaryMediaStreamSourceListener. OnTrackClosed events are always generated after ElementaryMediaStreamSource state change handlers are executed, so stopping packet sending mechanism there is safe. However, the source can enter multiple states from the ReadyState::kOpen state, so using a track listener is preferred.

The platform can close tracks due to both the application actions and the user interaction with the device. The application must properly handle close events and expect tracks to close at any moment. The most common occurrence that causes tracks to close is the user switching active applications (see Multitasking). Another common example would be the user performing a Seek operation.

4. Ending the Stream

Note

This section does not apply in low latency modes, as stream duration is set to infinite there.

When playback is coming to an end and all remaining Elementary Media Packets are buffered, the application must mark each track as ended. It can be done by using one of 2 possible methods: AppendEndOfTrack() or AppendEndOfTrackAsync(). AppendEndOfTrack() works synchronously: it returns a result when operation is finished (either the track was closed or an error occurred):

auto result = track->AppendEndOfTrack(session_id);

if (!result) {
   // Handle error
}

The AppendEndOfTrackAsync() method returns a result synchronously when end of track has been appended and the track starts closing (or an error occurred before that happened). All errors that occur when track is closing will be signalled through the ElementaryMediaTrackListener::OnAppendError() event. For details, see Handling Async Append Errors.

Multithreaded applications should use session ID.

Note

While each track is marked as ended separately, all the tracks should end on a timestamp close to the stream duration set with ElementaryMediaStreamSource::SetDuration().

Seeking

Note

This section does not apply in low latency modes, as they do not support the Seek operation.

The Seek operation allows the user to jump to a new playback position. It is performed either through the HTMLMediaElement interface when controls are enabled, or programmatically by calling HTMLMediaElement::SetCurrentTime().

Whenever a new current time value is set:

1. If ElementaryMediaStreamSource is ReadyState::kOpen:
   1. ElementaryMediaStreamSource state is changed to ReadyState::kOpenPending and ElementaryMediaStreamSourceListener::OnSourceOpenPending() is fired.
   2. ElementaryMediaTracks are closed and ElementaryMediaTrackListener::OnTrackClosed() with CloseReason::kTrackSeeking is fired for each track.
   3. ElementaryMediaStreamSourceListener::OnSessionIdChanged() is fired for each track.
2. ElementaryMediaStreamSourceListener::OnSeek() is fired with a new current time value.
3. If ElementaryMediaStreamSource was ReadyState::kOpen prior Seek:
   1. ElementaryMediaTracks are opened and ElementaryMediaTrackListener::OnTrackOpen() is fired for each track.
   2. ElementaryMediaStreamSource state is changed to ReadyState::kOpen and ElementaryMediaStreamSourceListener::OnSourceOpen() is fired.

Note

When multiple Seeks happen in a close succession, step 2 can be repeated multiple times, with multiple ElementaryMediaStreamSourceListener::OnSeek() events delivered between the source opening and closing.

Looping

Note

This section does not apply in low latency modes, as low latency streams have infinite duration.

If an HTMLMediaElement has the loop property set to true, the WASM Player automatically generates a Seek back to the beginning (0s) when playback reaches the end of the stream.