top

Native Player Reference Application

This article presents architecture of an application that uses NaCl Player and describes the NaCl Player reference application (Native Player) in details.

Native Player is a NaCl Player reference application which implements chosen use cases presented in the Use Cases article. The application shows the flow of media data, starting from the data download, going through the demuxing process and eventually displaying the content on the TV.

Native Player is available under the permissive MIT license. The application features extensible, modular design. Therefore it can be reused and customized to the user needs.

This article is guaranteed to be compatible with the attached reference application package. However, the newest version of the reference application can be downloaded from the Native Player GitHub page.

Sample architecture of an application using NaCl Player

NaCl Player is used by the Native Client module. All player logic is done in the NaCl module. This includes operations such as user authorization, downloading of media content, demuxing (parsing) of media content, encrypting DRM-protected content and passing demuxed elementary stream packets into NaCl Player.

Note

This section shows a proposed general architecture for an application that uses NaCl Player which is only loosely based on the Native Player reference application. However the reference application doesn’t implement some of the presented elements (such as user authorization or custom DRM system).

Details of the Native Player reference application are presented in the following sections.

The below diagram shows a structure and data flow of an application using NaCl Player:

Components

  • External Data - A place where media content is held.
    • Media Server - Provides media container data.
    • DRM Server - Provides information needed to decrypt DRM-protected content.
  • Application - A Tizen widget which is run by end-user.
    • Native Client - A NaCl module which is responsible for the application logic. NaCl Player is used by Application.
      • Communicator - Responsible for communication from and to the Javascript (JS is not included on the diagram).
      • Controller - Responsible for controlling NaCl Player.
      • Data Provider - Responsible for downloading data from External Data.
      • Data Demuxer - Responsible for demuxing media content into Elementary Stream Packets.
  • Platform - Contains the player implementation.
  • NaCl Player - Responsible for performing functions called by Application. NaCl Player is responsible for decoding packets, playing audio and rendering video on the screen.

Actions

Note

Only the most important activities are presented. NaCl applications use HTML5 for user interaction, but it is omitted to keep the diagram clear.

  • Authorization [optional] - Authorizes a user to check the user’s access to the requested content. It can be done right after the application is launched.
  • Download - Downloads chunks of the requested media content from the Media Server.
  • Demux - Demuxes downloaded data into Elementary Stream packets.
    Depending on the content two scenarios are possible in this activity:
    • Protected content - Demuxer detects protected data that needs to be decrypted. Please note that decryption may require a communication with the DRM Server.
      1. Platform DRM - the content is decrypted with one of the platform-supported DRMs. The application configures DRM information in NaCl Player and decryption is done by the platform.
        In this case Encrypted Elementary Stream Packets are passed to Controller.
      2. Custom DRM - the content is decrypted inside the NaCl Module. The application can use any protection system which it implements. In particular, this can be a custom dedicated DRM system.
        Decrypt - the application decrypts content and unencrypted Elementary Stream Packets are passed to Controller.
    • Clear content - Demuxer parses content into unencrypted Elementary Stream Packets which are passed to Controller.
  • Control - Controller receives Elementary Stream packets and appends them to NaCl Player.

After these activities a playback of the requested content is possible.

URLDataSource in Native Player

The reference application supports URLDataSource in two scenarios:

  1. Playback from the MPEG4 media container with external subtitles,
  2. Playback from the MPEG4 media container with internal subtitles.

The application only uses Controller and Communicator components as it can be seen below:

In these scenarios the application is responsible only for controlling the media player (play, pause, seek, etc.). Content downloading and demuxing is done internally by the platform.

In case of external subtitles the application can only decide to show or hide subtitles. For internal subtitles the application can choose which text track should be displayed or hide text track whatsoever.

The code snippets below present the most important parts of the reference application code. For more details please refer to the source code of the attached Native Player application.

Playback from MPEG4 media container with external/internal subtitles

The UrlPlayerController class manages a playback from the URLDataSource. It aggregates all other necessary objects:

class UrlPlayerController : public PlayerController {
// ...
 private:
// ...
  PlayerListeners listeners_;
  std::shared_ptr<Samsung::NaClPlayer::MediaDataSource> data_source_;
  std::shared_ptr<Samsung::NaClPlayer::MediaPlayer> player_;
  std::unique_ptr<Samsung::NaClPlayer::TextTrackInfo> text_track_;
  std::vector<Samsung::NaClPlayer::TextTrackInfo> text_track_list_;
  std::shared_ptr<Communication::MessageSender> message_sender_;
// ...
};

native_player/inc/player/url_player/url_player_controller.h

Setting up

The SubtitleListener class handles NaCl Player subtitle events, forwarding the received subtitle text and duration to the Javascript application component. It is Javascript component responsibility to display the subtitle. The implementation of the SubtitleListener class is provided below.

void SubtitleListener::OnShowSubtitle(TimeTicks duration, const char* text) {
  Var varText = Var(text);
  LOG("Got subtitle: %s , duration: %f", text, duration);
  if (auto message_sender = message_sender_.lock()) {
    message_sender->ShowSubtitles(duration, varText);
  }
}

native_player/inc/player/url_player/url_player_controller.h

Next necessary thing is the implementation of the MediaEventsListener interface. The OnBufferingComplete event signalizes that NaCl Player has buffered enough data to start a playback. This information is relayed to the Javascript component.

void MediaBufferingListener::OnBufferingComplete() {
  LOGM("Event: Buffering complete! Now you may play.");
  if (auto message_sender = message_sender_.lock()) {
    message_sender->BufferingCompleted();
  }
  // ...
}

native_player/inc/player/url_player/url_player_controller.h

Once the listeners are defined, a MediaPlayer object and listeners are created. Listeners are then assigned to the player object.

void UrlPlayerController::InitPlayer(const std::string& url,
                                     const std::string& subtitle,
                                     const std::string& encoding) {
  // * In case of external subtitles the subtitle argument subtitle should point
  //   to subtitles file.
  // * In case of internal subtitles the subtitle argument subtitle should be an
  //   empty string.
  // ...
  player_ = make_shared<MediaPlayer>();
  // initialize a listeners and register them in  the player
  listeners_.player_listener =
      make_shared<MediaPlayerListener>(message_sender_);
  listeners_.buffering_listener =
      make_shared<MediaBufferingListener>(message_sender_);
  listeners_.subtitle_listener =
      make_shared<SubtitleListener>(message_sender_);

  player_->SetMediaEventsListener(listeners_.player_listener);
  player_->SetBufferingListener(listeners_.buffering_listener);
  player_->SetSubtitleListener(listeners_.subtitle_listener);

native_player/src/player/url_player/url_player_controller.cc

External subtitles must be attached to the player prior to a data source. The code below checks if a URL address to the external subtitles file was provided and pass the address to the player.

// register external subtitles source if defined
if (!subtitle.empty()) {
  text_track_ = MakeUnique<TextTrackInfo>();
  int32_t ret = player_->AddExternalSubtitles(subtitle,
                                              encoding,
                                              *text_track_);
  // ...
}

native_player/src/player/url_player/url_player_controller.cc

Initialization

When all the player objects are set up, it’s time to initialize a URLDataSource object.

URLDataSource initialization code is presented below. The URLDataSource object is constructed to handle a given URL address and attached to the player as MediaDataSource.

void UrlPlayerController::InitializeUrlPlayer(
    const std::string& content_container_url) {
  // ...
  data_source_ = make_shared<URLDataSource>(content_container_url);
  player_->AttachDataSource(*data_source_);
  // ...

native_player/src/player/url_player/url_player_controller.cc

Managing subtitles

Once the data source is attached, information about internal subtitles embedded in the multimedia container file can retrieved. The UrlPlayerController::PostTextTrackInfo() method retrieves information about all available text tracks (both internal and external) and sends the list to the Javascript component for presentation in the UI.

void UrlPlayerController::PostTextTrackInfo() {
  int32_t ret = player_->GetTextTracksList(text_track_list_);
  if (ret == ErrorCodes::Success) {
    LOG("GetTextTrackInfo called successfully");
    message_sender_->SetTextTracks(text_track_list_);
  } else {
    LOG_ERROR("GetTextTrackInfo call failed, code: %d", ret);
  }
}

native_player/src/player/url_player/url_player_controller.cc

Playback

The above code completes initialization of the player in the URLDataSource scenario. After receiving the OnBufferingComplete event, the application may start the playback.

void UrlPlayerController::Play() {
  // ...
  int32_t ret = player_->Play();
  // ...
}

native_player/src/player/url_player/url_player_controller.cc

The application will receive SubtitleListener::OnShowSubtitle() events that contain internal/external subtitle texts that should be displayed. This is done by passing a subtitle event data to Javascript component, which displays subtitles.

Application & files reference

The most important parts of the URLDataSource scenario were presented above. To see a whole code used in this scenario please refer to following class in files of the reference application:

ESDataSource in Native Player

The reference application supports ESDataSource in two scenarios:

  1. a playback of the clear MPEG-DASH content with the external subtitles,
  2. a playback of the encrypted MPEG-DASH content with the PlayReady DRM protection.

The application uses all four components as it can be seen below:

In these scenarios the application is responsible for content downloading, content demuxing and controlling the media player (play, pause, seek, etc.).

In these scenarios only external subtitles may be applied.

In ESDataSource scenarios Data Provider and Data Demuxer components are in use.

Data Provider

In order to playback MPEG-DASH content, the application needs to parse a DASH manifest file and download appropriate media file or it’s chunk.

To provide above functionality the DashManifest class has been implemented. DashManifest uses the libdash library.

This class is also responsible for extracting Content Protection DRM initialization information included in the DASH manifest via the ContentProtectionVisitor class.

For more details please refer to file in the native_player/src/dash folder in the reference application folder structure

Data Demuxer

NaCl Player uses Elementary Stream Packets for a playback. In order to provide ES Packets media content need to be demuxed into a series of ES Packets.

To provide above functionality and support the modular design of the application the StreamDemuxer interface has been created.

In order to demux media content in Native Player the FFMpegDemuxer class implementing the StreamDemuxer interface has been introduced.

FFMpegDemuxer uses the ffmpeg library.

StreamDemuxer supports clear and encrypted Elementary Stream Packets.

FFMpegDemuxer extracts from the provided media content configurations for Elementary Streams and protection data (pssh_box, DRM initialization data).

FFMpeg demuxer

The StreamDemuxer::Parse() method is called whenever there is data that should be parsed.

void FFMpegDemuxer::Parse(const std::vector<uint8_t>& data) {
  // ...
    parser_thread_.message_loop().PostWork(
        callback_factory_.NewCallback(&FFMpegDemuxer::StartParsing));
 // ...
}

native_player/src/demuxer/ffmpeg_demuxer.cc

In order to keep the application responsive, actual parsing is done in the side thread. The parsing implementation that relies on the ffmpeg library is provided below.

Note

Please note that parsed packets are delivered to the application using es_data_callback_, a callback registered during the StreamDemuxer class initialization.

void FFMpegDemuxer::StartParsing(int32_t) {
  // ...
  if (!streams_initialized_) InitStreamInfo();
  AVPacket pkt;
  av_init_packet(&pkt);
  pkt.data = NULL;
  pkt.size = 0;
  while (!exited_) {
  // ...
    unique_ptr<ElementaryStreamPacket> es_pkt;
    int32_t ret = av_read_frame(format_context_, &pkt);
    if (ret < 0) {
      if (ret == AVERROR_EOF) {
        exited_ = true;
        packet_msg = kEndOfStream;
      }
    // ...
    } else {
      es_pkt = MakeESPacketFromAVPacket(&pkt);
    }
    if (es_data_callback_) es_data_callback_(packet_msg, std::move(es_pkt));
    av_free_packet(&pkt);
  }
  // ...
}

native_player/src/demuxer/ffmpeg_demuxer.cc

The following function is a support method that converts the FFMpeg elementary packet (AVPacket) into an ElementaryStreamPacket object used by the player.

unique_ptr<ElementaryStreamPacket> FFMpegDemuxer::MakeESPacketFromAVPacket(
    AVPacket* pkt) {
  auto es_packet = MakeUnique<ElementaryStreamPacket>(pkt->data, pkt->size);
  AVStream* s = format_context_->streams[pkt->stream_index];
  // setting ES packet information
  es_packet->SetPts(ToTimeTicks(pkt->pts, s->time_base) + timestamp_);
  es_packet->SetDts(ToTimeTicks(pkt->dts, s->time_base) + timestamp_);
  es_packet->SetDuration(ToTimeTicks(pkt->duration, s->time_base));
  es_packet->SetKeyFrame(pkt->flags == 1);
  AVEncInfo* enc_info = reinterpret_cast<AVEncInfo*>(
      av_packet_get_side_data(pkt, AV_PKT_DATA_ENCRYPT_INFO, NULL));
  if (!enc_info) return es_packet;
  // packet encrypted
  es_packet->SetKeyId(enc_info->kid, kKidLength);
  es_packet->SetIv(enc_info->iv, enc_info->iv_size);
  for (uint32_t i = 0; i < enc_info->subsample_count; ++i) {
    es_packet->AddSubsample(enc_info->subsamples[i].bytes_of_clear_data,
                            enc_info->subsamples[i].bytes_of_enc_data);
  }
  return es_packet;
}

native_player/src/demuxer/ffmpeg_demuxer.cc

For more details about FFMpegDemuxer implementation, including stream configuration discovery, please refer to files in the folder native_player/src/demuxer folder in the reference application folder structure.

Playback of MPEG-DASH content

Code snippets presented below are the most important parts of the code that playbacks DASH content. For more details please refer to the source code of the attached Native Player application.

Setting up

The ESDashPlayerController class manages a playback from ESDataSource. It aggregates all other necessary objects:

class EsDashPlayerController : public PlayerController {
// ...
 private:
// ...
  PlayerListeners listeners_;
  std::shared_ptr<Samsung::NaClPlayer::MediaDataSource> data_source_;
  std::shared_ptr<Samsung::NaClPlayer::MediaPlayer> player_;
  std::unique_ptr<Samsung::NaClPlayer::TextTrackInfo> text_track_;

  std::shared_ptr<Communication::MessageSender> message_sender_;

  std::unique_ptr<DashManifest> dash_parser_;
  std::array<std::shared_ptr<StreamManager>,
             static_cast<int32_t>(StreamType::MaxStreamTypes)> streams_;
  std::vector<VideoStream> video_representations_;
  std::vector<AudioStream> audio_representations_;
// ...
};

native_player/inc/player/es_dash_player/es_dash_player_controller.h

A player and listeners are created in the code presented below. Listeners are then assigned to the player object.

void EsDashPlayerController::InitPlayer(const std::string& mpd_file_path,
                                        const std::string& subtitle,
                                        const std::string& encoding) {
  // ...
  player_ = make_shared<MediaPlayer>();
  listeners_.player_listener =
      make_shared<MediaPlayerListener>(message_sender_);
  listeners_.buffering_listener =
      make_shared<MediaBufferingListener>(message_sender_,
                                          shared_from_this());

  player_->SetMediaEventsListener(listeners_.player_listener);
  player_->SetBufferingListener(listeners_.buffering_listener);
  // ...
  InitializeSubtitles(subtitle, encoding);
  InitializeDash(mpd_file_path);
}

native_player/src/player/es_dash_player/es_dash_player_controller.cc

External subtitles must be attached to the player before a data source. The code below checks if a URL address to the external subtitles file was provided and pass the address to the player.

void EsDashPlayerController::InitializeSubtitles(const std::string& subtitle,
                                                 const std::string& encoding) {
  if (subtitle.empty()) return;
  text_track_ = MakeUnique<TextTrackInfo>();
  int32_t ret = player_->AddExternalSubtitles(subtitle, encoding, *text_track_);
  listeners_.subtitle_listener =
      make_shared<SubtitleListener>(message_sender_);
  player_->SetSubtitleListener(listeners_.subtitle_listener);
}

native_player/src/player/es_dash_player/es_dash_player_controller.cc

Initialization

The code snippet below initializes the DASH parser.

Important

Please note a DrmPlayreadyContentProtection visitor object instantiation. DRM-specific data are not part of the DASH standard, so they cannot be parsed by the libdash. DASH defines a place where DRM-specific data is placed: a ContentProtection manifest section. A ContentProtectionVisitor passed to the DashManifest::ParseMPD() method that uses libdash will be notified about the contents of the ContentProtection section. The application should handle configuration of the DRM it uses. In case of the reference application, the DRM used is PlayReady.

void EsDashPlayerController::InitializeDash(
    const std::string& mpd_file_path) {
  // only PlayReady is supported in Native Player
  unique_ptr<DrmPlayReadyContentProtectionVisitor> visitor =
      MakeUnique<DrmPlayReadyContentProtectionVisitor>();
  dash_parser_ = DashManifest::ParseMPD(mpd_file_path, visitor.get());
  // ...
  data_source_ = std::make_shared<ESDataSource>();
  // ...
  video_representations_ = dash_parser_->GetVideoStreams();
  audio_representations_ = dash_parser_->GetAudioStreams();
  // ...
}

native_player/src/player/es_dash_player/es_dash_player_controller.cc

Initialize a video stream from the DASH manifest. An audio stream is analogous and is omitted in this article.

void EsDashPlayerController::InitializeVideoStream(
    Samsung::NaClPlayer::DRMType drm_type) {
  if (video_representations_.empty()) return;
  // ...
  VideoStream s = GetHighestBitrateStream(video_representations_);
  // ...
  if (s.description.content_protection) {  // DRM content detected
    auto drm_listener = make_shared<DrmPlayReadyListener>(instance_, player_);
    drm_listener->SetContentProtectionDescriptor(
        std::static_pointer_cast<DrmPlayReadyContentProtectionDescriptor>(
            s.description.content_protection));
    player_->SetDRMListener(drm_listener);
  }
  // stream manager is created to manage downloading and demuxing
  auto& stream_manager = streams_[static_cast<int32_t>(StreamType::Video)];
  stream_manager = make_shared<StreamManager>(instance_, StreamType::Video);
  bool success = stream_manager->Initialize(
      dash_parser_->GetVideoSequence(s.description.id),
      data_source_,
      std::bind(&EsDashPlayerController::OnStreamConfigured, this, _1),
      drm_type);
  // ...
}

native_player/src/player/es_dash_player/es_dash_player_controller.cc

The code snippet above uses the StreamManager object. It is used to manage elementary streams: download a proper representation, demux it into elementary stream packets and send it to the player ready for a playback.

StreamManager initialization:

bool StreamManager::Impl::Initialize(
    unique_ptr<MediaSegmentSequence> segment_sequence,
    shared_ptr<ESDataSource> es_data_source,
    std::function<void(StreamType)> stream_configured_callback,
    DRMType drm_type,
    std::shared_ptr<ElementaryStreamListener> listener) {
  // ...
  // dd a stream to ESDataSource
  if (stream_type_ == StreamType::Video) {
    auto video_stream = make_shared<VideoElementaryStream>();
    result = es_data_source->AddStream(*video_stream, listener);
    elementary_stream_ =
        std::static_pointer_cast<ElementaryStream>(video_stream);
  } else if (stream_type_ == StreamType::Audio) {
    auto audio_stream = make_shared<AudioElementaryStream>();
    result = es_data_source->AddStream(*audio_stream, listener);
    elementary_stream_ =
        std::static_pointer_cast<ElementaryStream>(audio_stream);
  }
  // ...
  // initialize stream demuxer - create and set stream configs listeners
  if (!InitParser()) {
    LOG_ERROR("Failed to initialize parser or config listeners");
    return false;
  }
 return ParseInitSegment();
}

native_player/src/player/es_dash_player/stream_manager.cc