This section provides an extension to Using Tizen WASM Player with RenderingMode::kVideoTexture of ElementaryMediaStreamSource. It presents how to extend an existing WASM Player application with Video Decoder functionalities using the Tizen WASM Video Decoder Sample.
Tizen WASM Player API Documentation
Tizen WASM Video Decoder Sample
To enable the WASM Video Decoder in an existing WASM Player application, you must:
The WASM Video Decoder allows the application to fill a requested GL texture with decoded frame, instead of rendering it on a HTMLMediaElement.
HTMLMediaElement
To change the ElementaryMediaStreamSource rendering mode from Media Element to Video Texture, RenderingMode::kMediaElement must be replaced by RenderingMode::kVideoTexture:
ElementaryMediaStreamSource
RenderingMode::kMediaElement
RenderingMode::kVideoTexture
using LatencyMode = samsung::wasm::ElementaryMediaStreamSource::LatencyMode; using RenderingMode = samsung::wasm::ElementaryMediaStreamSource::RenderingMode; auto elementary_media_stream_source = std::make_unique<samsung::wasm::ElementaryMediaStreamSource>(LatencyMode::kNormal, RenderingMode::kVideoTexture);
This section covers the configuration of GL context using the Samsung Emscripten SDK. You must first make the canvas accessible, and then prepare either SDL or EGL for GL initialization.
GL context in WASM is associated with a <canvas> HTML element. To make it possible for WASM to use it:
<canvas>
<canvas id="canvas" width=1600 height=900></canvas>
Module
Module = { ... canvas: (function() { return document.getElementById('canvas'); })(), }
Now you can access the <canvas> HTML element from the WASM module.
canvas
int width; int height; emscripten_get_canvas_element_size("#canvas", &width, &height);
To initialize GL using SDL:
SDL_Window
window_ = SDL_CreateWindow("VideoTexture", SDL_WINDOWPOS_CENTERED, SDL_WINDOWPOS_CENTERED, width, height, SDL_WINDOW_OPENGL | SDL_WINDOW_SHOWN);
SDL_GLContext
gl_context_ = SDL_GL_CreateContext(window_); SDL_GL_MakeCurrent(window_, gl_context_);
SDL_Init(SDL_INIT_VIDEO); SDL_GL_SetAttribute(SDL_GL_CONTEXT_MAJOR_VERSION, 2); // Indicates GLES version to use SDL_GL_SetAttribute(SDL_GL_CONTEXT_MINOR_VERSION, 0); SDL_GL_SetAttribute(SDL_GL_DOUBLEBUFFER, 1); SDL_GL_SetAttribute(SDL_GL_DEPTH_SIZE, 24); // Indicates GL color depth SDL_GL_SetAttribute(SDL_GL_MULTISAMPLEBUFFERS, 1); SDL_GL_SetAttribute(SDL_GL_MULTISAMPLESAMPLES, 4); // Turns on multisampling
As an alternative for SDL initialization, GL can also be initialized using the EGL wrapper:
const EGLint attrib_list[] = { EGL_RED_SIZE, 8, EGL_GREEN_SIZE, 8, EGL_BLUE_SIZE, 8, EGL_ALPHA_SIZE, EGL_DONT_CARE, EGL_DEPTH_SIZE, EGL_DONT_CARE, EGL_STENCIL_SIZE, EGL_DONT_CARE, EGL_SAMPLE_BUFFERS, 0, EGL_NONE }; const EGLint context_attribs[] = { EGL_CONTEXT_CLIENT_VERSION, 2, EGL_NONE }; EGLint num_configs; EGLint major_version; EGLint minor_version; EGLConfig config; EGLDisplay display = eglGetDisplay(EGL_DEFAULT_DISPLAY); eglInitialize(display, &major_version, &minor_version); eglGetConfigs(display, NULL, 0, &num_configs), EGL_TRUE) eglChooseConfig(display, attrib_list, &config, 1, &num_configs)
EGLSurface
EGLSurface surface = eglCreateWindowSurface(display, config, NULL, NULL);
EGLContext
EGLContext context = eglCreateContext(display, config, EGL_NO_CONTEXT, context_attribs); eglMakeCurrent(display, surface, surface, context);
To initialize GL:
Create a texture.
glGenTextures(1, &texture_);
The texture is filled with video frames decoded by WASM Player.
Set a viewport to allow GL to automatically scale rendering to the provided viewport.
glViewport(0, 0, width, height);
Compile shaders and link them to the program:
const char kVertexShader[] = "varying vec2 v_texCoord; \n" "attribute vec4 a_position; \n" "attribute vec2 a_texCoord; \n" "uniform vec2 v_scale; \n" "void main() \n" "{ \n" " v_texCoord = v_scale * a_texCoord; \n" " gl_Position = a_position; \n" "}";
const char kFragmentShaderExternal[] = "#extension GL_OES_EGL_image_external : require \n" "precision mediump float; \n" "varying vec2 v_texCoord; \n" "uniform samplerExternalOES s_texture; \n" "void main() \n" "{ \n" " gl_FragColor = texture2D(s_texture, v_texCoord); \n" "}
Note #extension GL_OES_EGL_image_external : require and the sampler uniform samplerExternalOES are required for Video Decoder functionality.
#extension GL_OES_EGL_image_external : require and the sampler uniform samplerExternalOES are required for Video Decoder functionality.
#extension GL_OES_EGL_image_external : require
uniform samplerExternalOES
void CreateShader(GLuint program, GLenum type, const char* source, int size) { GLuint shader = glCreateShader(type); glShaderSource(shader, 1, &source, &size); glCompileShader(shader); glAttachShader(program, shader); glDeleteShader(shader); }
program_ = glCreateProgram(); CreateShader(program_, GL_VERTEX_SHADER, kVertexShader, strlen(kVertexShader)); CreateShader(program_, GL_FRAGMENT_SHADER, kFragmentShaderExternal, strlen(kFragmentShaderExternal)); glLinkProgram(program_); glUseProgram(program_);
To use GLES 3 (WebGL 2) for WASM Video Decoder functionality (optional):
#version 300 es
to
#extension GL_OES_EGL_image_external_essl3 : require
texture
texture2D
SDL_GL_SetAttribute(SDL_GL_CONTEXT_MAJOR_VERSION, 3);
Note Note that the EGL provided by the Emscripten SDK does not support setting the GL major version, so it is not possible to use GLES 3.0 using EGL wrapper.
Note that the EGL provided by the Emscripten SDK does not support setting the GL major version, so it is not possible to use GLES 3.0 using EGL wrapper.
Register the GL Context for WASM Video Decoder by informing ElementaryMediaTrack about current graphics context:
ElementaryMediaTrack
video_track_.RegisterCurrentGraphicsContext();
To implement the WASM Video Decoder rendering loop:
Request the video texture fill. The decoding loop that fills the texture with a decoded video frame can be started after OnTrackOpen event is received or when the HTMLMediaElement::Play callback is called. When the texture is filled with the video frame, draw it:
OnTrackOpen
HTMLMediaElement::Play
void VideoDecoderTrackDataPump::RequestNewVideoTexture() { video_track_.FillTextureWithNextFrame( texture_, [this](samsung::wasm::OperationResult result) { if (result != samsung::wasm::OperationResult::kSuccess) { std::cout << "Filling texture with next frame failed" << std::endl; return; } Draw(); }); }
Draw the texture. Like in any C++ application, you need to provide a rendering loop. In the WASM Video Decoder application, the loop is created with the cycle illustrated in the following image:
Figure 1. WASM Video Decoder Rendering Loop
glActiveTexture(GL_TEXTURE0); glBindTexture(GL_TEXTURE_EXTERNAL_OES, texture_);
Note The texture target used for Video Decoder functionality must always be set to GL_TEXTURE_EXTERNAL_OES.
The texture target used for Video Decoder functionality must always be set to GL_TEXTURE_EXTERNAL_OES.
GL_TEXTURE_EXTERNAL_OES
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4)
emscripten_request_animation_frame(&CAPIOnDrawTextureCompleted, this);
emscripten_request_animation_frame
int CAPIOnDrawTextureCompleted(double /* time */, void* thiz) { if (thiz) static_cast<VideoDecoderTrackDataPump*>(thiz)->OnDrawCompleted(); return 0; }
Recycle the video picture It is important to always recycle the video picture after it has been drawn. To do so, call the RecycleTexture method in the emscripten_request_animation_frame callback:
RecycleTexture
void VideoDecoderTrackDataPump::OnDrawCompleted() { video_track_.RecycleTexture(texture_); RequestNewVideoTexture(); }
When rendering must be stopped, end the rendering loop. To properly end a rendering loop, the application needs to handle ElementaryMediaTrack::FillTextureWithNextFrame errors, such as OperationResult::kAlreadyDestroyed when a track was stopped before calling this method or OperationResult::kAborted when a track was stopped after calling this method.
ElementaryMediaTrack::FillTextureWithNextFrame
OperationResult::kAlreadyDestroyed
OperationResult::kAborted
Before invalidating the pointer that has already been provided to the emscripten_request_animation_frame callback, abort that callback withemscripten_cancel_animation_frame, by providing the callback ID returned by the emscripten_request_animation_frame method.
emscripten_cancel_animation_frame