Video Decoder Usage Guide
This section provides an extension to WASM Player Usage Guide article with RenderingMode::kVideoTexture of ElementaryMediaStreamSource. It presents how to extend existing WASM Player application with Video Decoder functionalities using Tizen WASM Video Decoder Sample.
Related Info
Overview
The aim of this article is to present how to modify existing WASM Player application to use RenderingMode::kVideoTexture
functionality of ElementaryMediaStreamSource
.
This mode allows the application to fill requested GL texture with decoded frame, instead of rendering it on a HTMLMediaElement
.
WASM Player -> WASM Video Decoder steps
Setting Video Texture rendering mode for WASM Player
To change ElementaryMediaStreamSource
rendering mode from Media Element to Video Texture RenderingMode::kMediaElement
should be replaced by RenderingMode::kVideoTexture
:
using LatencyMode = samsung::wasm::ElementaryMediaStreamSource::LatencyMode;
using RenderingMode = samsung::wasm::ElementaryMediaStreamSource::RenderingMode;
auto elementary_media_stream_source = std::make_unique<samsung::wasm::ElementaryMediaStreamSource>(LatencyMode::kNormal, RenderingMode::kVideoTexture);
GL context in Emscripten
Make canvas accessible from Emscripten
GL context in WASM is associated with a canvas HTML element. To make it possible for WASM to use it, the following steps need to be applied:
-
Create a
canvas
element in the application's HTML file that runs WASM module:<canvas id="canvas" width=1600 height=900></canvas>
-
Extend Emscripten
Module
object with information about the created canvas element:Module = { ... canvas: (function() { return document.getElementById('canvas'); })(), }
Now you can access
canvas
HTML element from the WASM module. -
Get
canvas
dimensions from C++ code:int width; int height; emscripten_get_canvas_element_size("#canvas", &width, &height);
These variables will be used later for context initialization.
Using SDL for GL Initialization
To initialize GL using SDL, the following steps need to be performed:
-
Create
SDL_Window
with the desired parameters:window_ = SDL_CreateWindow("VideoTexture", SDL_WINDOWPOS_CENTERED, SDL_WINDOWPOS_CENTERED, width, height, SDL_WINDOW_OPENGL | SDL_WINDOW_SHOWN);
-
Get a
SDL_GLContext
context from the window and make it the current context:gl_context_ = SDL_GL_CreateContext(window_); SDL_GL_MakeCurrent(window_, gl_context_);
-
Initialize SDL. Sample configuration:
SDL_Init(SDL_INIT_VIDEO); SDL_GL_SetAttribute(SDL_GL_CONTEXT_MAJOR_VERSION, 2); // Indicates GLES version to use SDL_GL_SetAttribute(SDL_GL_CONTEXT_MINOR_VERSION, 0); SDL_GL_SetAttribute(SDL_GL_DOUBLEBUFFER, 1); SDL_GL_SetAttribute(SDL_GL_DEPTH_SIZE, 24); // Indicates GL color depth SDL_GL_SetAttribute(SDL_GL_MULTISAMPLEBUFFERS, 1); SDL_GL_SetAttribute(SDL_GL_MULTISAMPLESAMPLES, 4); // Turns on multisampling
Using EGL for GL Initialization
As an alternative for SDL initialization, initialization using EGL wrapper can be performed:
-
Initialize EGL config. Sample configuration:
const EGLint attrib_list[] = { EGL_RED_SIZE, 8, EGL_GREEN_SIZE, 8, EGL_BLUE_SIZE, 8, EGL_ALPHA_SIZE, EGL_DONT_CARE, EGL_DEPTH_SIZE, EGL_DONT_CARE, EGL_STENCIL_SIZE, EGL_DONT_CARE, EGL_SAMPLE_BUFFERS, 0, EGL_NONE }; const EGLint context_attribs[] = { EGL_CONTEXT_CLIENT_VERSION, 2, EGL_NONE }; EGLint num_configs; EGLint major_version; EGLint minor_version; EGLConfig config; EGLDisplay display = eglGetDisplay(EGL_DEFAULT_DISPLAY); eglInitialize(display, &major_version, &minor_version); eglGetConfigs(display, NULL, 0, &num_configs), EGL_TRUE) eglChooseConfig(display, attrib_list, &config, 1, &num_configs)
-
Create
EGLSurface
:EGLSurface surface = eglCreateWindowSurface(display, config, NULL, NULL);
-
Get
EGLContext
from the window surface and make it the current context:EGLContext context = eglCreateContext(display, config, EGL_NO_CONTEXT, context_attribs); eglMakeCurrent(display, surface, surface, context);
GL Initialization
Create a texture
This texture will be filled with video frames decoded by WASM Player
glGenTextures(1, &texture_);
Set viewport
Setting viewport is required to allow GL to automatically scale rendering to provided viewport
glViewport(0, 0, width, height);
Compiling shaders and linking program
-
Define a vertex shader. Sample shader:
const char kVertexShader[] = "varying vec2 v_texCoord; \n" "attribute vec4 a_position; \n" "attribute vec2 a_texCoord; \n" "uniform vec2 v_scale; \n" "void main() \n" "{ \n" " v_texCoord = v_scale * a_texCoord; \n" " gl_Position = a_position; \n" "}";
-
Define a fragment shader. Sample shader:
const char kFragmentShaderExternal[] = "#extension GL_OES_EGL_image_external : require \n" "precision mediump float; \n" "varying vec2 v_texCoord; \n" "uniform samplerExternalOES s_texture; \n" "void main() \n" "{ \n" " gl_FragColor = texture2D(s_texture, v_texCoord); \n" "}
Note :Please note
#extension GL_OES_EGL_image_external : require
and sampleruniform samplerExternalOES
are required for Video Decoder functionality.
-
Create compile shader function:
void CreateShader(GLuint program, GLenum type, const char* source, int size) { GLuint shader = glCreateShader(type); glShaderSource(shader, 1, &source, &size); glCompileShader(shader); glAttachShader(program, shader); glDeleteShader(shader); }
-
Create program, compile shaders and link them into the program:
program_ = glCreateProgram(); CreateShader(program_, GL_VERTEX_SHADER, kVertexShader, strlen(kVertexShader)); CreateShader(program_, GL_FRAGMENT_SHADER, kFragmentShaderExternal, strlen(kFragmentShaderExternal)); glLinkProgram(program_); glUseProgram(program_);
GLES 3 (WebGL 2)
It is possible to use GLES 3 for WASM Video Decoder functionality.
To do so:
-
Add version information at the beginning of both vertex and fragment shaders:
#version 300 es
-
Change:
#extension GL_OES_EGL_image_external : require
to
#extension GL_OES_EGL_image_external_essl3 : require
in the fragment shader.
-
Use
texture
keyword instead oftexture2D
in the fragment shader definition. -
Set GL major version to 3:
SDL_GL_SetAttribute(SDL_GL_CONTEXT_MAJOR_VERSION, 3);
Note :Please note that EGL provided by emscripten does not support setting GL major version, so it is not possible to use GLES 3.0 using EGL wrapper.
Registering GL context for WASM Video Decoder
Inform ElementaryMediaTrack
about current graphics context:
video_track_.RegisterCurrentGraphicsContext();
Video Decoder rendering loop
Requesting video texture fill
The decoding loop that fills the texture with a decoded video frame can be started after OnTrackOpen
event is received or when HTMLMediaElement::Play
callback is called.
When the texture is filled with the video frame, drawing should be performed:
void VideoDecoderTrackDataPump::RequestNewVideoTexture() {
video_track_.FillTextureWithNextFrame(
texture_, [this](samsung::wasm::OperationResult result) {
if (result != samsung::wasm::OperationResult::kSuccess) {
std::cout << "Filling texture with next frame failed" << std::endl;
return;
}
Draw();
});
}
Drawing
Like in any C++ application, we need to provide a rendering loop.
In WASM Video Decoder application, we achieve it by providing the following cycle:
ElementaryMediaTrack::FillTextureWithNextFrame
-async-> GL rendering operations -sync-> emscripten_request_animation_frame
-async-> CAPIOnDrawTextureCompleted
-sync-> ElementaryMediaTrack::RecycleTexture
-sync-> ElementaryMediaTrack::FillTextureWithNextFrame
-
Prepare GL texture for drawing:
glActiveTexture(GL_TEXTURE0); glBindTexture(GL_TEXTURE_EXTERNAL_OES, texture_);
Note :Texture target used for Video Decoder functionality must always be set to
GL_TEXTURE_EXTERNAL_OES
-
Request drawing the texture:
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4)
-
Request an animation frame:
emscripten_request_animation_frame(&CAPIOnDrawTextureCompleted, this);
-
Define a global callback function for
emscripten_request_animation_frame
:int CAPIOnDrawTextureCompleted(double /* time */, void* thiz) { if (thiz) static_cast<VideoDecoderTrackDataPump*>(thiz)->OnDrawCompleted(); return 0; }
Recycling video picture
It is important to always recycle video picture after it has been drawn.
To do so, we need to call RecycleTexture
method in emscripten_request_animation_frame
callback:
void VideoDecoderTrackDataPump::OnDrawCompleted() {
video_track_.RecycleTexture(texture_);
RequestNewVideoTexture();
}
End rendering loop
To properly end a rendering loop when rendering should be stopped, the application should always handle ElementaryMediaTrack::FillTextureWithNextFrame
errors - eg. OperationResult::kAlreadyDestroyed
when a track
was stopped before calling this method or OperationResult::kAborted
when a track was stopped after calling this method.
Before invalidating the pointer that has already been provided to the emscripten_request_animation_frame
callback, you should abort that callback with the emscripten_cancel_animation_frame
by providing a callback id returned by the emscripten_request_animation_frame
method.