Using Tizen WASM Video Decoder
using tizen wasm video decoder this section provides an extension to using tizen wasm player with renderingmode kvideotexture of elementarymediastreamsource it presents how to extend an existing wasm player application with video decoder functionalities using the tizen wasm video decoder sample related info tizen wasm player api documentation samples tizen wasm video decoder sample on github to enable the wasm video decoder in an existing wasm player application, you must set the video rendering mode configure the gl context initialize the gl implement the rendering loop the wasm video decoder allows the application to fill a requested gl texture with decoded frame, instead of rendering it on a htmlmediaelement setting video texture rendering mode for wasm player to change the elementarymediastreamsource rendering mode from media element to video texture, renderingmode kmediaelement must be replaced by renderingmode kvideotexture using latencymode = samsung wasm elementarymediastreamsource latencymode; using renderingmode = samsung wasm elementarymediastreamsource renderingmode; auto elementary_media_stream_source = std make_unique<samsung wasm elementarymediastreamsource> latencymode knormal, renderingmode kvideotexture ; configuring gl context this section covers the configuration of gl context using the samsung emscripten sdk you must first make the canvas accessible, and then prepare either sdl or egl for gl initialization making canvas accessible from emscripten gl context in wasm is associated with a <canvas> html element to make it possible for wasm to use it create a <canvas> element in the application's html file that runs the wasm module <canvas id="canvas" width=1600 height=900></canvas> extend the emscripten module object with information about the created canvas element module = { canvas function { return document getelementbyid 'canvas' ; } , } now you can access <canvas> html element from the wasm module for context initialization, get the canvas dimensions from the c++ code int width; int height; emscripten_get_canvas_element_size "#canvas", &width, &height ; these variables will be used later for context initialization using sdl for gl initialization to initialize gl using sdl create sdl_window with the desired parameters window_ = sdl_createwindow "videotexture", sdl_windowpos_centered, sdl_windowpos_centered, width, height, sdl_window_opengl | sdl_window_shown ; get an sdl_glcontext context from the window, and make it the current context gl_context_ = sdl_gl_createcontext window_ ; sdl_gl_makecurrent window_, gl_context_ ; initialize sdl for example sdl_init sdl_init_video ; sdl_gl_setattribute sdl_gl_context_major_version, 2 ; // indicates gles version to use sdl_gl_setattribute sdl_gl_context_minor_version, 0 ; sdl_gl_setattribute sdl_gl_doublebuffer, 1 ; sdl_gl_setattribute sdl_gl_depth_size, 24 ; // indicates gl color depth sdl_gl_setattribute sdl_gl_multisamplebuffers, 1 ; sdl_gl_setattribute sdl_gl_multisamplesamples, 4 ; // turns on multisampling using egl for gl initialization as an alternative for sdl initialization, gl can also be initialized using the egl wrapper initialize the egl config for example const eglint attrib_list[] = { egl_red_size, 8, egl_green_size, 8, egl_blue_size, 8, egl_alpha_size, egl_dont_care, egl_depth_size, egl_dont_care, egl_stencil_size, egl_dont_care, egl_sample_buffers, 0, egl_none }; const eglint context_attribs[] = { egl_context_client_version, 2, egl_none }; eglint num_configs; eglint major_version; eglint minor_version; eglconfig config; egldisplay display = eglgetdisplay egl_default_display ; eglinitialize display, &major_version, &minor_version ; eglgetconfigs display, null, 0, &num_configs , egl_true eglchooseconfig display, attrib_list, &config, 1, &num_configs create eglsurface eglsurface surface = eglcreatewindowsurface display, config, null, null ; get eglcontext from the window surface and make it the current context eglcontext context = eglcreatecontext display, config, egl_no_context, context_attribs ; eglmakecurrent display, surface, surface, context ; initializing gl to initialize gl create a texture the texture is filled with video frames decoded by wasm player glgentextures 1, &texture_ ; set a viewport to allow gl to automatically scale rendering to the provided viewport glviewport 0, 0, width, height ; compile shaders and link them to the program define a vertex shader for example const char kvertexshader[] = "varying vec2 v_texcoord; \n" "attribute vec4 a_position; \n" "attribute vec2 a_texcoord; \n" "uniform vec2 v_scale; \n" "void main \n" "{ \n" " v_texcoord = v_scale * a_texcoord; \n" " gl_position = a_position; \n" "}"; define a fragment shader for example const char kfragmentshaderexternal[] = "#extension gl_oes_egl_image_external require \n" "precision mediump float; \n" "varying vec2 v_texcoord; \n" "uniform samplerexternaloes s_texture; \n" "void main \n" "{ \n" " gl_fragcolor = texture2d s_texture, v_texcoord ; \n" "} note#extension gl_oes_egl_image_external require and the sampler uniform samplerexternaloes are required for video decoder functionality create the compile shader function void createshader gluint program, glenum type, const char* source, int size { gluint shader = glcreateshader type ; glshadersource shader, 1, &source, &size ; glcompileshader shader ; glattachshader program, shader ; gldeleteshader shader ; } create a program, compile shaders, and link them into the program program_ = glcreateprogram ; createshader program_, gl_vertex_shader, kvertexshader, strlen kvertexshader ; createshader program_, gl_fragment_shader, kfragmentshaderexternal, strlen kfragmentshaderexternal ; gllinkprogram program_ ; gluseprogram program_ ; to use gles 3 webgl 2 for wasm video decoder functionality optional add version information at the beginning of both vertex and fragment shaders #version 300 es in the fragment shader, change the following line #extension gl_oes_egl_image_external require to #extension gl_oes_egl_image_external_essl3 require in the fragment shader in the fragment shader definition, use the texture keyword instead of texture2d set the gl major version to 3 sdl_gl_setattribute sdl_gl_context_major_version, 3 ; notenote that the egl provided by the emscripten sdk does not support setting the gl major version, so it is not possible to use gles 3 0 using egl wrapper register the gl context for wasm video decoder by informing elementarymediatrack about current graphics context video_track_ registercurrentgraphicscontext ; implementing the rendering loop to implement the wasm video decoder rendering loop request the video texture fill the decoding loop that fills the texture with a decoded video frame can be started after ontrackopen event is received or when the htmlmediaelement play callback is called when the texture is filled with the video frame, draw it void videodecodertrackdatapump requestnewvideotexture { video_track_ filltexturewithnextframe texture_, [this] samsung wasm operationresult result { if result != samsung wasm operationresult ksuccess { std cout << "filling texture with next frame failed" << std endl; return; } draw ; } ; } draw the texture like in any c++ application, you need to provide a rendering loop in the wasm video decoder application, the loop is created with the cycle illustrated in the following image prepare the gl texture for drawing glactivetexture gl_texture0 ; glbindtexture gl_texture_external_oes, texture_ ; notethe texture target used for video decoder functionality must always be set to gl_texture_external_oes request drawing the texture gldrawarrays gl_triangle_strip, 0, 4 request an animation frame emscripten_request_animation_frame &capiondrawtexturecompleted, this ; define a global callback function for emscripten_request_animation_frame int capiondrawtexturecompleted double /* time */, void* thiz { if thiz static_cast<videodecodertrackdatapump*> thiz ->ondrawcompleted ; return 0; } recycle the video picture it is important to always recycle the video picture after it has been drawn to do so, call the recycletexture method in the emscripten_request_animation_frame callback void videodecodertrackdatapump ondrawcompleted { video_track_ recycletexture texture_ ; requestnewvideotexture ; } when rendering must be stopped, end the rendering loop to properly end a rendering loop, the application needs to handle elementarymediatrack filltexturewithnextframe errors, such as operationresult kalreadydestroyed when a track was stopped before calling this method or operationresult kaborted when a track was stopped after calling this method before invalidating the pointer that has already been provided to the emscripten_request_animation_frame callback, abort that callback withemscripten_cancel_animation_frame, by providing the callback id returned by the emscripten_request_animation_frame method