Implement Flex Mode on an Unreal Engine Game
implement flex mode on an unreal engine game objective learn how to implement flex mode on an unreal engine game using android jetpack windowmanager and raw java native interface jni overview the flexible hinge and glass display on galaxy foldable devices, such as the galaxy z fold4 and galaxy z flip4, let the phone remains propped open while you use apps when the phone is partially folded, it will go into flex mode apps will reorient to fit the screen, letting you watch videos or play games without holding the phone for example, you can set the device on a flat surface, like on a table, and use the bottom half of the screen to navigate unfold the phone to use the apps in full screen mode, and partially fold it again to return to flex mode to provide users with a convenient and versatile foldable experience, developers need to optimize their apps to meet the flex mode standard set up your environment you will need the following epic games launcher with unreal engine 4 or later visual studio or any source code editor samsung galaxy foldable device galaxy z fold2, z fold3, or newer galaxy z flip, z flip3, or newer remote test lab if physical device is not available requirements samsung account java runtime environment jre 7 or later with java web start internet environment where port 2600 is available create and set up your project after launching unreal engine from the epic games launcher, follow the steps below to start your project in the select or create new project window, choose games as a new project category and click next select third person as template, then click next to proceed noteyou can implement flex mode on any template or existing projects and use this code lab activity as a reference in the project settings window, set the following type of project c++ target platform mobile / tablet performance characteristics scalable 3d or 2d real-time raytracing raytracing disabled include an additional content pack no starter content project name tutorial_project click create project wait for the engine to finish loading and open the unreal editor once the project is loaded, go to edit > project settings > platforms > android click the configure now button if the project is not yet configured for the android platform then, proceed with the following apk packaging and build settings a apk packaging set target sdk version to 30 set orientation to full sensor change the maximum supported aspect ratio to 2 8 aspect ratio of galaxy z fold3 in decimal to prevent black bars from appearing on the cover display leave it if your game does not need to use the cover display enable use display cutout region?, to prevents black bars at the edge of the main screen otherwise, leave it unchecked b build disable support opengl es3 1 and enable support vulkan notecurrently, there is a problem with opengl es and the split-screen system being investigated the only option right now is to turn off opengl es and use vulkan instead enable native resize event the resize event of a game when switching between displays is disabled in the engine by default however, this behavior can be easily enabled by setting android enablenativeresizeevent=1 in the deviceprofile currently, the only way to create a profile for foldable devices is by creating a specific rule for each device to save time in this code lab, enable the native resize event for all android devices instead locate and open the tutorial_project > config folder in file explorer inside the config folder, create a new folder named android create a new file called androiddeviceprofiles ini and open this file in a text editor, such as visual studio copy below deviceprofile code to the newly created androiddeviceprofiles ini file [deviceprofiles] +deviceprofilenameandtypes=android,android [android deviceprofile] devicetype=android baseprofilename= +cvars=r mobilecontentscalefactor=1 0 +cvars=slate absoluteindices=1 +cvars=r vulkan delayacquirebackbuffer=2 +cvars=r vulkan robustbufferaccess=1 +cvars=r vulkan descriptorsetlayoutmode=2 ; don't enable vulkan by default specific device profiles can set this cvar to 0 to enable vulkan +cvars=r android disablevulkansupport=1 +cvars=r android disablevulkansm5support=1 ; pf_b8g8r8a8 +cvars=r defaultbackbufferpixelformat=0 +cvars=android enablenativeresizeevent=1 ; previewallowlistcvars and previewdenylistcvars are arrays of cvars that are included or excluded from being applied in mobile preview ; if any previewallowlistcvars is set, cvars are denied by default previewallowlistcvars=none this is a copy of the default android deviceprofile from the existing basedeviceprofiles ini file but with the enabled nativeresizeevent console variable cvars notethis step is not required when you only want to implement flex mode yet, it's recommended, to allow applications to run seamlessly from main to cover display without stretching and squashing the game, by enabling the nativeresizeevent create a new plugin and import the foldablehelper foldablehelper is a java file that you can use in different projects it provides an interface to the android jetpack windowmanager library, enabling application developers to support new device form factors and multi-window environments before proceeding, read how to use jetpack windowmanager in android game dev and learn the details of how foldablehelper uses windowmanager library to retrieve information about the folded state of the device flat for normal mode and half-opened for flex mode , window size, and orientation of the fold on the screen download the foldablehelper java file here foldablehelper java 5 64 kb to import the foldablehelper java file to the project, follow the steps below go to edit > plugins in the unreal editor click the new plugin button and select blank to create a blank plugin in the name field, type foldables_tutorial and click the create plugin button in file explorer, locate and open tutorial_project > plugins folder go to plugins > foldables_tutorial > source> foldables_tutorial > private and create a new folder called java copy the foldablehelper java file into java folder open the tutorial_project sln file in visual studio in the same private folder path, add a new filter called java right-click on the java filter and click add > existing item locate the foldablehelper java file, then click add to include this java file in the build modify java activity to use foldablehelper unreal plugin language upl is a simple xml-based language created by epic games for manipulating xml and returning strings using upl, you can utilize the foldablehelper java file by modifying the java activity and related gradle files as follows in visual studio, right-click on source > foldables_tutorial folder, then click add > new item > web > xml file xml create an xml file called foldables_tutorial_upl xml ensure that the file location is correct before clicking add in the newly created xml file, include the foldablehelper java file in the build by copying the java folder to the build directory <root xmlns android="http //schemas android com/apk/res/android"> <prebuildcopies> <copydir src="$s plugindir /private/java" dst="$s builddir /src/com/samsung/android/gamedev/foldable" /> </prebuildcopies> set up the gradle dependencies in the build gradle file by adding the following in the xml file <buildgradleadditions> <insert> dependencies { implementation filetree dir 'libs', include ['* jar'] implementation "org jetbrains kotlin kotlin-stdlib-jdk8 1 6 0" implementation "androidx core core 1 7 0" implementation "androidx core core-ktx 1 7 0" implementation "androidx appcompat appcompat 1 4 0" implementation "androidx window window 1 0 0" implementation "androidx window window-java 1 0 0" } android{ compileoptions{ sourcecompatibility javaversion version_1_8 targetcompatibility javaversion version_1_8 } } </insert> </buildgradleadditions> next, modify the gameactivity <gameactivityimportadditions> <insert> <!-- package name of foldablehelper --> import com samsung android gamedev foldable foldablehelper; </insert> </gameactivityimportadditions> <gameactivityoncreateadditions> <insert> foldablehelper init this ; </insert> </gameactivityoncreateadditions> <gameactivityonstartadditions> <insert> foldablehelper start this ; </insert> </gameactivityonstartadditions> <gameactivityonstopadditions> <insert> foldablehelper stop ; </insert> </gameactivityonstopadditions> </root> gameactivityimportadditions adds the com samsung android gamedev foldable foldablehelper into the gameactivity with the existing imports gameactivityoncreateadditions adds the code to the oncreate method inside the gameactivity gameactivityonstartadditions adds the code to the onstart method inside the gameactivity gameactivityonstopadditions adds the code to the onstop method inside the gameactivity save the xml file then, ensure that the engine uses the upl file by modifying the foldables_tutorial build cs script, located in the same folder as the foldables_tutorial_upl xml file after the dynamicallyloadedmodulenames addrange call, add the following if target platform == unrealtargetplatform android { additionalpropertiesforreceipt add "androidplugin", moduledirectory + "\\foldables_tutorial_upl xml" ; } this means that the game engine will use the upl file if the platform is android otherwise, the foldablehelper won’t work implement a storage struct the next thing to implement is a struct, the native version of java’s foldablelayoutinfo class to store the data retrieved from the java code using a struct, do the following in content browser of unreal editor, right-click on c++ classes > add/import content then, click new c++ class select none for the parent class and click next name the new class as foldablelayoutinfo assign it to the foldables_tutorial plugin then, click create class delete the created foldablelayoutinfo cpp file and only keep its header file in the header file called foldablelayoutinfo h, set up a struct to store all needed data from the windowmanager #pragma once #include "core h" enum efoldstate { undefined_state, flat, half_opened }; enum efoldorientation { undefined_orientation, horizontal, vertical }; enum efoldocclusiontype { undefined_occlusion, none, full }; struct ffoldablelayoutinfo { efoldstate state; efoldorientation orientation; efoldocclusiontype occlusiontype; fvector4 foldbounds; fvector4 currentmetrics; fvector4 maxmetrics; bool isseparating; ffoldablelayoutinfo state efoldstate undefined_state , orientation efoldorientation undefined_orientation , occlusiontype efoldocclusiontype undefined_occlusion , foldbounds -1, -1, -1, -1 , currentmetrics -1, -1, -1, -1 , maxmetrics -1, -1, -1, -1 , isseparating false { } }; implement jni code to implement jni, create a new c++ class with no parent and name it foldables_helper assign the class to the same plugin, then modify the c++ header and source files as follows in the created header file foldables_helper h , include foldablelayoutinfo h #include "foldablelayoutinfo h" then, declare a multicast_delegate to serve as a listener for passing the data from the java implementation to the rest of the engine declare_multicast_delegate_oneparam fonlayoutchangeddelegate, ffoldablelayoutinfo ; lastly, set up the methods and member variables class foldables_tutorial_api ffoldables_helper { public static void init ; static bool haslistener; static fonlayoutchangeddelegate onlayoutchanged; }; moving to the source file foldables_helper cpp , set up the definitions for the methods and member variables created in the header file bool ffoldables_helper haslistener = false; fonlayoutchangeddelegate ffoldables_helper onlayoutchanged; void ffoldables_helper init { haslistener = true; } now, in the same source file, create the native version of the onlayoutchanged function created in the foldablehelper java file since the java onlayoutchanged function only works on android, surround the function with an #if directive to ensure that it compiles only on android #if platform_android #endif within this directive, copy the code below to use the jni definition of the java onlayoutchanged function extern "c" jniexport void jnicall java_com_samsung_android_gamedev_foldable_foldablehelper_onlayoutchanged jnienv * env, jclass clazz, jobject jfoldablelayoutinfo { create the ffoldablelayoutinfo to store the data retrieved from java ffoldablelayoutinfo result; retrieve the field ids of the foldablelayoutinfo and rect objects created in the java file //java foldablelayoutinfo field ids jclass jfoldablelayoutinfocls = env->getobjectclass jfoldablelayoutinfo ; jfieldid currentmetricsid = env->getfieldid jfoldablelayoutinfocls, "currentmetrics", "landroid/graphics/rect;" ; jfieldid maxmetricsid = env->getfieldid jfoldablelayoutinfocls, "maxmetrics", "landroid/graphics/rect;" ; jfieldid hingeorientationid = env->getfieldid jfoldablelayoutinfocls, "hingeorientation", "i" ; jfieldid stateid = env->getfieldid jfoldablelayoutinfocls, "state", "i" ; jfieldid occlusiontypeid = env->getfieldid jfoldablelayoutinfocls, "occlusiontype", "i" ; jfieldid isseparatingid = env->getfieldid jfoldablelayoutinfocls, "isseparating", "z" ; jfieldid boundsid = env->getfieldid jfoldablelayoutinfocls, "bounds", "landroid/graphics/rect;" ; jobject currentmetricsrect = env->getobjectfield jfoldablelayoutinfo, currentmetricsid ; //java rect object field ids jclass rectcls = env->getobjectclass currentmetricsrect ; jfieldid leftid = env->getfieldid rectcls, "left", "i" ; jfieldid topid = env->getfieldid rectcls, "top", "i" ; jfieldid rightid = env->getfieldid rectcls, "right", "i" ; jfieldid bottomid = env->getfieldid rectcls, "bottom", "i" ; retrieve the current windowmetrics and store it in the ffoldablelayoutinfo as an fintvector4 // currentmetrics int left = env->getintfield currentmetricsrect, leftid ; int top = env->getintfield currentmetricsrect, topid ; int right = env->getintfield currentmetricsrect, rightid ; int bottom = env->getintfield currentmetricsrect, bottomid ; // store currentmetrics rect to fvector4 result currentmetrics = fintvector4{ left, top, right, bottom }; do the same for the other variables in the java foldablelayoutinfo // maxmetrics jobject maxmetricsrect = env->getobjectfield jfoldablelayoutinfo, maxmetricsid ; left = env->getintfield maxmetricsrect, leftid ; top = env->getintfield maxmetricsrect, topid ; right = env->getintfield maxmetricsrect, rightid ; bottom = env->getintfield maxmetricsrect, bottomid ; //store maxmetrics rect to fvector4 result maxmetrics = fintvector4{ left, top, right, bottom }; int hingeorientation = env->getintfield jfoldablelayoutinfo, hingeorientationid ; int state = env->getintfield jfoldablelayoutinfo, stateid ; int occlusiontype = env->getintfield jfoldablelayoutinfo, occlusiontypeid ; bool isseparating = env->getbooleanfield jfoldablelayoutinfo, isseparatingid ; // store the values to an object for unreal result orientation = tenumasbyte<efoldorientation> hingeorientation + 1 ; result state = tenumasbyte<efoldstate> state + 1 ; result occlusiontype = tenumasbyte<efoldocclusiontype> occlusiontype + 1 ; result isseparating = isseparating; // boundsrect jobject boundsrect = env->getobjectfield jfoldablelayoutinfo, boundsid ; left = env->getintfield boundsrect, leftid ; top = env->getintfield boundsrect, topid ; right = env->getintfield boundsrect, rightid ; bottom = env->getintfield boundsrect, bottomid ; // store maxmetrics rect to fvector4 result foldbounds = fintvector4{ left, top, right, bottom }; broadcast the result via the onlayoutchanged delegate for use in the engine if ffoldables_helper haslistener { ue_log logtemp, warning, text "broadcast" ; ffoldables_helper onlayoutchanged broadcast result ; } } create a player controller and two ui states this section focuses on adding a player controller and creating two user interface ui states for flat and flex modes these objects are needed for the flex mode logic implementation following are the steps to add a player controller and create two ui states add a new player controller blueprint in content browser, go to content > thirdpersoncpp and right-click on blueprints > add/import content > blueprint class pick player controller as its parent class rename it as flexplayercontroller notethe flexplayercontroller added is generic and can be replaced by your custom player controller in an actual project add a new c++ class with actor component as its parent class name it as foldables_manager and assign it to the foldables_tutorial plugin click the create class button open the flexplayercontroller blueprint by double-clicking it click open full blueprint editor attach the actor component to the flexplayercontroller in the left pane, click add component, then find and select the foldables_manager next, create a pair of userwidget classes for the ui layouts needed flat mode ui for the full screen or normal mode; and flex mode ui for split-screen in add c++ class window, select the show all classes checkbox find and pick userwidget as the parent class then, click next name the new user widget as flatui and attach it to the plugin click next repeat the process but name the new user widget as flexui you might get an error when trying to compile stating that the userwidget is an undeclared symbol to fix this, open the foldables_tutorial build cs file, and in the publicdependencymodulenames addrange call, add "inputcore" and "umg" to the list create a pair of blueprints made from subclasses of these two user widgets right-click on content and create a new folder called foldui inside the newly created folder, right-click to add a new blueprint class in all classes, choose flatui and click the select button rename the blueprint as bp_flatui in the same folder, repeat the process but choose the flexui class and rename the blueprint as bp_flexui double-click on bp_flatui and bp_flexui, then design your two uis like below to visualize switching between flat and flex mode flat ui flex ui notethis code lab activity does not cover the steps on how to create or design ui if you want to learn about how to create your own game design in unreal engine 4, refer to unreal motion graphics ui designer guide implement the flex mode logic after creating the flexplayercontroller and the two ui states bp_flatui and bp_flexui , you can now implement flex mode logic in the foldables_manager open the foldables_manager h and include the necessary c++ header files #pragma once #include "coreminimal h" #include "components/actorcomponent h" #include "engine h" #include "flatui h" #include "flexui h" #include "foldables_helper h" #include "foldables_manager generated h" remove the line below to save a little bit of performance as this component doesn't need to tick public // called every frame virtual void tickcomponent float deltatime, eleveltick ticktype, factorcomponenttickfunction* thistickfunction override; set up the functions needed in foldables_manager the constructor, a function to create the ui widgets the implementation of onlayoutchanged delegate public // sets default values for this component's properties ufoldables_manager ; void createwidgets ; protected // called when the game starts virtual void beginplay override; protected void onlayoutchanged ffoldablelayoutinfo foldablelayoutinfo ; then, set up the variables needed references to the flat and flex ui classes references to the flat and flex ui objects mark the pointers as uproperty to ensure that garbage collection does not delete the objects they point to tsubclassof<uuserwidget> flatuiclass; tsubclassof<uuserwidget> flexuiclass; uproperty class uflatui* flatui; uproperty class uflexui* flexui; finally, define a new private function restoreflatmode , to disable flex mode and return to normal mode private void restoreflatmode ; moving over to foldables_manager cpp, implement the constructor using the constructorhelpers, find the ui classes and set the variables to store these classes also, set the bcanevertick to false to prevent the component from ticking and remove the code block of tickcomponent function // sets default values for this component's properties ufoldables_manager ufoldables_manager { primarycomponenttick bcanevertick = false; static constructorhelpers fclassfinder<uflatui> flatuibpclass text "/game/foldui/bp_flatui" ; static constructorhelpers fclassfinder<uflexui> flexuibpclass text "/game/foldui/bp_flexui" ; if flatuibpclass succeeded { flatuiclass = flatuibpclass class; } if flexuibpclass succeeded { flexuiclass = flexuibpclass class; } } next, set up the beginplay function to link the delegate to the onlayoutchanged function, to initialize the foldables_helper, and to create the widgets ready for use in the first frame // called when the game starts void ufoldables_manager beginplay { super beginplay ; ffoldables_helper onlayoutchanged adduobject this, &ufoldables_manager onlayoutchanged ; ffoldables_helper init ; createwidgets ; } set up the createwidgets function to create the widgets using the ui classes acquired in the constructor add the flatui widget to the viewport, assuming the app opens in normal mode until it receives the foldablelayoutinfo void ufoldables_manager createwidgets { flatui = createwidget<uflatui> aplayercontroller* getowner , flatuiclass, fname text "flatui" ; flexui = createwidget<uflexui> aplayercontroller* getowner , flexuiclass, fname text "flexui" ; flatui->addtoviewport ; } afterward, create the onlayoutchanged function, which will be called via a delegate inside this function, check whether the device’s folded state is half_opened if so, check whether the orientation of the fold is horizontal void ufoldables_manager onlayoutchanged ffoldablelayoutinfo foldablelayoutinfo { //if state is now flex if foldablelayoutinfo state == efoldstate half_opened { if foldablelayoutinfo orientation == efoldorientation horizontal { notefor this third person template, splitting the screen vertically isn’t ideal from a user experience ux point of view for this code lab activity, split the screen only on the horizontal fold top and bottom screen if you want it vertically, you need to use the same principle in the next step but for the x-axis instead of the y-axis you must also ensure that you have a flex ui object for the vertical layout if the device is both on flex mode and horizontal fold, change the viewport to only render on the top screen using the normalized position of the folding feature then in an asynctask on the game thread, disable the flatui and enable the flexui however, if the device is on normal mode, then return to flat ui using restoreflatmode function //horizontal split float foldratio = float foldablelayoutinfo foldbounds y / foldablelayoutinfo currentmetrics w - foldablelayoutinfo currentmetrics y ; gengine->gameviewport->splitscreeninfo[0] playerdata[0] sizex = 1 0f; gengine->gameviewport->splitscreeninfo[0] playerdata[0] sizey = foldratio; asynctask enamedthreads gamethread, [=] { if flatui->isinviewport { flatui->removefromparent ; } if !flexui->isinviewport { flexui->addtoviewport 0 ; } } ; } else { restoreflatmode ; } } else { restoreflatmode ; } } reverse the flex mode implementation logic to create the restoreflatmode function by setting the viewport to fill the screen, then disable the flexui and enable the flatui void ufoldables_manager restoreflatmode { gengine->gameviewport->splitscreeninfo[0] playerdata[0] sizex = 1 0f; gengine->gameviewport->splitscreeninfo[0] playerdata[0] sizey = 1 0f; asynctask enamedthreads gamethread, [=] { if !flatui->isinviewport { flatui->addtoviewport 0 ; } if flexui->isinviewport { flexui->removefromparent ; } } ; } set up a game mode and attach the flexplayercontroller the game mode defines the game rules, scoring, and any game-specific behavior set up the game mode in unreal editor by creating a blueprint class in the content > thirdpersoncpp > blueprints folder pick game mode base as the parent class and rename it as flexgamemode double-click on flexgamemode in the drop-down menu next to player controller class, choose the flexplayercontroller lastly, go to edit > project settings > project > maps & modes and select flexgamemode as the default gamemode build and run the app go to edit > package project > android to build the apk ensure that the android development environment for unreal is already set up to your computer noteif the build failed due to corrupted build tools, consider downgrading the version to 30 or lower using the sdk manager or, simply rename d8 bat to the name of the missing file dx bat in sdk path > build-tools > version number directory and, in lib folder of the same directory, rename d8 jar to dx jar after packaging your android project, run the game app on a foldable galaxy device and see how the ui switches from normal to flex mode if you don’t have any physical device, you can also test it on a remote test lab device tipwatch this tutorial video and know how to easily test your app via remote test lab you're done! congratulations! you have successfully achieved the goal of this code lab now, you can implement flex mode in your unreal engine game app by yourself! if you're having trouble, you may download this file flex mode on unreal complete code 20 16 mb to learn more, visit www developer samsung com/galaxy-z www developer samsung com/galaxy-gamedev