implement flex mode on an unreal engine game objective learn how to implement flex mode on an unreal engine game using android jetpack windowmanager and raw java native interface (jni). overview the flexible hinge and glass display on galaxy foldable devices, such as the galaxy z fold4 and galaxy z flip4, let the phone remains propped open while you use apps. when the phone is partially folded, it will go into flex mode. apps will reorient to fit the screen, letting you watch videos or play games without holding the phone. for example, you can set the device on a flat surface, like on a table, and use the bottom half of the screen to navigate. unfold the phone to use the apps in full screen mode, and partially fold it again to return to flex mode. to provide users with a convenient and versatile foldable experience, developers need to optimize their apps to meet the flex mode standard. set up your environment you will need the following: epic games launcher with unreal engine 4 or later visual studio or any source code editor samsung galaxy foldable device: galaxy z fold2, z fold3, or newer galaxy z flip, z flip3, or newer remote test lab (if physical device is not available) requirements: samsung account java runtime environment (jre) 7 or later with java web start internet environment where port 2600 is available create and set up your project after launching unreal engine from the epic games launcher, follow the steps below to start your project: in the select or create new project window, choose games as a new project category and click next. select third person as template, then click next to proceed. noteyou can implement flex mode on any template or existing projects and use this code lab activity as a reference. in the project settings window, set the following: type of project: c++ target platform: mobile / tablet performance characteristics: scalable 3d or 2d real-time raytracing: raytracing disabled include an additional content pack: no starter content project name: tutorial_project click create project. wait for the engine to finish loading and open the unreal editor. once the project is loaded, go to edit > project settings > platforms > android. click the configure now button if the project is not yet configured for the android platform. then, proceed with the following apk packaging and build settings: a. apk packaging set target sdk version to 30. set orientation to full sensor. change the maximum supported aspect ratio to 2.8 (aspect ratio of galaxy z fold3 in decimal) to prevent black bars from appearing on the cover display. leave it if your game does not need to use the cover display. enable use display cutout region?, to prevents black bars at the edge of the main screen. otherwise, leave it unchecked. b. build disable support opengl es3.1 and enable support vulkan. notecurrently, there is a problem with opengl es and the split-screen system being investigated. the only option right now is to turn off opengl es and use vulkan instead. enable native resize event the resize event of a game when switching between displays is disabled in the engine by default. however, this behavior can be easily enabled by setting android.enablenativeresizeevent=1 in the deviceprofile. currently, the only way to create a profile for foldable devices is by creating a specific rule for each device. to save time in this code lab, enable the native resize event for all android devices instead. locate and open the tutorial_project > config folder in file explorer. inside the config folder, create a new folder named android. create a new file called androiddeviceprofiles.ini and open this file in a text editor, such as visual studio. copy below deviceprofile code to the newly created androiddeviceprofiles.ini file. [deviceprofiles] +deviceprofilenameandtypes=android,android [android deviceprofile] devicetype=android baseprofilename= +cvars=r.mobilecontentscalefactor=1.0 +cvars=slate.absoluteindices=1 +cvars=r.vulkan.delayacquirebackbuffer=2 +cvars=r.vulkan.robustbufferaccess=1 +cvars=r.vulkan.descriptorsetlayoutmode=2 ; don't enable vulkan by default. specific device profiles can set this cvar to 0 to enable vulkan. +cvars=r.android.disablevulkansupport=1 +cvars=r.android.disablevulkansm5support=1 ; pf_b8g8r8a8 +cvars=r.defaultbackbufferpixelformat=0 +cvars=android.enablenativeresizeevent=1 ; previewallowlistcvars and previewdenylistcvars are arrays of cvars that are included or excluded from being applied in mobile preview. ; if any previewallowlistcvars is set, cvars are denied by default. previewallowlistcvars=none this is a copy of the default android deviceprofile from the existing basedeviceprofiles.ini file but with the enabled nativeresizeevent console variable (cvars). notethis step is not required when you only want to implement flex mode. yet, it's recommended, to allow applications to run seamlessly from main to cover display without stretching and squashing the game, by enabling the nativeresizeevent. create a new plugin and import the foldablehelper foldablehelper is a java file that you can use in different projects. it provides an interface to the android jetpack windowmanager library, enabling application developers to support new device form factors and multi-window environments. before proceeding, read how to use jetpack windowmanager in android game dev and learn the details of how foldablehelper uses windowmanager library to retrieve information about the folded state of the device (flat for normal mode and half-opened for flex mode), window size, and orientation of the fold on the screen. download the foldablehelper.java file here: foldablehelper.java (5.64 kb) to import the foldablehelper.java file to the project, follow the steps below: go to edit > plugins in the unreal editor. click the new plugin button and select blank to create a blank plugin. in the name field, type foldables_tutorial and click the create plugin button. in file explorer, locate and open tutorial_project > plugins folder. go to plugins > foldables_tutorial > source> foldables_tutorial > private and create a new folder called java. copy the foldablehelper.java file into java folder. open the tutorial_project.sln file in visual studio. in the same private folder path, add a new filter called java. right-click on the java filter and click add > existing item. locate the foldablehelper.java file, then click add to include this java file in the build. modify java activity to use foldablehelper unreal plugin language (upl) is a simple xml-based language created by epic games for manipulating xml and returning strings. using upl, you can utilize the foldablehelper.java file by modifying the java activity and related gradle files as follows: in visual studio, right-click on source > foldables_tutorial folder, then click add > new item > web > xml file (.xml). create an xml file called foldables_tutorial_upl.xml. ensure that the file location is correct before clicking add. in the newly created xml file, include the foldablehelper.java file in the build by copying the java folder to the build directory. <root xmlns:android="http://schemas.android.com/apk/res/android"> <prebuildcopies> <copydir src="$s(plugindir)/private/java" dst="$s(builddir)/src/com/samsung/android/gamedev/foldable" /> </prebuildcopies> set up the gradle dependencies in the build.gradle file by adding the following in the xml file: <buildgradleadditions> <insert> dependencies { implementation filetree(dir: 'libs', include: ['*.jar']) implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.6.0" implementation "androidx.core:core:1.7.0" implementation "androidx.core:core-ktx:1.7.0" implementation "androidx.appcompat:appcompat:1.4.0" implementation "androidx.window:window:1.0.0" implementation "androidx.window:window-java:1.0.0" } android{ compileoptions{ sourcecompatibility javaversion.version_1_8 targetcompatibility javaversion.version_1_8 } } </insert> </buildgradleadditions> next, modify the gameactivity: <gameactivityimportadditions> <insert> <!-- package name of foldablehelper --> import com.samsung.android.gamedev.foldable.foldablehelper; </insert> </gameactivityimportadditions> <gameactivityoncreateadditions> <insert> foldablehelper.init(this); </insert> </gameactivityoncreateadditions> <gameactivityonstartadditions> <insert> foldablehelper.start(this); </insert> </gameactivityonstartadditions> <gameactivityonstopadditions> <insert> foldablehelper.stop(); </insert> </gameactivityonstopadditions> </root> gameactivityimportadditions adds the com.samsung.android.gamedev.foldable.foldablehelper into the gameactivity with the existing imports. gameactivityoncreateadditions adds the code to the oncreate() method inside the gameactivity. gameactivityonstartadditions adds the code to the onstart() method inside the gameactivity. gameactivityonstopadditions adds the code to the onstop() method inside the gameactivity. save the xml file. then, ensure that the engine uses the upl file by modifying the foldables_tutorial.build.cs script, located in the same folder as the foldables_tutorial_upl.xml file. after the dynamicallyloadedmodulenames.addrange call, add the following: if (target.platform == unrealtargetplatform.android) { additionalpropertiesforreceipt.add("androidplugin", moduledirectory + "\\foldables_tutorial_upl.xml"); } this means that the game engine will use the upl file if the platform is android. otherwise, the foldablehelper won’t work. implement a storage struct the next thing to implement is a struct, the native version of java’s foldablelayoutinfo class. to store the data retrieved from the java code using a struct, do the following: in content browser of unreal editor, right-click on c++ classes > add/import content. then, click new c++ class. select none for the parent class and click next. name the new class as foldablelayoutinfo. assign it to the foldables_tutorial plugin. then, click create class. delete the created foldablelayoutinfo.cpp file and only keep its header file. in the header file called foldablelayoutinfo.h, set up a struct to store all needed data from the windowmanager. #pragma once #include "core.h" enum efoldstate { undefined_state, flat, half_opened }; enum efoldorientation { undefined_orientation, horizontal, vertical }; enum efoldocclusiontype { undefined_occlusion, none, full }; struct ffoldablelayoutinfo { efoldstate state; efoldorientation orientation; efoldocclusiontype occlusiontype; fvector4 foldbounds; fvector4 currentmetrics; fvector4 maxmetrics; bool isseparating; ffoldablelayoutinfo() : state(efoldstate::undefined_state), orientation(efoldorientation::undefined_orientation), occlusiontype(efoldocclusiontype::undefined_occlusion), foldbounds(-1, -1, -1, -1), currentmetrics(-1, -1, -1, -1), maxmetrics(-1, -1, -1, -1), isseparating(false) { } }; implement jni code to implement jni, create a new c++ class with no parent and name it foldables_helper. assign the class to the same plugin, then modify the c++ header and source files as follows: in the created header file (foldables_helper.h), include foldablelayoutinfo.h. #include "foldablelayoutinfo.h" then, declare a multicast_delegate to serve as a listener for passing the data from the java implementation to the rest of the engine. declare_multicast_delegate_oneparam(fonlayoutchangeddelegate, ffoldablelayoutinfo); lastly, set up the methods and member variables. class foldables_tutorial_api ffoldables_helper { public: static void init(); static bool haslistener; static fonlayoutchangeddelegate onlayoutchanged; }; moving to the source file (foldables_helper.cpp), set up the definitions for the methods and member variables created in the header file. bool ffoldables_helper::haslistener = false; fonlayoutchangeddelegate ffoldables_helper::onlayoutchanged; void ffoldables_helper::init() { haslistener = true; } now, in the same source file, create the native version of the onlayoutchanged() function created in the foldablehelper.java file. since the java onlayoutchanged() function only works on android, surround the function with an #if directive to ensure that it compiles only on android. #if platform_android #endif within this directive, copy the code below to use the jni definition of the java onlayoutchanged() function. extern "c" jniexport void jnicall java_com_samsung_android_gamedev_foldable_foldablehelper_onlayoutchanged(jnienv * env, jclass clazz, jobject jfoldablelayoutinfo) { create the ffoldablelayoutinfo to store the data retrieved from java. ffoldablelayoutinfo result; retrieve the field ids of the foldablelayoutinfo and rect objects created in the java file. //java foldablelayoutinfo field ids jclass jfoldablelayoutinfocls = env->getobjectclass(jfoldablelayoutinfo); jfieldid currentmetricsid = env->getfieldid(jfoldablelayoutinfocls, "currentmetrics", "landroid/graphics/rect;"); jfieldid maxmetricsid = env->getfieldid(jfoldablelayoutinfocls, "maxmetrics", "landroid/graphics/rect;"); jfieldid hingeorientationid = env->getfieldid(jfoldablelayoutinfocls, "hingeorientation", "i"); jfieldid stateid = env->getfieldid(jfoldablelayoutinfocls, "state", "i"); jfieldid occlusiontypeid = env->getfieldid(jfoldablelayoutinfocls, "occlusiontype", "i"); jfieldid isseparatingid = env->getfieldid(jfoldablelayoutinfocls, "isseparating", "z"); jfieldid boundsid = env->getfieldid(jfoldablelayoutinfocls, "bounds", "landroid/graphics/rect;"); jobject currentmetricsrect = env->getobjectfield(jfoldablelayoutinfo, currentmetricsid); //java rect object field ids jclass rectcls = env->getobjectclass(currentmetricsrect); jfieldid leftid = env->getfieldid(rectcls, "left", "i"); jfieldid topid = env->getfieldid(rectcls, "top", "i"); jfieldid rightid = env->getfieldid(rectcls, "right", "i"); jfieldid bottomid = env->getfieldid(rectcls, "bottom", "i"); retrieve the current windowmetrics and store it in the ffoldablelayoutinfo as an fintvector4. // currentmetrics int left = env->getintfield(currentmetricsrect, leftid); int top = env->getintfield(currentmetricsrect, topid); int right = env->getintfield(currentmetricsrect, rightid); int bottom = env->getintfield(currentmetricsrect, bottomid); // store currentmetrics rect to fvector4 result.currentmetrics = fintvector4{ left, top, right, bottom }; do the same for the other variables in the java foldablelayoutinfo. // maxmetrics jobject maxmetricsrect = env->getobjectfield(jfoldablelayoutinfo, maxmetricsid); left = env->getintfield(maxmetricsrect, leftid); top = env->getintfield(maxmetricsrect, topid); right = env->getintfield(maxmetricsrect, rightid); bottom = env->getintfield(maxmetricsrect, bottomid); //store maxmetrics rect to fvector4 result.maxmetrics = fintvector4{ left, top, right, bottom }; int hingeorientation = env->getintfield(jfoldablelayoutinfo, hingeorientationid); int state = env->getintfield(jfoldablelayoutinfo, stateid); int occlusiontype = env->getintfield(jfoldablelayoutinfo, occlusiontypeid); bool isseparating = env->getbooleanfield(jfoldablelayoutinfo, isseparatingid); // store the values to an object for unreal result.orientation = tenumasbyte<efoldorientation>(hingeorientation + 1); result.state = tenumasbyte<efoldstate>(state + 1); result.occlusiontype = tenumasbyte<efoldocclusiontype>(occlusiontype + 1); result.isseparating = isseparating; // boundsrect jobject boundsrect = env->getobjectfield(jfoldablelayoutinfo, boundsid); left = env->getintfield(boundsrect, leftid); top = env->getintfield(boundsrect, topid); right = env->getintfield(boundsrect, rightid); bottom = env->getintfield(boundsrect, bottomid); // store maxmetrics rect to fvector4 result.foldbounds = fintvector4{ left, top, right, bottom }; broadcast the result via the onlayoutchanged delegate for use in the engine. if (ffoldables_helper::haslistener) { ue_log(logtemp, warning, text("broadcast")); ffoldables_helper::onlayoutchanged.broadcast(result); } } create a player controller and two ui states this section focuses on adding a player controller and creating two user interface (ui) states for flat and flex modes. these objects are needed for the flex mode logic implementation. following are the steps to add a player controller and create two ui states : add a new player controller blueprint. in content browser, go to content > thirdpersoncpp and right-click on blueprints > add/import content > blueprint class. pick player controller as its parent class. rename it as flexplayercontroller. notethe flexplayercontroller added is generic and can be replaced by your custom player controller in an actual project. add a new c++ class with actor component as its parent class. name it as foldables_manager and assign it to the foldables_tutorial plugin. click the create class button. open the flexplayercontroller blueprint by double-clicking it. click open full blueprint editor. attach the actor component to the flexplayercontroller. in the left pane, click add component, then find and select the foldables_manager. next, create a pair of userwidget classes for the ui layouts needed: flat mode ui for the full screen or normal mode; and flex mode ui for split-screen. in add c++ class window, select the show all classes checkbox. find and pick userwidget as the parent class. then, click next. name the new user widget as flatui and attach it to the plugin. click next. repeat the process but name the new user widget as flexui. you might get an error when trying to compile stating that the userwidget is an undeclared symbol. to fix this, open the foldables_tutorial.build.cs file, and in the publicdependencymodulenames.addrange call, add "inputcore" and "umg" to the list. create a pair of blueprints made from subclasses of these two user widgets. right-click on content and create a new folder called foldui. inside the newly created folder, right-click to add a new blueprint class. in all classes, choose flatui and click the select button. rename the blueprint as bp_flatui. in the same folder, repeat the process but choose the flexui class and rename the blueprint as bp_flexui. double-click on bp_flatui and bp_flexui, then design your two uis like below to visualize switching between flat and flex mode: flat ui flex ui notethis code lab activity does not cover the steps on how to create or design ui. if you want to learn about how to create your own game design in unreal engine 4, refer to unreal motion graphics ui designer guide. implement the flex mode logic after creating the flexplayercontroller and the two ui states (bp_flatui and bp_flexui), you can now implement flex mode logic in the foldables_manager. open the foldables_manager.h and include the necessary c++ header files: #pragma once #include "coreminimal.h" #include "components/actorcomponent.h" #include "engine.h" #include "flatui.h" #include "flexui.h" #include "foldables_helper.h" #include "foldables_manager.generated.h" remove the line below to save a little bit of performance as this component doesn't need to tick. public: // called every frame virtual void tickcomponent(float deltatime, eleveltick ticktype, factorcomponenttickfunction* thistickfunction) override; set up the functions needed in foldables_manager: the constructor, a function to create the ui widgets the implementation of onlayoutchanged delegate public: // sets default values for this component's properties ufoldables_manager(); void createwidgets(); protected: // called when the game starts virtual void beginplay() override; protected: void onlayoutchanged(ffoldablelayoutinfo foldablelayoutinfo); then, set up the variables needed: references to the flat and flex ui classes references to the flat and flex ui objects mark the pointers as uproperty to ensure that garbage collection does not delete the objects they point to. tsubclassof<uuserwidget> flatuiclass; tsubclassof<uuserwidget> flexuiclass; uproperty() class uflatui* flatui; uproperty() class uflexui* flexui; finally, define a new private function restoreflatmode(), to disable flex mode and return to normal mode. private: void restoreflatmode(); moving over to foldables_manager.cpp, implement the constructor. using the constructorhelpers, find the ui classes and set the variables to store these classes. also, set the bcanevertick to false to prevent the component from ticking and remove the code block of tickcomponent() function. // sets default values for this component's properties ufoldables_manager::ufoldables_manager() { primarycomponenttick.bcanevertick = false; static constructorhelpers::fclassfinder<uflatui> flatuibpclass(text("/game/foldui/bp_flatui")); static constructorhelpers::fclassfinder<uflexui> flexuibpclass(text("/game/foldui/bp_flexui")); if (flatuibpclass.succeeded()) { flatuiclass = flatuibpclass.class; } if (flexuibpclass.succeeded()) { flexuiclass = flexuibpclass.class; } } next, set up the beginplay() function to link the delegate to the onlayoutchanged() function, to initialize the foldables_helper, and to create the widgets ready for use in the first frame. // called when the game starts void ufoldables_manager::beginplay() { super::beginplay(); ffoldables_helper::onlayoutchanged.adduobject(this, &ufoldables_manager::onlayoutchanged); ffoldables_helper::init(); createwidgets(); } set up the createwidgets() function to create the widgets using the ui classes acquired in the constructor. add the flatui widget to the viewport, assuming the app opens in normal mode until it receives the foldablelayoutinfo. void ufoldables_manager::createwidgets() { flatui = createwidget<uflatui>((aplayercontroller*)getowner(), flatuiclass, fname(text("flatui"))); flexui = createwidget<uflexui>((aplayercontroller*)getowner(), flexuiclass, fname(text("flexui"))); flatui->addtoviewport(); } afterward, create the onlayoutchanged() function, which will be called via a delegate. inside this function, check whether the device’s folded state is half_opened. if so, check whether the orientation of the fold is horizontal. void ufoldables_manager::onlayoutchanged(ffoldablelayoutinfo foldablelayoutinfo) { //if state is now flex if (foldablelayoutinfo.state == efoldstate::half_opened) { if (foldablelayoutinfo.orientation == efoldorientation::horizontal) { notefor this third person template, splitting the screen vertically isn’t ideal from a user experience (ux) point of view. for this code lab activity, split the screen only on the horizontal fold (top and bottom screen). if you want it vertically, you need to use the same principle in the next step but for the x-axis instead of the y-axis. you must also ensure that you have a flex ui object for the vertical layout. if the device is both on flex mode and horizontal fold, change the viewport to only render on the top screen using the normalized position of the folding feature. then in an asynctask on the game thread, disable the flatui and enable the flexui. however, if the device is on normal mode, then return to flat ui using restoreflatmode() function. //horizontal split float foldratio = (float)foldablelayoutinfo.foldbounds.y / (foldablelayoutinfo.currentmetrics.w - foldablelayoutinfo.currentmetrics.y); gengine->gameviewport->splitscreeninfo[0].playerdata[0].sizex = 1.0f; gengine->gameviewport->splitscreeninfo[0].playerdata[0].sizey = foldratio; asynctask(enamedthreads::gamethread, [=]() { if (flatui->isinviewport()) { flatui->removefromparent(); } if (!flexui->isinviewport()) { flexui->addtoviewport(0); } }); } else { restoreflatmode(); } } else { restoreflatmode(); } } reverse the flex mode implementation logic to create the restoreflatmode() function by setting the viewport to fill the screen, then disable the flexui and enable the flatui. void ufoldables_manager::restoreflatmode() { gengine->gameviewport->splitscreeninfo[0].playerdata[0].sizex = 1.0f; gengine->gameviewport->splitscreeninfo[0].playerdata[0].sizey = 1.0f; asynctask(enamedthreads::gamethread, [=]() { if (!flatui->isinviewport()) { flatui->addtoviewport(0); } if (flexui->isinviewport()) { flexui->removefromparent(); } }); } set up a game mode and attach the flexplayercontroller the game mode defines the game rules, scoring, and any game-specific behavior. set up the game mode in unreal editor by creating a blueprint class in the content > thirdpersoncpp > blueprints folder. pick game mode base as the parent class and rename it as flexgamemode. double-click on flexgamemode. in the drop-down menu next to player controller class, choose the flexplayercontroller. lastly, go to edit > project settings > project > maps & modes and select flexgamemode as the default gamemode. build and run the app go to edit > package project > android to build the apk. ensure that the android development environment for unreal is already set up to your computer. noteif the build failed due to corrupted build tools, consider downgrading the version to 30 or lower using the sdk manager. or, simply rename d8.bat to the name of the missing file (dx.bat) in (sdk path) > build-tools > (version number) directory and, in lib folder of the same directory, rename d8.jar to dx.jar. after packaging your android project, run the game app on a foldable galaxy device and see how the ui switches from normal to flex mode. if you don’t have any physical device, you can also test it on a remote test lab device. tipwatch this tutorial video and know how to easily test your app via remote test lab. you're done! congratulations! you have successfully achieved the goal of this code lab. now, you can implement flex mode in your unreal engine game app by yourself! if you're having trouble, you may download this file: flex mode on unreal complete code (20.16 mb) to learn more, visit: www.developer.samsung.com/galaxy-z www.developer.samsung.com/galaxy-gamedev
