Implement Flex Mode into a Unity game
implement flex mode into a unity game objective learn how to implement flex mode into a unity game using android jetpack windowmanager and unity's java native interface jni wrapper overview the flexible hinge and glass display on galaxy foldable devices, such as the galaxy z fold4 and galaxy z flip4, let the phone remains propped open while you use apps when the phone is partially folded, it will go into flex mode apps will reorient to fit the screen, letting you watch videos or play games without holding the phone for example, you can set the device on a flat surface, like on a table, and use the bottom half of the screen to navigate unfold the phone to use the apps in full screen mode, and partially fold it again to return to flex mode to provide users with a convenient and versatile foldable experience, developers need to optimize their apps to meet the flex mode standard set up your environment you will need the following unity hub with unity 2022 3 5f1 or later must have android build support visual studio or any source code editor galaxy z fold2 or newer remote test lab if physical device is not available requirements samsung account java runtime environment jre 7 or later with java web start internet environment where port 2600 is available sample code here is a sample project for you to start coding in this code lab download it and start your learning experience! flex mode on unity sample code 1 17 gb start your project after downloading the sample project files, follow the steps below to open your project launch the unity hub click projects > open locate the unzipped project folder and click open to add the project to the hub and open in the editor notethe sample project was created in unity 2022 3 5f1 if you prefer using a different unity version, click choose another editor version when prompted and select a higher version of unity configure android player settings to ensure that the project runs smoothly on the android platform, configure the player settings as follows go to file > build settings under platform, choose android and click switch platform wait until this action finishes importing necessary assets and compiling scripts then, click player settings to open the project settings window go to player > other settings and scroll down to see target api level set it to api level 33 as any less than this will result in a dependency error regarding an lstar variable you can set the minimum api level on lower levels without any problem next, in the resolution and presentation settings, enable resizable window it is also recommended that render outside safe area is enabled to prevent black bars on the edges of the screen lastly, enable the custom main manifest, custom main gradle template, and custom gradle properties template in the publishing settings after closing the project settings window, check for the new folder structure created within your assets in the project window the newly created android folder contains androidmanifest xml, gradletemplate properties, and maintemplate gradle files import the foldablehelper and add dependencies foldablehelper is a java file that you can use in different projects it provides an interface to the android jetpack windowmanager library, enabling application developers to support new device form factors and multi-window environments before proceeding, read how to use jetpack windowmanager in android game dev and learn the details of how foldablehelper uses windowmanager library to retrieve information about the folded state of the device flat for normal mode and half-opened for flex mode , window size, and orientation of the fold on the screen download the foldablehelper java file here foldablehelper java 6 22 kb to import the foldablehelper java file and add dependencies to the project, follow the steps below in assets > plugins > android, right-click and select import new asset locate and choose the foldablehelper java file, then click import next, open the gradletemplate properties file to any source code editor like visual studio and add the following lines below the **additional_properties** marker android useandroidx = true android enablejetifier = true useandroidx sets the project to use the appropriate androidx libraries instead of support libraries enablejetifier automatically migrates existing third-party libraries to use androidx by rewriting their binaries lastly, open the maintemplate gradle file and add the dependencies for the artifacts needed for the project **apply_plugins** dependencies { implementation filetree dir 'libs', include ['* jar'] implementation "androidx appcompat appcompat 1 6 1" implementation "androidx core core 1 10 1" implementation "androidx core core-ktx 1 10 1" implementation "androidx window window 1 0 0" implementation "androidx window window-java 1 0 0" implementation "org jetbrains kotlin kotlin-stdlib-jdk8 1 9 0" **deps**} create a new playeractivity to implement flex mode on your applications, you must make necessary changes to the activity since it is impossible to access and change the original unityplayeractivity, you need to create a new playeractivity that inherits from the original to do this create a new file named foldableplayeractivity java and import it into the android folder, same as when you imported the foldablehelper java file to extend the built-in playeractivity from unity, write below code in the foldableplayeractivity java file package com unity3d player; import android os bundle; import com unity3d player unityplayeractivity; import com samsung android gamedev foldable foldablehelper; import com samsung android gamedev foldable foldablehelper windowinfolayoutlistener; import android util log; public class foldableplayeractivity extends unityplayeractivity { @override protected void oncreate bundle savedinstancestate { super oncreate savedinstancestate ; foldablehelper init this ; } @override protected void onstart { super onstart ; foldablehelper start this ; } @override protected void onstop { super onstop ; foldablehelper stop ; } @override protected void onrestart { super onrestart ; foldablehelper init this ; } public void attachunitylistener windowinfolayoutlistener listener { foldablehelper attachnativelistener listener, this ; } } oncreate calls the foldablehelper init to ensure that the windowinfotracker and metrics calculator gets created as soon as possible onstart calls the foldablehelper start since the first windowlayoutinfo doesn't get created until onstart onstop calls the foldablehelper stop to ensure that when the application closes, the listener gets cleaned up onrestart calls foldablehelper init when returning to the app after switching away windowinfotracker must be re-initialized; otherwise, flex mode will no longer update after creating the foldableplayeractivity, ensure that the game uses it open the androidmanifest xml file and change the activity name to the one you've just created <activity android name="com unity3d player foldableplayeractivity" android theme="@style/unitythemeselector"> … </activity> store foldablelayoutinfo data to flexproxy implement a native listener that receives calls from java when the device state changes by following these steps use the androidjavaproxy provided by unity in its jni implementation androidjavaproxy is a class that implements a java interface, so the next thing you need to do is create an interface in the foldablehelper java file public interface windowinfolayoutlistener { void onchanged foldablelayoutinfo layoutinfo ; } this interface replaces the temporary native function therefore, remove the code below from the foldablehelper java file public static native void onlayoutchanged foldablelayoutinfo resultinfo ; then, go to the assets > flex_scripts folder and right-click to create a new c# script called flexproxy cs inside this script, replace the automatically generated class with the flexproxy class inheriting from androidjavaproxy public class flexproxy androidjavaproxy { } in flexproxy class, define the variables needed to store the data from foldablelayoutinfo and use enumerators for the folded state, hinge orientation, and occlusion type for the various bounds, use unity's rectint type also, use a boolean to store whether the data has been updated or not public enum state { undefined, flat, half_opened }; public enum orientation { undefined, horizontal, vertical }; public enum occlusiontype { undefined, none, full }; public state state = state undefined; public orientation orientation = orientation undefined; public occlusiontype occlusiontype = occlusiontype undefined; public rectint foldbounds; public rectint currentmetrics; public rectint maxmetrics; public bool needsupdate = false; next, define what java class the flexproxy is going to implement by using the interface's fully qualified name as below public flexproxy base "com samsung android gamedev foldable foldablehelper$windowinfolayoutlistener" { } com samsung android gamedev foldable is the package name of the foldablehelper java file foldablehelper$windowinfolayoutlistener is the class and interface name separated by a $ after linking the proxy to the java interface, create a helper method to simplify java to native conversions private rectint converttorectint androidjavaobject rect { if rect != null { var left = rect get<int> "left" ; var top = rect get<int> "top" ; var width = rect call<int> "width" ; var height = rect call<int> "height" ; return new rectint xmin left, ymin top, width width, height height ; } else { return new rectint -1, -1, -1, -1 ; } } this method takes a java rect object and converts it into a unity c# rectint now, use this converttorectint function for the onchanged function to retrieve the information from the java object and store it in the flex proxy class public void onchanged androidjavaobject layoutinfo { foldbounds = converttorectint layoutinfo get<androidjavaobject> "bounds" ; currentmetrics = converttorectint layoutinfo get<androidjavaobject> "currentmetrics" ; maxmetrics = converttorectint layoutinfo get<androidjavaobject> "maxmetrics" ; orientation = orientation layoutinfo get<int> "hingeorientation" + 1 ; state = state layoutinfo get<int> "state" + 1 ; occlusiontype = occlusiontype layoutinfo get<int> "occlusiontype" + 1 ; needsupdate = true; } implement native flex mode this section focuses on creating the flex mode split-screen effect on the game’s ui create a new c# script in the flex_scripts folder called flexmodemanager cs after creating the script, define the variables you need for this implementation public class flexmodemanager monobehaviour { private flexproxy windowmanagerlistener; [serializefield] private camera maincamera; [serializefield] private camera skyboxcamera; [serializefield] private canvas controlscanvas; [serializefield] private canvas healthcanvas; [serializefield] private gameobject flexbg; [serializefield] private gameobject uiblur; windowmanagerlistener is the callback object which receives the foldablelayoutinfo from the foldablehelper java implementation maincamera and skyboxcamera are two cameras to modify in this project for creating a seamless flex mode implementation controlscanvas and healthcanvas are the two ui elements to manipulate for implementing the flex mode flexbg is a background image to disable in normal mode and enable in flex mode to fill the bottom screen uiblur is a background blur element used as part of the tutorial text it doesn't function properly with flex mode, so disabling it when in flex mode makes the ui looks better next, in the start method, create a new instance of the flexproxy class and attach it to the unity application's activity via the attachunitylistener function void start { windowmanagerlistener = new flexproxy ; using androidjavaclass javaclass = new androidjavaclass "com unity3d player unityplayer" { using androidjavaobject activity = javaclass getstatic<androidjavaobject> "currentactivity" { activity call "attachunitylistener", windowmanagerlistener ; } } } in the update method, check if the windowmanagerlistener has received any new data on the folded state of the device if the system needs an update, then call updateflexmode void update { if windowmanagerlistener needsupdate { updateflexmode ; } } create the updateflexmode method to enable or disable flex mode notethis project is set up for landscape mode only the implementation discussed in this code lab only covers setting up flex mode with a horizontal fold in landscape however, if you were able to follow, it should be simple to set up something similar for different fold orientations on devices such as the galaxy z flip series of devices private void updateflexmode { } check the folded state of the device via the windowmanagerlistener if the device is half_opened, implement flex mode private void updateflexmode { if windowmanagerlistener state == flexproxy state half_opened { } } to split the ui screen horizontally, set the anchor points of the controlscanvas and the healthcanvas so they are locked at the bottom screen or below the fold also, set the viewports of the maincamera and skyboxcamera to be above the fold - which is the top screen next, set the anchors for the flexbg object and enable it to fill the space behind the ui on the bottom screen deactivate the uiblur element if it exists the ui blur element is only present at level 1 of the demo game a check is necessary to ensure the flex mode manager works on the second level private void updateflexmode { if windowmanagerlistener state == flexproxy state half_opened { float lowerscreenanchormax = float windowmanagerlistener foldbounds ymin / windowmanagerlistener currentmetrics height; recttransform controlscanvastransform = controlscanvas getcomponent<recttransform> ; recttransform healthcanvastransform = healthcanvas getcomponent<recttransform> ; recttransform flexbgtransform = flexbg getcomponent<recttransform> ; controlscanvastransform anchormin = new vector2 0, 0 ; controlscanvastransform anchormax = new vector2 1, lowerscreenanchormax ; healthcanvastransform anchormin = new vector2 0, lowerscreenanchormax ; healthcanvastransform anchormax = new vector2 0, lowerscreenanchormax ; flexbgtransform anchormin = new vector2 0, 0 ; flexbgtransform anchormax = new vector2 1, lowerscreenanchormax ; float upperscreenrectheight = float windowmanagerlistener foldbounds ymax / windowmanagerlistener currentmetrics height; maincamera rect = new rect 0, upperscreenrectheight, 1, upperscreenrectheight ; skyboxcamera rect = new rect 0, upperscreenrectheight, 1, upperscreenrectheight ; flexbg setactive true ; if uiblur != null uiblur setactive false ; } } return the ui to full screen when the device is no longer in flex mode by disabling flexbg; enabling uiblur when it exists; and setting all the anchor points and viewports back to their original values and finally, inform the windowmanagerlistener that it doesn't need an update else { recttransform controlscanvastransform = controlscanvas getcomponent<recttransform> ; recttransform healthcanvastransform = healthcanvas getcomponent<recttransform> ; controlscanvastransform anchormin = new vector2 0, 0 ; controlscanvastransform anchormax = new vector2 1, 1 ; healthcanvastransform anchormin = new vector2 0, 1 ; healthcanvastransform anchormax = new vector2 0, 1 ; maincamera rect = new rect 0, 0, 1, 1 ; skyboxcamera rect = new rect 0, 0, 1, 1 ; flexbg setactive false ; if uiblur != null uiblur setactive true ; } windowmanagerlistener needsupdate = false; set up the scenes for flex mode go back to the unity editor in assets > 3dgamekit > scenes > gameplay, double-click on the level 1 scene to open it right-click in the hierarchy window and select create empty name the new gameobject as flexmanager or a similar name to reflect its purpose select the flexmanager object and click the add component button in the inspector window type in flexmodemanager, and select the script when it shows up select the relevant objects for each camera, canvas and game object as below do the same for level 2 but leave the ui blur empty build and run the app go to file > build settings click build at the bottom of the window to build the apk after building the apk, run the game app on a foldable galaxy device and see how the ui switches from normal to flex mode if you don’t have any physical device, you can also test it on a remote test lab device tipwatch this tutorial video and know how to easily test your app via remote test lab you're done! congratulations! you have successfully achieved the goal of this code lab now, you can implement flex mode in your unity game app by yourself! to learn more, visit www developer samsung com/galaxy-z www developer samsung com/galaxy-gamedev