Implement Flex Mode into a Unity game
Objective
Learn how to implement Flex mode into a Unity game using Android Jetpack WindowManager and Unity's Java Native Interface (JNI) Wrapper.
Overview
The flexible hinge and glass display on Galaxy foldable devices, such as the Galaxy Z Fold4 and Galaxy Z Flip4, let the phone remains propped open while you use apps. When the phone is partially folded, it will go into Flex mode.
Apps will reorient to fit the screen, letting you watch videos or play games without holding the phone. For example, you can set the device on a flat surface, like on a table, and use the bottom half of the screen to navigate. Unfold the phone to use the apps in full screen mode, and partially fold it again to return to Flex mode. To provide users with a convenient and versatile foldable experience, developers need to optimize their apps to meet the Flex mode standard.
Set up your environment
You will need the following:
-
Unity Hub with Unity 2022.3.5f1 or later (must have Android Build Support)
-
Visual Studio or any source code editor
-
Galaxy Z Fold2 or newer
-
Remote Test Lab (if physical device is not available)
Requirements:
- Samsung account
- Java Runtime Environment (JRE) 7 or later with Java Web Start
- Internet environment where port 2600 is available
Sample Code
Here is a sample project for you to start coding in this Code Lab. Download it and start your learning experience!
Start your project
After downloading the sample project files, follow the steps below to open your project:
- Launch the Unity Hub.
- Click Projects > Open.
- Locate the unzipped project folder and click Open to add the project to the Hub and open in the Editor.
Configure Android Player Settings
To ensure that the project runs smoothly on the Android platform, configure the Player Settings as follows:
- Go to File > Build Settings.
- Under Platform, choose Android and click Switch Platform. Wait until this action finishes importing necessary assets and compiling scripts.
- Then, click Player Settings to open the Project Settings window.
- Go to Player > Other Settings and scroll down to see Target API Level. Set it to API level 33 as any less than this will result in a dependency error regarding an
LStar
variable. - You can set the Minimum API Level on lower levels without any problem.
- Next, in the Resolution and Presentation settings, enable Resizable Window. It is also recommended that Render outside safe area is enabled to prevent black bars on the edges of the screen.
- Lastly, enable the Custom Main Manifest, Custom Main Gradle Template, and Custom Gradle Properties Template in the Publishing Settings.
- After closing the Project Settings window, check for the new folder structure created within your Assets in the Project window. The newly created Android folder contains
AndroidManifest.xml
,gradleTemplate.properties
, andmainTemplate.gradle
files.
Import the FoldableHelper and add dependencies
FoldableHelper is a Java file that you can use in different projects. It provides an interface to the Android Jetpack WindowManager library, enabling application developers to support new device form factors and multi-window environments.
Before proceeding, read How to Use Jetpack WindowManager in Android Game Dev and learn the details of how FoldableHelper
uses WindowManager
library to retrieve information about the folded state of the device (FLAT
for Normal mode and HALF-OPENED
for Flex mode), window size, and orientation of the fold on the screen.
Download the FoldableHelper.java
file here:
To import the FoldableHelper.java
file and add dependencies to the project, follow the steps below:
- In Assets > Plugins > Android, right-click and select Import New Asset.
- Locate and choose the
FoldableHelper.java
file, then click Import.
-
Next, open the
gradleTemplate.properties
file to any source code editor like Visual Studio and add the following lines below the**ADDITIONAL_PROPERTIES**
marker.android.useAndroidX = true android.enableJetifier = true
useAndroidX
sets the project to use the appropriate AndroidX libraries instead of support libraries.enableJetifier
automatically migrates existing third-party libraries to use AndroidX by rewriting their binaries.
-
Lastly, open the
mainTemplate.gradle
file and add the dependencies for the artifacts needed for the project.**APPLY_PLUGINS** dependencies { implementation fileTree(dir: 'libs', include: ['*.jar']) implementation "androidx.appcompat:appcompat:1.6.1" implementation "androidx.core:core:1.10.1" implementation "androidx.core:core-ktx:1.10.1" implementation "androidx.window:window:1.0.0" implementation "androidx.window:window-java:1.0.0" implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.0" **DEPS**}
Create a new PlayerActivity
To implement Flex mode on your applications, you must make necessary changes to the Activity
. Since it is impossible to access and change the original UnityPlayerActivity
, you need to create a new PlayerActivity
that inherits from the original. To do this:
-
Create a new file named
FoldablePlayerActivity.java
and import it into the Android folder, same as when you imported theFoldableHelper.java
file.
-
To extend the built-in
PlayerActivity
from Unity, write below code in theFoldablePlayerActivity.java
file.package com.unity3d.player; import android.os.Bundle; import com.unity3d.player.UnityPlayerActivity; import com.samsung.android.gamedev.foldable.FoldableHelper; import com.samsung.android.gamedev.foldable.FoldableHelper.WindowInfoLayoutListener; import android.util.Log; public class FoldablePlayerActivity extends UnityPlayerActivity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); FoldableHelper.init(this); } @Override protected void onStart() { super.onStart(); FoldableHelper.start(this); } @Override protected void onStop() { super.onStop(); FoldableHelper.stop(); } @Override protected void onRestart() { super.onRestart(); FoldableHelper.init(this); } public void attachUnityListener(WindowInfoLayoutListener listener){ FoldableHelper.attachNativeListener(listener, this); } }
onCreate()
calls theFoldableHelper.init()
to ensure that theWindowInfoTracker
and Metrics Calculator gets created as soon as possible.onStart()
calls theFoldableHelper.start()
since the firstWindowLayoutInfo
doesn't get created untilonStart()
.onStop()
calls theFoldableHelper.stop()
to ensure that when the application closes, the listener gets cleaned up.onRestart()
callsFoldableHelper.init()
when returning to the app after switching away.WindowInfoTracker
must be re-initialized; otherwise, Flex Mode will no longer update.
-
After creating the
FoldablePlayerActivity
, ensure that the game uses it. Open theAndroidManifest.xml
file and change theActivity
name to the one you've just created.<activity android:name="com.unity3d.player.FoldablePlayerActivity" android:theme="@style/UnityThemeSelector"> … </activity>
Store FoldableLayoutInfo data to FlexProxy
Implement a native listener that receives calls from Java when the device state changes by following these steps:
-
Use the AndroidJavaProxy provided by Unity in its JNI implementation.
AndroidJavaProxy
is a class that implements a Java interface, so the next thing you need to do is create an interface in theFoldableHelper.java
file.public interface WindowInfoLayoutListener { void onChanged(FoldableLayoutInfo LayoutInfo); }
-
This interface replaces the temporary native function. Therefore, remove the code below from the
FoldableHelper.java
file:public static native void OnLayoutChanged(FoldableLayoutInfo resultInfo);
-
Then, go to the Assets > Flex_Scripts folder and right-click to create a new C# script called FlexProxy.cs.
-
Inside this script, replace the automatically generated class with the
FlexProxy
class inheriting fromAndroidJavaProxy
:public class FlexProxy : AndroidJavaProxy { }
-
In
FlexProxy
class, define the variables needed to store the data from FoldableLayoutInfo and use enumerators for the folded state, hinge orientation, and occlusion type. For the various bounds, use Unity's RectInt type. Also, use a boolean to store whether the data has been updated or not.public enum State { UNDEFINED, FLAT, HALF_OPENED }; public enum Orientation { UNDEFINED, HORIZONTAL, VERTICAL }; public enum OcclusionType { UNDEFINED, NONE, FULL }; public State state = State.UNDEFINED; public Orientation orientation = Orientation.UNDEFINED; public OcclusionType occlusionType = OcclusionType.UNDEFINED; public RectInt foldBounds; public RectInt currentMetrics; public RectInt maxMetrics; public bool needsUpdate = false;
-
Next, define what Java class the
FlexProxy
is going to implement by using the interface's fully qualified name as below:public FlexProxy() : base("com.samsung.android.gamedev.foldable.FoldableHelper$WindowInfoLayoutListener") { }
com.samsung.android.gamedev.foldable
is the package name of theFoldableHelper.java
file.FoldableHelper$WindowInfoLayoutListener
is the class and interface name separated by a$
.
-
After linking the proxy to the Java interface, create a helper method to simplify Java to native conversions.
private RectInt ConvertToRectInt(AndroidJavaObject rect) { if(rect != null) { var left = rect.Get<int>("left"); var top = rect.Get<int>("top"); var width = rect.Call<int>("width"); var height = rect.Call<int>("height"); return new RectInt(xMin: left, yMin: top, width: width, height: height); } else { return new RectInt(-1, -1, -1, -1); } }
This method takes a Java Rect object and converts it into a Unity C#
RectInt
.
-
Now, use this
ConvertToRectInt()
function for theonChanged()
function to retrieve the information from the Java Object and store it in the Flex Proxy Class:public void onChanged(AndroidJavaObject LayoutInfo) { foldBounds = ConvertToRectInt(LayoutInfo.Get<AndroidJavaObject>("bounds")); currentMetrics = ConvertToRectInt(LayoutInfo.Get<AndroidJavaObject>("currentMetrics")); maxMetrics = ConvertToRectInt(LayoutInfo.Get<AndroidJavaObject>("maxMetrics")); orientation = (Orientation)(LayoutInfo.Get<int>("hingeOrientation") + 1); state = (State)(LayoutInfo.Get<int>("state") + 1); occlusionType = (OcclusionType)(LayoutInfo.Get<int>("occlusionType") + 1); needsUpdate = true; }
Implement native Flex mode
This section focuses on creating the Flex mode split-screen effect on the game’s UI.
-
Create a new C# script in the Flex_Scripts folder called
FlexModeManager.cs
.
-
After creating the script, define the variables you need for this implementation.
public class FlexModeManager : MonoBehaviour { private FlexProxy windowManagerListener; [SerializeField] private Camera mainCamera; [SerializeField] private Camera skyBoxCamera; [SerializeField] private Canvas controlsCanvas; [SerializeField] private Canvas healthCanvas; [SerializeField] private GameObject flexBG; [SerializeField] private GameObject UIBlur;
windowManagerListener
is the callback object which receives theFoldableLayoutInfo
from theFoldableHelper.java
implementationmainCamera
andskyBoxCamera
are two cameras to modify in this project for creating a seamless Flex mode implementationcontrolsCanvas
andhealthCanvas
are the two UI elements to manipulate for implementing the Flex modeflexBG
is a background image to disable in Normal mode and enable in Flex mode to fill the bottom screenUIBlur
is a background blur element used as part of the tutorial text. It doesn't function properly with Flex mode, so disabling it when in Flex mode makes the UI looks better.
-
Next, in the
Start()
method, create a new instance of theFlexProxy
class and attach it to the Unity application's activity via theattachUnityListener
function.void Start() { windowManagerListener = new FlexProxy(); using (AndroidJavaClass javaClass = new AndroidJavaClass("com.unity3d.player.UnityPlayer")) { using (AndroidJavaObject activity = javaClass.GetStatic<AndroidJavaObject>("currentActivity")) { activity.Call("attachUnityListener", windowManagerListener); } } }
-
In the
Update()
method, check if thewindowManagerListener
has received any new data on the folded state of the device. -
If the system needs an update, then call
UpdateFlexMode()
.void Update() { if (windowManagerListener.needsUpdate) { UpdateFlexMode(); } }
-
Create the
UpdateFlexMode()
method to enable or disable Flex mode.NoteThis project is set up for landscape mode only. The implementation discussed in this Code Lab only covers setting up flex mode with a horizontal fold in landscape. However, if you were able to follow, it should be simple to set up something similar for different fold orientations on devices such as the Galaxy Z Flip series of devices.private void UpdateFlexMode() { }
-
Check the folded state of the device via the
windowManagerListener
. If the device isHALF_OPENED
, implement Flex mode.private void UpdateFlexMode() { if (windowManagerListener.state == FlexProxy.State.HALF_OPENED) { } }
-
To split the UI screen horizontally, set the anchor points of the
controlsCanvas
and thehealthCanvas
so they are locked at the bottom screen or below the fold. Also, set the viewports of themainCamera
andskyBoxCamera
to be above the fold - which is the top screen. -
Next, set the anchors for the
flexBG
object and enable it to fill the space behind the UI on the bottom screen. -
Deactivate the
UIBlur
element if it exists. The UI Blur element is only present at Level 1 of the demo game. A check is necessary to ensure the Flex Mode Manager works on the second level.private void UpdateFlexMode() { if (windowManagerListener.state == FlexProxy.State.HALF_OPENED) { float lowerScreenAnchorMax = (float)windowManagerListener.foldBounds.yMin / windowManagerListener.currentMetrics.height; RectTransform controlsCanvasTransform = controlsCanvas.GetComponent<RectTransform>(); RectTransform healthCanvasTransform = healthCanvas.GetComponent<RectTransform>(); RectTransform flexBGTransform = flexBG.GetComponent<RectTransform>(); controlsCanvasTransform.anchorMin = new Vector2(0, 0); controlsCanvasTransform.anchorMax = new Vector2(1, lowerScreenAnchorMax); healthCanvasTransform.anchorMin = new Vector2(0, lowerScreenAnchorMax); healthCanvasTransform.anchorMax = new Vector2(0, lowerScreenAnchorMax); flexBGTransform.anchorMin = new Vector2(0, 0); flexBGTransform.anchorMax = new Vector2(1, lowerScreenAnchorMax); float upperScreenRectHeight = (float)windowManagerListener.foldBounds.yMax / windowManagerListener.currentMetrics.height; mainCamera.rect = new Rect(0, upperScreenRectHeight, 1, upperScreenRectHeight); skyBoxCamera.rect = new Rect(0, upperScreenRectHeight, 1, upperScreenRectHeight); flexBG.SetActive(true); if(UIBlur != null) UIBlur.SetActive(false); } }
-
Return the UI to full screen when the device is no longer in Flex mode by:
- disabling
flexBG
; - enabling
UIBlur
when it exists; and - setting all the anchor points and viewports back to their original values.
- disabling
-
And finally, inform the
windowManagerListener
that it doesn't need an update.else { RectTransform controlsCanvasTransform = controlsCanvas.GetComponent<RectTransform>(); RectTransform healthCanvasTransform = healthCanvas.GetComponent<RectTransform>(); controlsCanvasTransform.anchorMin = new Vector2(0, 0); controlsCanvasTransform.anchorMax = new Vector2(1, 1); healthCanvasTransform.anchorMin = new Vector2(0, 1); healthCanvasTransform.anchorMax = new Vector2(0, 1); mainCamera.rect = new Rect(0, 0, 1, 1); skyBoxCamera.rect = new Rect(0, 0, 1, 1); flexBG.SetActive(false); if (UIBlur != null) UIBlur.SetActive(true); } windowManagerListener.needsUpdate = false;
Set up the scenes for Flex Mode
- Go back to the Unity Editor. In Assets > 3DGameKit > Scenes > Gameplay, double-click on the Level 1 scene to open it.
- Right-click in the Hierarchy window and select Create Empty.
- Name the new GameObject as FlexManager or a similar name to reflect its purpose.
- Select the
FlexManager
object and click the Add Component button in the Inspector window. Type inFlexModeManager
, and select the script when it shows up.
- Select the relevant objects for each camera, canvas and game object as below:
- Do the same for Level 2 but leave the UI Blur empty.
Build and run the app
Go to File > Build Settings. Click Build at the bottom of the window to build the APK.
After building the APK, run the game app on a foldable Galaxy device and see how the UI switches from Normal to Flex mode. If you don’t have any physical device, you can also test it on a Remote Test Lab device.
You're done!
Congratulations! You have successfully achieved the goal of this Code Lab. Now, you can implement Flex mode in your Unity game app by yourself!
To learn more, visit:
www.developer.samsung.com/galaxy-z
www.developer.samsung.com/galaxy-gamedev