Implement Flex Mode into a Unity game


Objective

Learn how to implement Flex mode into a Unity game using Android Jetpack WindowManager and Unity's Java Native Interface (JNI) Wrapper.

Overview

The flexible hinge and glass display on Galaxy foldable devices, such as the Galaxy Z Fold4 and Galaxy Z Flip4, let the phone remains propped open while you use apps. When the phone is partially folded, it will go into Flex mode.

Apps will reorient to fit the screen, letting you watch videos or play games without holding the phone. For example, you can set the device on a flat surface, like on a table, and use the bottom half of the screen to navigate. Unfold the phone to use the apps in full screen mode, and partially fold it again to return to Flex mode. To provide users with a convenient and versatile foldable experience, developers need to optimize their apps to meet the Flex mode standard.

Set up your environment

You will need the following:

  • Unity Hub with Unity 2022.3.5f1 or later (must have Android Build Support)

  • Visual Studio or any source code editor

  • Galaxy Z Fold2 or newer

  • Remote Test Lab (if physical device is not available)

    Requirements:

    • Samsung account
    • Java Runtime Environment (JRE) 7 or later with Java Web Start
    • Internet environment where port 2600 is available

Sample Code

Here is a sample project for you to start coding in this Code Lab. Download it and start your learning experience!

Flex Mode on Unity Sample Code
(1.17 GB)

Start your project

After downloading the sample project files, follow the steps below to open your project:

  1. Launch the Unity Hub.
  2. Click Projects > Open.


  1. Locate the unzipped project folder and click Open to add the project to the Hub and open in the Editor.

Configure Android Player Settings

To ensure that the project runs smoothly on the Android platform, configure the Player Settings as follows:

  1. Go to File > Build Settings.
  2. Under Platform, choose Android and click Switch Platform. Wait until this action finishes importing necessary assets and compiling scripts.
  3. Then, click Player Settings to open the Project Settings window.
  4. Go to Player > Other Settings and scroll down to see Target API Level. Set it to API level 33 as any less than this will result in a dependency error regarding an LStar variable.
  5. You can set the Minimum API Level on lower levels without any problem.


  1. Next, in the Resolution and Presentation settings, enable Resizable Window. It is also recommended that Render outside safe area is enabled to prevent black bars on the edges of the screen.


  1. Lastly, enable the Custom Main Manifest, Custom Main Gradle Template, and Custom Gradle Properties Template in the Publishing Settings.


  1. After closing the Project Settings window, check for the new folder structure created within your Assets in the Project window. The newly created Android folder contains AndroidManifest.xml, gradleTemplate.properties, and mainTemplate.gradle files.

Import the FoldableHelper and add dependencies

FoldableHelper is a Java file that you can use in different projects. It provides an interface to the Android Jetpack WindowManager library, enabling application developers to support new device form factors and multi-window environments.

Before proceeding, read How to Use Jetpack WindowManager in Android Game Dev and learn the details of how FoldableHelper uses WindowManager library to retrieve information about the folded state of the device (FLAT for Normal mode and HALF-OPENED for Flex mode), window size, and orientation of the fold on the screen.

Download the FoldableHelper.java file here:

FoldableHelper.java
(6.22 KB)

To import the FoldableHelper.java file and add dependencies to the project, follow the steps below:

  1. In Assets > Plugins > Android, right-click and select Import New Asset.
  2. Locate and choose the FoldableHelper.java file, then click Import.


  1. Next, open the gradleTemplate.properties file to any source code editor like Visual Studio and add the following lines below the **ADDITIONAL_PROPERTIES** marker.

    android.useAndroidX = true
android.enableJetifier = true
    • useAndroidX sets the project to use the appropriate AndroidX libraries instead of support libraries.
    • enableJetifier automatically migrates existing third-party libraries to use AndroidX by rewriting their binaries.

  1. Lastly, open the mainTemplate.gradle file and add the dependencies for the artifacts needed for the project.

    **APPLY_PLUGINS**

dependencies {
    implementation fileTree(dir: 'libs', include: ['*.jar'])
    implementation "androidx.appcompat:appcompat:1.6.1"
    implementation "androidx.core:core:1.10.1"
    implementation "androidx.core:core-ktx:1.10.1"
    implementation "androidx.window:window:1.0.0"
    implementation "androidx.window:window-java:1.0.0"
    implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.0"
**DEPS**}

Create a new PlayerActivity

To implement Flex mode on your applications, you must make necessary changes to the Activity. Since it is impossible to access and change the original UnityPlayerActivity, you need to create a new PlayerActivity that inherits from the original. To do this:

  1. Create a new file named FoldablePlayerActivity.java and import it into the Android folder, same as when you imported the FoldableHelper.java file.


  1. To extend the built-in PlayerActivity from Unity, write below code in the FoldablePlayerActivity.java file.

    package com.unity3d.player;
import android.os.Bundle;
import com.unity3d.player.UnityPlayerActivity;
import com.samsung.android.gamedev.foldable.FoldableHelper;
import com.samsung.android.gamedev.foldable.FoldableHelper.WindowInfoLayoutListener;
import android.util.Log;
public class FoldablePlayerActivity extends UnityPlayerActivity
{

	@Override protected void onCreate(Bundle savedInstanceState)
	{
		super.onCreate(savedInstanceState);
		FoldableHelper.init(this);
	}
	@Override protected void onStart()
	{
		super.onStart();
		FoldableHelper.start(this);
	}
	@Override protected void onStop()
	{
		super.onStop();
		FoldableHelper.stop();
	}
	@Override protected void onRestart()
	{
		super.onRestart();
		FoldableHelper.init(this);
	}
	public void attachUnityListener(WindowInfoLayoutListener listener){
		FoldableHelper.attachNativeListener(listener, this);
	}

}
    • onCreate() calls the FoldableHelper.init() to ensure that the WindowInfoTracker and Metrics Calculator gets created as soon as possible.
    • onStart() calls the FoldableHelper.start() since the first WindowLayoutInfo doesn't get created until onStart().
    • onStop() calls the FoldableHelper.stop() to ensure that when the application closes, the listener gets cleaned up.
    • onRestart() calls FoldableHelper.init() when returning to the app after switching away. WindowInfoTracker must be re-initialized; otherwise, Flex Mode will no longer update.

  1. After creating the FoldablePlayerActivity, ensure that the game uses it. Open the AndroidManifest.xml file and change the Activity name to the one you've just created.

    <activity android:name="com.unity3d.player.FoldablePlayerActivity"
	  android:theme="@style/UnityThemeSelector">
…
</activity>

Store FoldableLayoutInfo data to FlexProxy

Implement a native listener that receives calls from Java when the device state changes by following these steps:

  1. Use the AndroidJavaProxy provided by Unity in its JNI implementation. AndroidJavaProxy is a class that implements a Java interface, so the next thing you need to do is create an interface in the FoldableHelper.java file.

    public interface WindowInfoLayoutListener {
	void onChanged(FoldableLayoutInfo LayoutInfo);
}

  1. This interface replaces the temporary native function. Therefore, remove the code below from the FoldableHelper.java file:

    public static native void OnLayoutChanged(FoldableLayoutInfo resultInfo);

  1. Then, go to the Assets > Flex_Scripts folder and right-click to create a new C# script called FlexProxy.cs.


  1. Inside this script, replace the automatically generated class with the FlexProxy class inheriting from AndroidJavaProxy:

    public class FlexProxy : AndroidJavaProxy
{
}

  1. In FlexProxy class, define the variables needed to store the data from FoldableLayoutInfo and use enumerators for the folded state, hinge orientation, and occlusion type. For the various bounds, use Unity's RectInt type. Also, use a boolean to store whether the data has been updated or not.

    public enum State { UNDEFINED, FLAT, HALF_OPENED };
public enum Orientation { UNDEFINED, HORIZONTAL, VERTICAL };
public enum OcclusionType { UNDEFINED, NONE, FULL };

public State state = State.UNDEFINED;
public Orientation orientation = Orientation.UNDEFINED;
public OcclusionType occlusionType = OcclusionType.UNDEFINED;
public RectInt foldBounds;
public RectInt currentMetrics;
public RectInt maxMetrics;
public bool needsUpdate = false;

  1. Next, define what Java class the FlexProxy is going to implement by using the interface's fully qualified name as below:

    public FlexProxy() : base("com.samsung.android.gamedev.foldable.FoldableHelper$WindowInfoLayoutListener") { }
    • com.samsung.android.gamedev.foldable is the package name of the FoldableHelper.java file.
    • FoldableHelper$WindowInfoLayoutListener is the class and interface name separated by a $.

  1. After linking the proxy to the Java interface, create a helper method to simplify Java to native conversions.

    private RectInt ConvertToRectInt(AndroidJavaObject rect)
{
	if(rect != null)
	{
		var left = rect.Get<int>("left");
		var top = rect.Get<int>("top");
		var width = rect.Call<int>("width");
		var height = rect.Call<int>("height");
		return new RectInt(xMin: left, yMin: top, width: width, height: height);
	}
	else
	{
		return new RectInt(-1, -1, -1, -1);
	}
}

    This method takes a Java Rect object and converts it into a Unity C# RectInt.


  1. Now, use this ConvertToRectInt() function for the onChanged() function to retrieve the information from the Java Object and store it in the Flex Proxy Class:

    public void onChanged(AndroidJavaObject LayoutInfo)
{
	foldBounds = ConvertToRectInt(LayoutInfo.Get<AndroidJavaObject>("bounds"));
	currentMetrics = ConvertToRectInt(LayoutInfo.Get<AndroidJavaObject>("currentMetrics"));
	maxMetrics = ConvertToRectInt(LayoutInfo.Get<AndroidJavaObject>("maxMetrics"));
	orientation = (Orientation)(LayoutInfo.Get<int>("hingeOrientation") + 1);
	state = (State)(LayoutInfo.Get<int>("state") + 1);
	occlusionType = (OcclusionType)(LayoutInfo.Get<int>("occlusionType") + 1);
	needsUpdate = true;
}

Implement native Flex mode

This section focuses on creating the Flex mode split-screen effect on the game’s UI.

  1. Create a new C# script in the Flex_Scripts folder called FlexModeManager.cs.

  1. After creating the script, define the variables you need for this implementation.

    public class FlexModeManager : MonoBehaviour
{
    private FlexProxy windowManagerListener;

    [SerializeField]
    private Camera mainCamera;
    [SerializeField]
    private Camera skyBoxCamera;

    [SerializeField]
    private Canvas controlsCanvas;

    [SerializeField]
    private Canvas healthCanvas;
    [SerializeField]
    private GameObject flexBG;

    [SerializeField]
    private GameObject UIBlur;
    • windowManagerListener is the callback object which receives the FoldableLayoutInfo from the FoldableHelper.java implementation
    • mainCamera and skyBoxCamera are two cameras to modify in this project for creating a seamless Flex mode implementation
    • controlsCanvas and healthCanvas are the two UI elements to manipulate for implementing the Flex mode
    • flexBG is a background image to disable in Normal mode and enable in Flex mode to fill the bottom screen
    • UIBlur is a background blur element used as part of the tutorial text. It doesn't function properly with Flex mode, so disabling it when in Flex mode makes the UI looks better.

  2. Next, in the Start() method, create a new instance of the FlexProxy class and attach it to the Unity application's activity via the attachUnityListener function.

    void Start()
{
    windowManagerListener = new FlexProxy();
    using (AndroidJavaClass javaClass = new AndroidJavaClass("com.unity3d.player.UnityPlayer"))
    {
	using (AndroidJavaObject activity = javaClass.GetStatic<AndroidJavaObject>("currentActivity"))
	{
	    activity.Call("attachUnityListener", windowManagerListener);
	}
    }
}

  1. In the Update() method, check if the windowManagerListener has received any new data on the folded state of the device.

  2. If the system needs an update, then call UpdateFlexMode().

    void Update()
{
    if (windowManagerListener.needsUpdate)
    {
	UpdateFlexMode();
    }
}

  1. Create the UpdateFlexMode() method to enable or disable Flex mode.

    private void UpdateFlexMode()
{


}

  1. Check the folded state of the device via the windowManagerListener. If the device is HALF_OPENED, implement Flex mode.

    private void UpdateFlexMode()
{
    if (windowManagerListener.state == FlexProxy.State.HALF_OPENED)
    {


    }
}

  1. To split the UI screen horizontally, set the anchor points of the controlsCanvas and the healthCanvas so they are locked at the bottom screen or below the fold. Also, set the viewports of the mainCamera and skyBoxCamera to be above the fold - which is the top screen.

  2. Next, set the anchors for the flexBG object and enable it to fill the space behind the UI on the bottom screen.

  3. Deactivate the UIBlur element if it exists. The UI Blur element is only present at Level 1 of the demo game. A check is necessary to ensure the Flex Mode Manager works on the second level.

    private void UpdateFlexMode()
{
    if (windowManagerListener.state == FlexProxy.State.HALF_OPENED)
    {
float lowerScreenAnchorMax = (float)windowManagerListener.foldBounds.yMin / windowManagerListener.currentMetrics.height;

RectTransform controlsCanvasTransform = controlsCanvas.GetComponent<RectTransform>();
RectTransform healthCanvasTransform =   healthCanvas.GetComponent<RectTransform>();
RectTransform flexBGTransform = flexBG.GetComponent<RectTransform>();

controlsCanvasTransform.anchorMin = new Vector2(0, 0);
controlsCanvasTransform.anchorMax = new Vector2(1, lowerScreenAnchorMax);
healthCanvasTransform.anchorMin =   new Vector2(0, lowerScreenAnchorMax);
healthCanvasTransform.anchorMax =   new Vector2(0, lowerScreenAnchorMax);
flexBGTransform.anchorMin = new Vector2(0, 0);
flexBGTransform.anchorMax = new Vector2(1, lowerScreenAnchorMax);

float upperScreenRectHeight = (float)windowManagerListener.foldBounds.yMax / windowManagerListener.currentMetrics.height;

mainCamera.rect =       new Rect(0, upperScreenRectHeight, 1, upperScreenRectHeight);
skyBoxCamera.rect =     new Rect(0, upperScreenRectHeight, 1, upperScreenRectHeight);

flexBG.SetActive(true);

if(UIBlur != null)
	    UIBlur.SetActive(false);

    }
}

  1. Return the UI to full screen when the device is no longer in Flex mode by:

    • disabling flexBG;
    • enabling UIBlur when it exists; and
    • setting all the anchor points and viewports back to their original values.

  2. And finally, inform the windowManagerListener that it doesn't need an update.

    else
{
    RectTransform controlsCanvasTransform = controlsCanvas.GetComponent<RectTransform>();
    RectTransform healthCanvasTransform =   healthCanvas.GetComponent<RectTransform>();

    controlsCanvasTransform.anchorMin = new Vector2(0, 0);
    controlsCanvasTransform.anchorMax = new Vector2(1, 1);
    healthCanvasTransform.anchorMin =   new Vector2(0, 1);
    healthCanvasTransform.anchorMax =   new Vector2(0, 1);

    mainCamera.rect =       new Rect(0, 0, 1, 1);
    skyBoxCamera.rect =     new Rect(0, 0, 1, 1);

    flexBG.SetActive(false);
    if (UIBlur != null)
	UIBlur.SetActive(true);
}
windowManagerListener.needsUpdate = false;

Set up the scenes for Flex Mode

  1. Go back to the Unity Editor. In Assets > 3DGameKit > Scenes > Gameplay, double-click on the Level 1 scene to open it.

  1. Right-click in the Hierarchy window and select Create Empty.
  2. Name the new GameObject as FlexManager or a similar name to reflect its purpose.

  1. Select the FlexManager object and click the Add Component button in the Inspector window. Type in FlexModeManager, and select the script when it shows up.

  1. Select the relevant objects for each camera, canvas and game object as below:

  1. Do the same for Level 2 but leave the UI Blur empty.

Build and run the app

Go to File > Build Settings. Click Build at the bottom of the window to build the APK.

After building the APK, run the game app on a foldable Galaxy device and see how the UI switches from Normal to Flex mode. If you don’t have any physical device, you can also test it on a Remote Test Lab device.

You're done!

Congratulations! You have successfully achieved the goal of this Code Lab. Now, you can implement Flex mode in your Unity game app by yourself!


To learn more, visit:
www.developer.samsung.com/galaxy-z
www.developer.samsung.com/galaxy-gamedev