Implement Flex Mode on a Unity Game

Objective

Learn how to implement Flex mode on a Unity game using Android Jetpack WindowManager and Unity's Java Native Interface (JNI) Wrapper.

Overview

The flexible hinge and glass display on Galaxy foldable devices, such as the Galaxy Z Fold4 and Galaxy Z Flip4, let the phone remains propped open while you use apps. When the phone is partially folded, it will go into Flex mode.

Apps will reorient to fit the screen, letting you watch videos or play games without holding the phone. For example, you can set the device on a flat surface, like on a table, and use the bottom half of the screen to navigate. Unfold the phone to use the apps in full screen mode, and partially fold it again to return to Flex mode. To provide users with a convenient and versatile foldable experience, developers need to optimize their apps to meet the Flex mode standard.

Set up your environment

You will need the following:

Unity Hub with Unity 2020.3.31f1 or later (must have Android Build Support)
Visual Studio or any source code editor
Samsung Galaxy Foldable device:
- Galaxy Z Fold2, Z Fold3, or newer
- Galaxy Z Flip, Z Flip3, or newer
Remote Test Lab (if physical device is not available)

Requirements:
- Samsung account
- Java Runtime Environment (JRE) 7 or later with Java Web Start
- Internet environment where port 2600 is available

Sample Code

Here is a sample project for you to start coding in this Code Lab. Download it and start your learning experience!

Flex Mode on Unity Sample Code

(43.39 KB)

Start your project

After downloading the sample project files, follow the steps below to open your project:

Launch the Unity Hub.
Click Projects > Open.

Locate the unzipped project folder and click Open to add the project to the Hub and open in the Editor.

NoteThe sample project was created in Unity 2020.3.31f1. If you prefer using a different Unity version, click Choose another Editor version when prompted and select a higher version of Unity.

Configure Android Player Settings

To ensure that the project runs smoothly on the Android platform, configure the Player Settings as follows:

Go to File > Build Settings.
Under Platform, choose Android and click Switch Platform. Wait until this action finishes importing necessary assets and compiling scripts.
Then, click Player Settings to open the Project Settings window.
Go to Player > Other Settings and scroll down to see Target API Level. Set it to API level 31 as any less than this will result in a dependency error regarding an LStar variable.
You can set the Minimum API Level on lower levels without any problem.

Next, in the Resolution and Presentation settings, enable Resizable Window. It is also recommended that Render outside safe area is enabled to prevent black bars on the edges of the screen.

Lastly, enable the Custom Main Manifest, Custom Main Gradle Template, and Custom Gradle Properties Template in the Publishing Settings.

After closing the Project Settings window, check for the new folder structure created within your Assets in the Project window. The newly created Android folder contains AndroidManifest.xml, gradleTemplate.properties, and mainTemplate.gradle files.

Import the FoldableHelper and add dependencies

FoldableHelper is a Java file that you can use in different projects. It provides an interface to the Android Jetpack WindowManager library, enabling application developers to support new device form factors and multi-window environments.

Before proceeding, read How to Use Jetpack WindowManager in Android Game Dev and learn the details of how FoldableHelper uses WindowManager library to retrieve information about the folded state of the device (FLAT for Normal mode and HALF-OPENED for Flex mode), window size, and orientation of the fold on the screen.

Download the FoldableHelper.java file here:

FoldableHelper.java

(171.80 KB)

To import the FoldableHelper.java file and add dependencies to the project, follow the steps below:

In Assets > Plugins > Android, right-click and select Import New Asset.
Locate and choose the FoldableHelper.java file, then click Import.

Next, open the gradleTemplate.properties file to any source code editor like Visual Studio and add the following lines below the **ADDITIONAL_PROPERTIES** marker.
```
android.useAndroidX = true
android.enableJetifier = true
```
- useAndroidX sets the project to use the appropriate AndroidX libraries instead of support libraries.
- enableJetifier automatically migrates existing third-party libraries to use AndroidX by rewriting their binaries.

Lastly, open the mainTemplate.gradle file and add the dependencies for the artifacts needed for the project.

**APPLY_PLUGINS**

dependencies {
    implementation fileTree(dir: 'libs', include: ['*.jar'])
    implementation "androidx.appcompat:appcompat:1.4.1"
    implementation "androidx.core:core:1.7.0"
    implementation "androidx.core:core-ktx:1.7.0"
    implementation "androidx.window:window:1.0.0"
    implementation "androidx.window:window-java:1.0.0"
    implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.6.0"
**DEPS**}

NoteYou may update the version of these dependencies when necessary, but be aware that there might be significant changes.

Create a new PlayerActivity

To implement Flex mode on your applications, you must make necessary changes to the Activity. Since it is impossible to access and change the original UnityPlayerActivity, you need to create a new PlayerActivity that inherits from the original. To do this:

Create a new file named FoldablePlayerActivity.java and import it into the Android folder, same as when you imported the FoldableHelper.java file.

To extend the built-in PlayerActivity from Unity, write below code in the FoldablePlayerActivity.java file.

package com.unity3d.player;
import android.os.Bundle;
import com.unity3d.player.UnityPlayerActivity;
import com.samsung.android.gamedev.foldable.FoldableHelper;
import android.util.Log;

public class FoldablePlayerActivity extends UnityPlayerActivity
{

	@Override protected void onCreate(Bundle savedInstanceState)
	{
		super.onCreate(savedInstanceState);
		FoldableHelper.init(this);
	}

	@Override protected void onStart()
	{
		super.onStart();
		FoldableHelper.start(this);
	}

	@Override protected void onStop()
	{
		super.onStop();
		FoldableHelper.stop();
	}

}

onCreate() calls the FoldableHelper.init() to ensure that the WindowInfoTracker and Metrics Calculator gets created as soon as possible.
onStart() calls the FoldableHelper.start() since the first WindowLayoutInfo doesn't get created until onStart().
onStop() calls the FoldableHelper.stop() to ensure that when the application closes, the listener gets cleaned up.

After creating the FoldablePlayerActivity, ensure that the game uses it. Open the AndroidManifest.xml file and change the Activity name to the one you've just created.
```
<activity android:name="FoldablePlayerActivity"
	  android:theme="@style/UnityThemeSelector">
…
</activity>
```

Store FoldableLayoutInfo data to FlexProxy

Implement a native listener that receives calls from Java when the device state changes by following these steps:

Use the AndroidJavaProxy provided by Unity in its JNI implementation. AndroidJavaProxy is a class that implements a Java interface, so the next thing you need to do is create an interface in the FoldableHelper.java file.
```
public interface WindowInfoLayoutListener {
	void onChanged(FoldableLayoutInfo LayoutInfo);
}
```

This interface replaces the temporary native function. Therefore, remove the code below:
```
public static native void OnLayoutChanged(FoldableLayoutInfo resultInfo);
```

Then, go to Assets > Scripts, and right-click to create a new C# script inside called FlexProxy.cs.

Inside this script, create the FlexProxy class inheriting from AndroidJavaProxy.
```
public class FlexProxy : AndroidJavaProxy
{
}
```

In FlexProxy class, define the variables needed to store the data from FoldableLayoutInfo and use enumerators for the folded state, hinge orientation, and occlusion type. For the various bounds, use Unity's RectInt type. Also, use a boolean to store whether the data has been updated or not.

public enum State { UNDEFINED, FLAT, HALF_OPENED };
public enum Orientation { UNDEFINED, HORIZONTAL, VERTICAL };
public enum OcclusionType { UNDEFINED, NONE, FULL };

public State state = State.UNDEFINED;
public Orientation orientation = Orientation.UNDEFINED;
public OcclusionType occlusionType = OcclusionType.UNDEFINED;
public RectInt foldBounds;
public RectInt currentMetrics;
public RectInt maxMetrics;
public bool hasUpdated = false;

Next, define what Java class the FlexProxy is going to implement by using the interface's fully qualified name as below:
```
public FlexProxy() : base("com.samsung.android.gamedev.foldable.FoldableHelper$WindowInfoLayoutListener") { }
```
- com.samsung.android.gamedev.foldable is the package name of the FoldableHelper.java file.
- FoldableHelper$WindowInfoLayoutListener is the class and interface name separated by a $.

After linking the proxy to the Java interface, create a helper method to simplify Java to native conversions.

private RectInt ConvertToRectInt(AndroidJavaObject rect)
{
	if(rect != null)
	{
		var left = rect.Get<int>("left");
		var top = rect.Get<int>("top");
		var width = rect.Call<int>("width");
		var height = rect.Call<int>("height");
		return new RectInt(xMin: left, yMin: top, width: width, height: height);
	}
	else
	{
		return new RectInt(-1, -1, -1, -1);
	}
}

This method takes a Java Rect object and converts it into a Unity C# RectInt.

Now, use this ConvertToRectInt() function for the onChanged() function:

public void onChanged(AndroidJavaObject LayoutInfo)
{
	foldBounds = ConvertToRectInt(LayoutInfo.Get<AndroidJavaObject>("bounds"));
	currentMetrics = ConvertToRectInt(LayoutInfo.Get<AndroidJavaObject>("currentMetrics"));
	maxMetrics = ConvertToRectInt(LayoutInfo.Get<AndroidJavaObject>("maxMetrics"));
	orientation = (Orientation)(LayoutInfo.Get<int>("hingeOrientation") + 1);
	state = (State)(LayoutInfo.Get<int>("state") + 1);
	occlusionType = (OcclusionType)(LayoutInfo.Get<int>("occlusionType") + 1);
	hasUpdated = true;
}

Attach FlexProxy to FoldableHelper

In this step, you need to attach the FlexProxy to the Java implementation. Modify the FoldableHelper.java and FoldablePlayerActivity.java files as follows:

In the FoldableHelper.java file, create a variable in FoldableHelper class where you can store the listener.
```
private static WindowInfoLayoutListener listener = null;
```

Create a method to receive the native Listener.

public static void attachNativeListener(WindowInfoLayoutListener nativeListener){
	listener = nativeListener;
}

To ensure that the listener is actually used, modify the LayoutStateChangeCallback to:
- call the listener.onChanged, the AndroidJavaProxy version of onChanged function, instead of calling the existing onChanged function in the Java file; and
- check if the listener exists at the time of calling.
```
@Override
public void accept(WindowLayoutInfo windowLayoutInfo) {		
	if (listener != null){
		FoldableLayoutInfo resultInfo = updateLayout(windowLayoutInfo, activity);
		listener.onChanged(resultInfo);
	}
}
```

Finally, in the FoldablePlayerActivity.java file, import WindowInfoLayoutListener.

import com.samsung.android.gamedev.foldable.FoldableHelper.WindowInfoLayoutListener;

Then, create a new method in FoldablePlayerActivity to pass the native listener to FoldableHelper.

public void attachUnityListener(WindowInfoLayoutListener listener){
	FoldableHelper.attachNativeListener(listener);
}

Implement native Flex mode

This section focuses on creating the Flex mode split-screen effect on the game’s user interface (UI).

Create a new C# script in the Scripts folder called FlexModeHelper.cs.
After creating the script, define the variables you need for this implementation.
```
public class FlexModeHelper : MonoBehaviour
{
	private FlexProxy WindowManagerListener;

	[SerializeField]
	private Camera MainCamera;
	[SerializeField]
	private Camera SubCamera;

	[SerializeField]
	private GameObject Flat_UI_Panel;
	[SerializeField]
	private GameObject Flex_UI_Panel;

	private RectTransform Flex_UI_Top;
	private RectTransform Flex_UI_Bottom;

	ScreenOrientation currentOrientation;

	bool isFold = false;
	bool isFlip = false;

	float landscapeFoV = 65;
	float portraitFoV;
```
- FlexProxy object is the callback object which receives the FoldableLayoutInfo from Java.
- MainCamera and SubCamera are two cameras creating the split-screen effect. However, Flex mode does not require the use of two cameras. So, if you only need one viewport, you can remove the SubCamera and replace it with any component you want.
- GameObjects, namely Flat_UI_Panel and Flex_UI_Panel, are the parent objects of two UI designs: Flat mode (full screen) and Flex mode (split screen).
- RectTransforms, namely Flex_UI_Top and Flex_UI_Bottom, are the two child objects within Flex_UI_Panel to serve as the top screen and bottom screen.
- ScreenOrientation object handles the Field of View (FoV) changes on the cover screen of the Galaxy Z Fold series (except Galaxy Fold) and the screen of non-foldable devices.
- isFold and isFlip are Booleans to store whether the app runs on a Galaxy Z Fold or Z Flip device and help create a consistent FoV across both devices.
- landscapeFoV and portraitFoV are two FoV values to keep a consistent FoV across both orientations of the device.
NoteIdentification of the device where the app is running is not necessary. Yet, it is recommended if you want to know how to customize the FoV of the game concerning the device model.

Next, construct a Start() method where you:

create a FlexProxy object and pass it into attachUnityListener on the Activity using Unity’s JNI implementation;
turn off the SubCamera initially, assuming that the game starts on Normal mode;
identify which device the app is running using SystemInfo.deviceModel. The model number of the Galaxy Z Fold series starts with "SM-F9", while it's "SM-F7" for the Galaxy Z Flip series;
calculate the portraitFoV from the landscapeFoV and the camera aspect ratio;
set the initial FoV depending on the device's orientation using Unity's Screen.orientation; and
retrieve the transforms of the Flex UI Top and Bottom panels.

void Start()
{
	WindowManagerListener = new FlexProxy();
	using (AndroidJavaClass javaClass = new AndroidJavaClass("com.unity3d.player.UnityPlayer"))
	{
		using (AndroidJavaObject activity = javaClass.GetStatic<AndroidJavaObject>("currentActivity"))
		{
			activity.Call("attachUnityListener", WindowManagerListener);
		}
	}

	SubCamera.enabled = false;

	string deviceModel = SystemInfo.deviceModel;
	if (deviceModel.Contains("SM-F9"))
	{
		isFold = true;
	}
	else
	{
		isFlip = deviceModel.Contains("SM-F7");
	}

	portraitFoV = Camera.HorizontalToVerticalFieldOfView(landscapeFoV, MainCamera.aspect);

	currentOrientation = Screen.orientation;

	if (currentOrientation == ScreenOrientation.Landscape || currentOrientation == ScreenOrientation.LandscapeLeft || currentOrientation == ScreenOrientation.LandscapeRight)
	{
		MainCamera.fieldOfView = landscapeFoV;
	}
	else
	{
		MainCamera.fieldOfView = portraitFoV;
	}

	Flex_UI_Top = (RectTransform)Flex_UI_Panel.transform.GetChild(0);
	Flex_UI_Bottom = (RectTransform)Flex_UI_Panel.transform.GetChild(1);

}

After the game starts, assign a task to FlexModeHelper to check if there is an update in the WindowManagerListener. The WindowManagerListener receives a call from Java when there is a change in the folded state.
If a change occurs, update the currentOrientation and run the UpdateFlexMode() method.

Alternatively, if the listener hasn't updated, check to see if the screen has changed orientation. Since the cover screen has no folding feature, FlexProxy will not update as the callback won't trigger. Instead, store the ScreenOrientation and compare if it matches the current screen orientation. Otherwise, change the FoV because the device has just rotated. Once we have figured out if the device has rotated, update the FoV of the MainCamera based on whether it's in landscape or portrait mode.

void Update()
{

	if (WindowManagerListener.hasUpdated)
	{
		currentOrientation = Screen.orientation;
		UpdateFlexMode();
	}
	else
	{
		if (Screen.orientation != currentOrientation)
		{
			currentOrientation = Screen.orientation;
			if (currentOrientation == ScreenOrientation.Landscape || currentOrientation == ScreenOrientation.LandscapeLeft || currentOrientation == ScreenOrientation.LandscapeRight)
			{
				MainCamera.fieldOfView = landscapeFoV;
			}
			else
			{
				MainCamera.fieldOfView = portraitFoV;
			}
		}
	}
}

Create the UpdateFlexMode() method to adjust the game UI according to the folded state of the device.
```
void UpdateFlexMode()
{
}
```

In this method, check if the folded state is HALF_OPENED. If so, enable the SubCamera for it to start rendering to the bottom screen and switch the active UI panel to the one created for Flex mode (Flex_UI_Panel).
```
if(WindowManagerListener.state == FlexProxy.State.HALF_OPENED)
{
	SubCamera.enabled = true;
	Flex_UI_Panel.SetActive(true);
	Flat_UI_Panel.SetActive(false);
```

Then, check whether the orientation of the fold is HORIZONTAL.
```
if (WindowManagerListener.orientation == FlexProxy.Orientation.HORIZONTAL)
{
```
NoteFor this sample game, splitting the screen isn’t ideal vertically from a user experience (UX) point of view. For this Code Lab activity, split the screen only on the horizontal fold (top and bottom screen). If you want to split the screen vertically, you need to use the same principle in the next step but for the X-axis instead of the Y-axis.
So, if the device is on Flex mode and horizontal fold, adjust the UI to place the MainCamera at the top and SubCamera at the bottom of the screen.

Locate the normalized location of the foldBounds.

float foldRatioTop = (float)WindowManagerListener.foldBounds.yMin / WindowManagerListener.currentMetrics.height;
float foldRatioBot = (float)WindowManagerListener.foldBounds.yMax / WindowManagerListener.currentMetrics.height;

Use these to set the render areas of the MainCamera and SubCamera above and below the foldBounds.

MainCamera.rect = new Rect(0, foldRatioTop, 1, foldRatioTop);
SubCamera.rect = new Rect(0, 0, 1, foldRatioBot);

Next, ensure that the FoVs are consistent across the two UIs. Reset the Field of View of each camera, and use the foldRatio to ensure that the size of objects in Flex mode appears roughly the same as in Flat mode. Also, use two slightly different FoVs for the Galaxy Fold and Flip devices.
```
if (isFold)
{
	MainCamera.fieldOfView = (landscapeFoV * foldRatioTop);
	SubCamera.fieldOfView = (landscapeFoV * foldRatioBot);
}
else
{
	if(isFlip)
	{
		MainCamera.fieldOfView = (landscapeFoV);
		SubCamera.fieldOfView = (landscapeFoV);
	}
}
```

Finally, update the two Child objects of the Flex_UI to ensure that they line up with the position of the fold.

Flex_UI_Top.anchorMin = new Vector2(0, foldRatioTop);
Flex_UI_Top.anchorMax = new Vector2(1, 1);
Flex_UI_Bottom.anchorMin = new Vector2(0, 0);
Flex_UI_Bottom.anchorMax = new Vector2(1, foldRatioBot);

However, if the device is not on Flex mode and the orientation of the fold is not horizontal, then run the RestoreFlatMode() method.

Notify the FlexProxy object that its data has been used.

		}
		else
		{
			RestoreFlatMode();
		}
	}
	else
	{
		RestoreFlatMode();
	}

	WindowManagerListener.hasUpdated = false;
}

Create the RestoreFlatMode() function where you:

set both cameras to render to the entire screen;
disable the SubCamera; and
disable the Flex UI and enable the Flat UI.

void RestoreFlatMode()
{
    MainCamera.rect = new Rect(0, 0, 1, 1);
    SubCamera.rect = new Rect(0, 0, 1, 1);
    SubCamera.enabled = false;
    Flex_UI_Panel.SetActive(false);
    Flat_UI_Panel.SetActive(true);

Also, check to see if the folded state is UNDEFINED, which means that the app is running on the cover screen of a Galaxy Z Fold device. If so, treat the screen as a rectangle and use either the landscapeFoV or the portraitFoV depending on the orientation. Additionally, since the app might have been initially opened on the main screen of a Galaxy Fold device, recalculate the portraitFoV using the aspect ratio of the cover screen.

If the folded state is not UNDEFINED, then the app is opened on the main screen of either a Galaxy Z Fold or Z Flip. In this case, set the FoV accordingly by using the isFold Boolean to check. If the Boolean returns true, treat the screen as a square. Otherwise, treat it as a rectangle and set the FoV based on the orientation.

	if (WindowManagerListener.state == FlexProxy.State.UNDEFINED)
	{
		if (currentOrientation == ScreenOrientation.Landscape || currentOrientation == ScreenOrientation.LandscapeLeft || currentOrientation == ScreenOrientation.LandscapeRight)
		{
			MainCamera.fieldOfView = landscapeFoV;
		}
		else
		{
			if(isFold)
				portraitFoV = Camera.HorizontalToVerticalFieldOfView(landscapeFoV, MainCamera.aspect);
			MainCamera.fieldOfView = portraitFoV;
		}
	}
	else
	{
		if (isFold)
		{
			MainCamera.fieldOfView = landscapeFoV;
		}
		else
		{
			if (currentOrientation == ScreenOrientation.Landscape || currentOrientation == ScreenOrientation.LandscapeLeft || currentOrientation == ScreenOrientation.LandscapeRight)
			{
				MainCamera.fieldOfView = landscapeFoV;
			}
			else
			{
				MainCamera.fieldOfView = portraitFoV;
			}
		}
	}
}

Set up Flex Scene

Return to the Unity Editor and create an empty GameObject in the scene.

Right-click on the Sample Scene > GameObject > Create Empty and name it FlexManager.

Select the FlexManager object, then drag and drop the FlexModeHelper script into the Inspector pane.

Then, select the Cameras and UI Panels like below:

Build and run the app

Go to File > Build Settings and ensure that the Scenes/SampleScene is selected in Scenes in Build. Click Build to build the APK.

After building the APK, run the game app on a foldable Galaxy device and see how the UI switches from Normal to Flex mode. If you don’t have any physical device, you can also test it on a Remote Test Lab device.

TipWatch this tutorial video and know how to easily test your app via Remote Test Lab.

You're done!

Congratulations! You have successfully achieved the goal of this Code Lab. Now, you can implement Flex mode in your Unity game app by yourself! If you're having trouble, you may download this file:

Flex Mode on Unity Complete Code

(631.05 MB)

To learn more, visit:
www.developer.samsung.com/galaxy-z
www.developer.samsung.com/galaxy-gamedev