Use AR Emoji on Games & 3D Apps


Objective

Learn how to use AR Emoji as character of games or 3D applications using Unity Engine.

Overview

Samsung Galaxy device can transform your face into an animated version, and it is a way to represent and express yourself. Once you have your avatar, your AR Emoji can mirror a wide range of emotions and body movements with advanced facial and body tracking. You can even impose your AR Emoji on top of a photo of yourself and watch it move just like you!

Other than these basic features, the Galaxy AR Emoji SDK for Unity enables you to use the user's AR Emoji avatar in your application, such as using it to represent a game character.

In this Code Lab, you will get to know about Unity packages that can help you to use AR Emoji models in your Unity application like games with simple code updates.

You will first learn about the basic pipeline for creation of app with AR Emoji character with AR Emoji SDK by loading a character model and rendering on the mobile app. Afterwards, you will learn a way of applying custom animation to the AR Emoji character.

You can make your own app with a simple sequence of operation and by following the steps in this Code Lab. Moreover, you can learn about the character file import process and get to know more how you can maximize Galaxy AR Emoji SDK.

Set up your environment

You will need the following:

  • Unity (LTS Release 2018.4.Xf1, Windows version)
  • Editor for editing Unity script (like Visual Studio)
  • Select different values to test how your game will look on monitors with different aspect ratios. This is set to free aspect by default. (Recommended: 1024x2048)
  • Samsung Galaxy device supporting AR Emoji

Create a Unity sample project

Run Unity and create an empty 3D sample project.

Configure the settings

  1. Go to Build Settings.

    a. File > Build Settings

    b. Platform > Android

Specific package name and keystore is necessary for testing. In the next steps, it’s explained in detail.

  1. Go to Player Settings > Other Settings > Identification > Package Name.

    Set the Package Name to samsung.com.fileprovidersample.

  2. Go to Player Settings > Publishing Settings.

    Browse keystore : fileprovidersample.jks

    a. Add keystore with browser (included Docs folder of GettingAREmoji.unitypackage).

    b. Keystore password : 123456

    c. Key

    • Alias : keyalias

    • Password : 123456

Import the sample app package

  1. Import the package.

    Run GettingAREmoji.unitypackage, then click Import.

  2. Check the imported sample app.

    a. Go to Assets > Scenes > GettingAREmojiSample Scene.

    b. Click Canvas.

Import the glTF Importer package to load AR Emoji model in Unity

  1. Import the package.

Run GLTFImporter.unitypackage, then click Import.

  1. Check the imported packages.

Integrate the glTF Importer to the sample app

  1. Go to the sample scene.

    a. Click Assets > Scenes > GettingEmojiSample.unity.

    b. Click Canvas.

    c. Check ButtonBehaviour (script) in Inspector.

  2. Define GLTFComponent in your application.

    a. Open Assets > Scripts > ButtonBehaviour.cs.

    b. Define glTFComponent:

    
    public class ButtonBehaviour : MonoBehaviour
    {
        public Button m_connectBtn;
        public Button m_getListBtn;
        public Button m_getFileBtn;
        public Button m_getPreviewBtn;
        public Text m_Text;
        public RawImage m_Image;
    
        public UnityGLTF.GLTFComponent gLTFComponent;
        
    

    c. Confirm GLTF component property appears in inspector (compare to Step 1).

  3. Add loadGLTF API in your app.

    a. Write the code below in OnClickLoadModel() of ButtonBehaviour.cs to load selected model:

    
    private void OnClickLoadModel()
    {
        m_Text.text = "";
        m_Text.text = "OnClickLoadModel";
        if (!isGetFile)
        {
            m_Text.text = "\nisGetFile : " + isGetFile + "\n";
            return;
        }
        //add below code
        if (filePath != null)
        {
            StartCoroutine(gLTFComponent.loadGLTF(filePath));
        }
    }
    
    

    b. Write the code below in Start() of ButtonBehaviour.cs to attach the GLTF Component:

    
    void Start()
    {
        m_Text.text += "\nStart";
        jc = new AndroidJavaClass("com.unity3d.player.UnityPlayer");
        jo = jc.GetStatic<AndroidJavaObject>("currentActivity");
    
        m_Dropdown.onValueChanged.AddListener(delegate {
        DropdownValueChanged(m_Dropdown);
        });
    
        if (GameObject.Find("GLTFComponent"))
        {
            GameObject o = GameObject.Find("GLTFComponent");
            //add below code
            gLTFComponent = o.GetComponent<UnityGLTF.GLTFComponent>();
        }
        
    
  4. Copy GLTFComponent prefab into your app to use GLTF Component property of Step 2.

    a. Click Assets > Prefabs.

    b. Drag GLTFComponent prefab to your Scene.

Build and run

  1. Go to File > Build Settings > Build and Run.

  1. Launch the sample app in your device.

    a. Click Get List and confirm AR Emoji lists saved in device.

    b. Select one of them.

    c. Click Get Preview and Load Preview.

    d. Confirm if the AR Emoji thumbnail is showing.

    e. Click Get File and Load Model.

    f. Confirm if the AR Emoji model is showing.

Create your own AR Emoji

You can create your own AR Emoji with the following steps:

  1. Run Camera application.

  2. Click AR Emoji and the + button.

  1. Take your selfie as guideline and choose your gender.

  1. Edit your AR Emoji.

    a. Customize looks, clothes, or accessories of your AR Emoji.

    b. Press Next, once done.

  2. Check your AR Emoji.

    a. Click Done button, once you complete editing.

    b. Check your AR Emoji in Mask mode.

    c. Click Studio button and check your AR Emoji in AR Emoji Studio.

Animation system overview

Unity has a rich and sophisticated animation system (sometimes referred to as Mecanim). While Mecanim is recommended for use in most situations, Unity has retained its legacy animation system which existed before Unity 4. Because you may need to use the legacy when working with older content created before Unity 4.

Here, the Mecanim, shown in the figure above, is explained mainly and legacy is also guaranteed:

  1. Animation Clips are one of the core elements to Unity’s animation system. Unity supports importing animation from external sources. The Animation Clips and the Animator Controller are brought together on a GameObject via the Animator Component.

  2. This manages the various animation states and the transitions between them using a so-called State Machine, which could be thought of as a kind of flow-chart, or a simple program written in a visual programming language within Unity.

  3. The animation controller has references to the animation clips used within it.

  4. The model is imported at run time. When a GLTFComponent is created, the animation controller is connected to the specific game object (rig_GRP).

Learn the basics of Animator Controller

To apply the animation to the model, you must have a basic knowledge of the animation controller. Developers can easily control animation with it.

An Animator Controller allows you to arrange and maintain a set of animation clips and associated Animation Transitions for a character or object. In most cases it is normal to have multiple animations and switch between them when certain game conditions occur. For example, you could switch from a walk animation clip to a jump animation clip whenever the space bar is pressed. However, even if you only have a single animation clip, you still need to place it into an Animator Controller to use it on a GameObject.

The Animator Controller has references to the animation clips used within it, and manages the various animation clips and the transitions between them using a State Machine, which could be thought of as a flow-chart of animation clips and transitions, or a simple program written in a visual programming language within Unity.

Here, the model is loaded runtime, and then an animation controller component is added to the model. So, you will learn how to make an animation clip and apply a simple animation through a State Machine.

Runtime Animation Controller of GLTFComponent.cs:


GameObject o = GameObject.Find("rig_GRP");
Animator animator = o.AddComponent<Animator>();
RuntimeAnimatorController controller = Resources.Load<RuntimeAnimatorController>("AnimController");
animator.runtimeAnimatorController = controller

Generate animation clips

Animation clips are one of the core elements to Unity’s animation system. Unity supports importing animation from external sources, and offers the ability to create animation clips from scratch within the editor using the Animation window.

Animation data collection

We provided a tool to make what works in our model. Before then, you need animation data. It can be made directly with a professional tool like Maya and 3Ds Max or it can be obtained for free like Mixamo (https://www.mixamo.com/).

An animation designed file (*.fbx) should be created as an animation clip (*.anim) for unity.

The animation clip can be generated using Unity Studio. It will be supported later by Windows and Mac operating systems.

  1. Import the package.

  2. Check the imported AnimationGenToolScene.

    Assets > Scenes

  3. In Game window, if you adjust the scale (1x), you can see the screen below.

  4. Click the Create .Anim button and the Windows Explorer will show up.

    a. Select the *.fbx file.

    b. When completed, the following information is displayed.

    c. Verify the generated *.anim file.

  5. If you want to check the animation, click the Load Model button.

Set up the Animation State Machines

Unity’s Animation State Machines provide a way to overview all of the animation clips related to a particular character and allow various events in the game (for example, user input) to trigger different animations.

State Machine Transitions exist to help you simplify complex State Machines. This allows you to have a higher level of abstraction over the state machine logic. Each view in the animator window has an Entry and Exit node. The animator window specifies the State Machine Transitions. The Entry node is used when transitioning into a state machine. The entry node will be evaluated and will branch to the destination state according to the conditions set. In this way, the entry node can control which state the state machine begins in, by evaluating the state of your parameters when the state machine begins. Because the state machine always has a default state, there will always be a default transition branching from the entry node to the default state.

Make the character walk

Here, use a walking animation clip. You can find it here:

Project > Assets > Resources > Mecanim > Walk.anim

  1. The Walk animation clips are placed and arranged in an Animator Controller.

    a. Double click the AnimController, then you can see the Animator window.

    b. Next, apply the animation by dragging the Walk animation to the button.

  2. Make the transition from Idle to Walk.

    • Right-click the Idle node.

    • Connect lines from Idle node toward Walk node.

  3. Select the connecting line for transition (Idle → Walk).

    a. Uncheck Has Exit Time.

    b. Set the parameter to Bool.

    Animation Parameters are variables that are defined within an Animator Controller that can be accessed and assigned values from scripts. This is how a script can control or affect the flow of the state machine.

    c. Add the condition of transition (Idle → Walk).

  4. Select the connecting line for transition (Idle ← Walk).

    Repeat in the same way as Step 3. However, the condition of walk should be set to false.

  5. Write the script.

  6. In ButtonBehaviour.cs, write the script that modifies parameters based on user input:

    
    public class ButtonBehaviour : MonoBehaviour
    {
        public Animator anim;
     
       // Update is called once per frame
        void Update()
        {
            if (anim == null)
            {
                GameObject o = GameObject.Find("rig_GRP");
                if (o != null)
                {
                    anim = o.GetComponent<Animator>();
                }
    
            }
            else
            {
                if (Input.GetKey(KeyCode.RightArrow))
                {
                    transform.rotation = Quaternion.Euler(0, -90, 0);
                    anim.SetBool("walk", true);
                }
                else if (Input.GetKey(KeyCode.LeftArrow))
                {
                    transform.rotation = Quaternion.Euler(0, 90, 0);
                    anim.SetBool("walk", true);
                }
                else if (Input.GetKey(KeyCode.DownArrow))
                {
                    transform.rotation = Quaternion.Euler(0, 0, 0);
                    anim.SetBool("walk", false);
                }
                else
                {
                    anim.SetBool("walk", false);
                }
            }
        }
    }
    
    
  7. Play and review the Walk Animation.

Then, push the right-button of the keyboard, to make your AR Emoji character move to the right using the Walk animation.

After Walk Animation, the state of the AR Emoji will be back to Idle animation.

You're done!

Congratulations! You have successfully achieved the goal of this Code Lab. Now, you can create AR Emoji, and may use it as a character in games or apps, all by yourself! If you're having trouble, you may download the file:

AR Emoji Complete Code
(183.37 MB)