top

ATT (Automatic Test Tool) User Guide

Published 2014-10-28 | (Compatible with SDK 5.0,5.1 and 2014 models)

ATT (Automatic Test Tool) User Guide

Introduction to ATT for SmartTV SDK

Samsung ATT (Automatic Test Tool) is a utility tool which provides features for automatic testing of SmartTV applications.

Figure 1 ATT

This tool has various features like creating test suite recording events from the Remote, playing test case, image comparisons etc. The tool enables users to record a sequence of events by running the application on the emulator. Recorded event sequences can be replayed to run a test case automatically. The user can also insert “Screen Capture” and “Loop” events between other events. The tool will capture the emulator screen automatically whenever a “Screen Capture” event occurs while executing the test.

At a high level, there are four steps to use this tool:

  • Creating a workspace (Test Case, Test Suite,Test Suite Group).
  • Recording of events
  • Editing the event sequence (optional)
  • Playing the recorded events/Executing the test case.

The rest of this document explains these steps in detail.

Prerequisites

  • ATT is supported on Windows 7(32/64 Bit), Linux (Ubuntu 12.04 LTS 32/64 Bit), and Mac 64, with Smart TV SDK version 5.0B and later.
  • User should set the appropriate TOOLS path from VirtualBox Manager ‣ Settings ‣ Shared Folders ‣ Machine Folders ‣ TOOLS for connection establishment between ATT and Emulator and saving the images of Capture Screen events.
  • Tools Folder name should be TOOLS.

Figure 2 VirtualBox TOOLS Folder settings

  • User should restart the Emulator to reflect changes made to the TOOLS path after setting the appropriate path.
  • “NAT” Network settings for Virtual machine should be selected for proper communication between ATT and Emulator

Figure 3 Virtual Box Network setting

Figure 4 Sound setting required for Emulator virtual machine

  • User should have the following sound setting for the Emulator Virtual Machine as shown in the image.

Installation

Download the right ATT package from SDK Download according to your operating system. Decompress the package in any folder. The ATT will be ready to use.

Launching the ATT

ATT will work on Windows, Linux and Mac platforms. It can be started on the different platforms as follows:

  1. Windows 32 and 64 bit systems - Double click on executable of AutomaticTestTool.
  2. Linux 32 and 64 bit systems - Double click on RunATT.sh file, a new dialog box will open then press “Run”.
  3. Mac systems - Right click on the executable of AutomaticTestTool and goto AutomaticTestTool.app ‣ Contents ‣ MacOS ‣ AutomaticTestTool.

Creation of Workspace

In the ATT (Automatic Test Tool) users can create a workspace by creating a Test Case as follows:

TC (Test Case) is one of the File menu options available to allow user to create a collection of Events. A TC can be created by selecting the menu options.
File ‣ New ‣ Test Case

Figure 5 Test Case

Figure 6 New Test Case

Figure 7 Newly created Test Case

TS (Test Suite) is one of the File menu options available to allow the user to create a collection of Test Case. A TS can be created by selecting the menu options.
File ‣ New ‣ Test Suite

Figure 8 Test Suite

Figure 9 New Test Suite

Figure 10 Newly created Test Suite

Figure 11 Adding New/Existing Test Case

Figure 12 Adding new Test Case in Test Suite

Figure 13 Newly added Test Case in Test Suite

TSG (Test Suite Group) is one of the File menu options available to allow the user to create a collection of Test Suites. TSG can be created by selecting menu option.
File ‣ New ‣ Test Suite Group

Figure 14 Test Suite Group

Figure 15 New Test Suite Group

Figure 16 Newly created Test Suite Group

Figure 17 Adding New/Existing Test Suite

Figure 18 Newly added Test Suite in Test Suite Group

Figure 19 Newly added Test Case in a Test Suite for a Test Suite Group

User can also rename the TSGs, TSs and TCs by double clicking on them and then entering the new name in the popup window which appears. Renaming is case insensitive and supports only English language.

Manual Insertion, Editing and Deletion of Events

ATT allows insertion of events manually in the event screen. Some of these events (such as Screen Capture) are not available on the Remote Control.

Inserting Remote Control Key events

Users can also manually insert all the remote control events in the list of events from the Remote Control provided in the ATT

Figure 20 Remote Control in the ATT to enter the event

Note

Only Basic Device is supported by the ATT.

Inserting “CAPTURE SCREEN” & “PARTIAL SCREEN CAPTURE” events

“CAPTURE SCREEN” event captures the running emulator screen shot. “CAPTURE SCREEN” event can be added after or before of an event in the event list by selecting the context menu which comes on right clicking of the events.

If “Create Ref Images” option is selected on ATT UI then the captured image will be stored as reference image for future test case execution. Otherwise if “Compare Images” option is selected then the captured image is compared with already stored reference image assuming that reference images is already created in past test case execution. Check Playing with “Creating Reference Images” option and Playing with “Compare Images” option for more information.

On Selection of the “Partial Screen Capture”, ATT pops up window to enter the capture area location and size. Maximum permissible is 0, 0, 960, and 540. And user can parallel see the location of the rectangle on the on the Emulator.

Figure 21 Context menu to Insert CAPTURE SCREEN Before/After

Figure 22 Newly added CAPTURE SCREEN before the event

Figure 23 Partial Capture Screen option

Figure 24 Partial Capture Screen popup window before the event

Figure 25 Partial Capture Screen window after the event with dynamic rectangle display

Inserting “DYNAMIC CAPTURE SCREEN” event

“DYNAMIC CAPTURE SCREEN” event gets the status of the any dynamic part of emulator screen like video and animation. It could be tested if a video or animation is playing or not playing in a part of emulator screen. “DYNAMIC CAPTURE SCREEN” event can be added after or before an existing event in the event list using context menu which comes on right click on the events.

On Selection of the “Dynamic Capture Screen”, ATT pops up window to enter the capture area location and size. In parallel, user can see the location of the rectangle on the on the Emulator (if the connection between ATT and emulator has been established). Once user presses OK button the event adds in the event list.

By double clicking on the “Dynamic Screen Play”/ “Dynamic Screen Pause”, User can edit the rectangle size and duration event.

Figure 26 DYNAMIC CAPTURE SCREEN popup window before the event

Figure 27 DYNAMIC CAPTURE SCREEN window after the event with dynamic rect display

Figure 28 “Dynamic Screen Play” event in the event list

Inserting “SOUND TEST” event

SOUND TEST event gets the status of the sound play/pause of the application running in the emulator. “SOUND TEST” event can be added after or before of an event in the event list.

On Selection of the “Sound Test”, it gives four sub menu options as it is shown in the following image. As user selects the option an event gets added in the event list.

Figure 29 SOUND TEST popup window before the event

Figure 30 SOUND PLAY TEST event in the event list

Inserting “Loop” event

The Loop Event is used to repeatedly execute the same event sequence. Mouse Right Click on event screen ‣ Insert loop After insert loop, screen is shown in the below picture.

Figure 31 Context menu to Insert Loop on event screen.

Figure 32 Newly inserted loop

Figure 33 Nested loop structure

Editing the Loop Counter

Once a loop is inserted, the user can set the counter by double clicking in the event screen on the counter, which makes it editable and enter the desired loop counter. It is shown in the following picture

Figure 34 Editing the counter

Updating Delay

To Delay the Event, double click in the event screen on the Delay (Unit), which makes it editable and user can enter the desired Delay Unit. Here one unit is equivalent of 10 milliseconds. Default delay unit will be taken automatically from the Emulator input. Default delay is set to 100ms when input is given through ATT Emulator. But default delay can be set manually also. Respective image is shown below.

Figure 35 Manual setting of Delay Event

Figure 36 Default delay unit setting for each event for ATT Input device.

Delete an Event

User can also delete any event from the list. Deletion of the particular event from the event list can be done by selecting the “Delete” option from the context menu (which comes on right click on the particular event).

Figure 37 Context menu to delete the event.

Open Captured Image / Open Reference Image

Users can view the captured/reference image by selecting the “Open Captured Image / Open Reference Image” on the context menu by right clicking event.

Figure 38 Context menu to Open the captured/Reference the event

Show in Folder

“Show in Folder” menu option opens the folder location where Captured/Reference Image is saved. Folder can be opened by right clicking on the context menu and choosing “Show in Folder”.

Figure 39 Context menu to Show in Folder

Figure 40 Opened folder location for Captured/Reference Image

Figure 41 Shared folder for capture image

Figure 42 Warning if TOOLS folder is not set in the VirtualBox

Note

The images captured from the ATT are saved in the shared path of the emulator.

Default path for captured images ‣ <shared path of TOOLS in emulator>/ATT/<test case>/CaptureImage

Default path for reference images. ‣ <shared path of TOOLS in emulator>/ATT/<test case>/RefImage

Note

If this shared folder path is changed when the emulator is running, the emulator has to be restarted to reflect the change.

Setting Default Work Station Location

Setting the Default Work Station Location enables user to set the default location to store all the TCs. Setting of the Default Work Station Location can be done by choosing the “Settings” from the View menu which is shown in the following picture.

Figure 43 Setting Menu

Figure 44 Setting the desired location for saving test case

Recording of Events

Recording of the event is required to prepare the event list for test case. It helps user in remembering the events. To start the recording user can press a button (picture of button) in “Record” group on ATT UI. Once recording is in progress, play area will not be active. User needs to finish or stop the recording to play the test cases. There is another button just below the record button to stop the recording.

Figure 45 ATT appears as above picture

Figure 46 ATT appears as above while recording

On pressing the record button for the very first time after the ATT is launched the users would be prompted to select the appropriate virtual machine which contains the Samsung emulator. This list appears as a drop down menu inside a terminal window which opens in the ATT.

Figure 47 Terminal window containing the list of virtual machines

Figure 48 Emulator Launching up window for the ATT

Recording with “Record Mouse Move” option

The ATT includes the option to record Mouse Move Events.. Mouse Move event recorder is disabled by default because capturing mouse movement generates a large number of events. The following picture displays the option to record the mouse move event.

Figure 49 Mouse move event record option

Figure 50 Fake mouse pointer used in the Emulator to play the recorded mouse events

Playing of Events (Test case execution)

Play is used to execute a TC. Once the user has recorded the test cases, he/she will be able to replay the recorded event sequences. Similar to event recording feature, when event playing is triggered first time, the user is prompted to choose the virtual machine which contains the Samsung Smart TV Emulator (Figure 23 and Figure 24). If the emulator is not up then ATT will launch the emulator. Once user has started playing the test case, record button will be disabled. To perform recording the user is required to stop the playing and press record button. “Play Event” and “Stop Playing Event” button are shown in the following picture.

Figure 51 ATT appears as above after recording

Figure 52 User should use Play button on ATT for playing the record

Reference Image feature:

  • A set of reference images (screen shots) can be generated by running test once.
  • Current captured images in all future execution of same tests can be compared with the generated reference images.
  • Result (pass/fail) is reported after execution.
  • It will be possible to change reference images in future if needed

Playing of Events has two options to execute the events sequence (at a time only one can be selected out of these two options).

  1. Create Reference Image
  2. Compare Images

Figure 53 Create Image checkbox

Playing with “Creating Reference Images” option

“Create Ref Images” is a feature which facilitates the user to capture the image so that the user can use it for further analysis, comparison etc. When this option is selected, the created images will be used as reference images and those will be saved at the pre-define location (workspace path).

  1. The following steps are required to capture the image. Click on the capture image as shown in the following picture
  2. Play events

Figure 54 Create Ref Image Check box checked

When Capture/Compare ref Image is added in the Test Case, users get the option to “Open Captured Image”, “Open Reference Image” and “Show in folder” option which is shown in the above picture.

Playing with “Compare Images” option

“Compare image” option provides a way to the user to compare the captured image with reference image. The following steps need to be followed to compare the captured image:

  1. Click on the compare image.
  2. Play events

Figure 55 Compare image option

After playing the complete recorded test cases, it pops up the result in a message which is shown in the following picture.

Figure 56 Compare image result popup

Viewing the Last result

Last result menu show the last result. It is stored at the default location. User can see last result using the View Menu ‣ Last result. It pops up a window which displays the last result.

Figure 57 Last result menu

Figure 58 Last result

Command line support for ATT

ATT can also be used via command line arguments and hence can be used for automation of test cases. To use this functionality the user needs to follow some simple steps.

  1. Open command line in Windows or terminal in Linux/Mac systems and go to the ATT package folder.

    e.g.: cd /home/<username>/ATT_Linux32_Bit

  2. User can start the ATT using following command

    1. Windows: AutomaticTestTool.exe
    2. Linux/Mac Machine: ./RunATT.sh
  3. Command line functionality is as follows:

    /help

    This command shows all the commands that can be used via command line. It generates the following output:

    ATT help command output

    /help                           Shows help. Ignore all other input parameters.
    /input=<filepath>               Input event file.
    /vmname=<Virtual_Machine_Name>  Virtual Machine Name.
    /createref                      Create reference images, can't use with /compare.
    /compare                        Compare images, can't use with /createref.
    /result[=<filepath>]            To save comparison result. If <filepath> is not defined default file path will be taken.
    /show                           Show UI of ATT. Default mode is Hide.
    /exit                           Exit from Automatic Test Tool.
    
    /input=<filepath>

    User needs to provide the file path to the .stti file to run that particular test case.

    Example: /input="/home/$USER/ATT_Package/ATT/TestCase1/TestCase1.stti"

    /vmname=<Virtual Machine Name>

    User will provide the Emulator virtual machine name for this parameter.

    User can get the virtual machine name using Oracle VM VirtualBox manager settings. Settings‣General‣Basic‣ Name

    Figure 59 VBox Emulator OVA settings

    /createref

    This tag is used after providing the /input and /vmname to create reference images

    /compare

    This tag is used for comparing captured images with the reference images. Please note that similar to the ATT functionality using UI /compare and /createref cannot be used together.

    /result=[<Desired store location for the result>]

    This tag is used along with /compare tag. User can either provide the file path to store the result file in .csv format or can just type /result in which case a default result file will be stored at the location of the .stti file.

    Example: /result="/home/$USER/TestCaseResults/TestCase1Results/csv". Default file path for /input=" /home/$USER/ATT_Package/ATT/TestCase1/TestCase1.stti" is /home/$USER/ATT_Package/ATT/TestCase1/Result/TestCase1_<time_stamp>.csv. For instance a result file generated at 2:30pm on 21st December 2013 will have the following name TestCase1_20131221_14h30m35s.csv

Abbreviation

Test Case TC
Test Suite TS
Test Suite Group TSG