ATT (Automatic Test Tool) User Guide
ATT (Automatic Test Tool) User Guide
- Introduction to ATT for SmartTV SDK
- Launching the ATT
- Creation of Workspace
- Manual Insertion, Editing and Deletion of Events
- Inserting Remote Control Key events
- Inserting “CAPTURE SCREEN” & “PARTIAL SCREEN CAPTURE” events
- Inserting “DYNAMIC CAPTURE SCREEN” event
- Inserting “SOUND TEST” event
- Inserting “Loop” event
- Editing the Loop Counter
- Updating Delay
- Delete an Event
- Open Captured Image / Open Reference Image
- Show in Folder
- Setting Default Work Station Location
- Recording of Events
- Playing of Events (Test case execution)
- Command line support for ATT
Samsung ATT (Automatic Test Tool) is a utility tool which provides features for automatic testing of SmartTV applications.
This tool has various features like creating test suite recording events from the Remote, playing test case, image comparisons etc. The tool enables users to record a sequence of events by running the application on the emulator. Recorded event sequences can be replayed to run a test case automatically. The user can also insert “Screen Capture” and “Loop” events between other events. The tool will capture the emulator screen automatically whenever a “Screen Capture” event occurs while executing the test.
At a high level, there are four steps to use this tool:
- Creating a workspace (Test Case, Test Suite,Test Suite Group).
- Recording of events
- Editing the event sequence (optional)
- Playing the recorded events/Executing the test case.
The rest of this document explains these steps in detail.
- ATT is supported on Windows 7(32/64 Bit), Linux (Ubuntu 12.04 LTS 32/64 Bit), and Mac 64, with Smart TV SDK version 5.0B and later.
- User should set the appropriate TOOLS path from VirtualBox Manager ‣ Settings ‣ Shared Folders ‣ Machine Folders ‣ TOOLS for connection establishment between ATT and Emulator and saving the images of Capture Screen events.
- Tools Folder name should be TOOLS.
- User should restart the Emulator to reflect changes made to the TOOLS path after setting the appropriate path.
- “NAT” Network settings for Virtual machine should be selected for proper communication between ATT and Emulator
- User should have the following sound setting for the Emulator Virtual Machine as shown in the image.
Download the right ATT package from SDK Download according to your operating system. Decompress the package in any folder. The ATT will be ready to use.
ATT will work on Windows, Linux and Mac platforms. It can be started on the different platforms as follows:
- Windows 32 and 64 bit systems - Double click on executable of AutomaticTestTool.
- Linux 32 and 64 bit systems - Double click on RunATT.sh file, a new dialog box will open then press “Run”.
- Mac systems - Right click on the executable of AutomaticTestTool and goto AutomaticTestTool.app ‣ Contents ‣ MacOS ‣ AutomaticTestTool.
In the ATT (Automatic Test Tool) users can create a workspace by creating a Test Case as follows:
TC (Test Case) is one of the File menu options available to allow user to create a collection of Events. A TC can be created by selecting the menu options.
File ‣ New ‣ Test Case
TS (Test Suite) is one of the File menu options available to allow the user to create a collection of Test Case. A TS can be created by selecting the menu options.
File ‣ New ‣ Test Suite
TSG (Test Suite Group) is one of the File menu options available to allow the user to create a collection of Test Suites. TSG can be created by selecting menu option.
File ‣ New ‣ Test Suite Group
User can also rename the TSGs, TSs and TCs by double clicking on them and then entering the new name in the popup window which appears. Renaming is case insensitive and supports only English language.
ATT allows insertion of events manually in the event screen. Some of these events (such as Screen Capture) are not available on the Remote Control.
Users can also manually insert all the remote control events in the list of events from the Remote Control provided in the ATT
Only Basic Device is supported by the ATT.
“CAPTURE SCREEN” event captures the running emulator screen shot. “CAPTURE SCREEN” event can be added after or before of an event in the event list by selecting the context menu which comes on right clicking of the events.
If “Create Ref Images” option is selected on ATT UI then the captured image will be stored as reference image for future test case execution. Otherwise if “Compare Images” option is selected then the captured image is compared with already stored reference image assuming that reference images is already created in past test case execution. Check Playing with “Creating Reference Images” option and Playing with “Compare Images” option for more information.
On Selection of the “Partial Screen Capture”, ATT pops up window to enter the capture area location and size. Maximum permissible is 0, 0, 960, and 540. And user can parallel see the location of the rectangle on the on the Emulator.
“DYNAMIC CAPTURE SCREEN” event gets the status of the any dynamic part of emulator screen like video and animation. It could be tested if a video or animation is playing or not playing in a part of emulator screen. “DYNAMIC CAPTURE SCREEN” event can be added after or before an existing event in the event list using context menu which comes on right click on the events.
On Selection of the “Dynamic Capture Screen”, ATT pops up window to enter the capture area location and size. In parallel, user can see the location of the rectangle on the on the Emulator (if the connection between ATT and emulator has been established). Once user presses OK button the event adds in the event list.
By double clicking on the “Dynamic Screen Play”/ “Dynamic Screen Pause”, User can edit the rectangle size and duration event.
SOUND TEST event gets the status of the sound play/pause of the application running in the emulator. “SOUND TEST” event can be added after or before of an event in the event list.
On Selection of the “Sound Test”, it gives four sub menu options as it is shown in the following image. As user selects the option an event gets added in the event list.
The Loop Event is used to repeatedly execute the same event sequence. Mouse Right Click on event screen ‣ Insert loop After insert loop, screen is shown in the below picture.
Once a loop is inserted, the user can set the counter by double clicking in the event screen on the counter, which makes it editable and enter the desired loop counter. It is shown in the following picture
To Delay the Event, double click in the event screen on the Delay (Unit), which makes it editable and user can enter the desired Delay Unit. Here one unit is equivalent of 10 milliseconds. Default delay unit will be taken automatically from the Emulator input. Default delay is set to 100ms when input is given through ATT Emulator. But default delay can be set manually also. Respective image is shown below.
User can also delete any event from the list. Deletion of the particular event from the event list can be done by selecting the “Delete” option from the context menu (which comes on right click on the particular event).
Users can view the captured/reference image by selecting the “Open Captured Image / Open Reference Image” on the context menu by right clicking event.
“Show in Folder” menu option opens the folder location where Captured/Reference Image is saved. Folder can be opened by right clicking on the context menu and choosing “Show in Folder”.
The images captured from the ATT are saved in the shared path of the emulator.
Default path for captured images ‣ <shared path of TOOLS in emulator>/ATT/<test case>/CaptureImage
Default path for reference images. ‣ <shared path of TOOLS in emulator>/ATT/<test case>/RefImage
If this shared folder path is changed when the emulator is running, the emulator has to be restarted to reflect the change.
Setting the Default Work Station Location enables user to set the default location to store all the TCs. Setting of the Default Work Station Location can be done by choosing the “Settings” from the View menu which is shown in the following picture.
Recording of the event is required to prepare the event list for test case. It helps user in remembering the events. To start the recording user can press a button (picture of button) in “Record” group on ATT UI. Once recording is in progress, play area will not be active. User needs to finish or stop the recording to play the test cases. There is another button just below the record button to stop the recording.
On pressing the record button for the very first time after the ATT is launched the users would be prompted to select the appropriate virtual machine which contains the Samsung emulator. This list appears as a drop down menu inside a terminal window which opens in the ATT.
The ATT includes the option to record Mouse Move Events.. Mouse Move event recorder is disabled by default because capturing mouse movement generates a large number of events. The following picture displays the option to record the mouse move event.
Play is used to execute a TC. Once the user has recorded the test cases, he/she will be able to replay the recorded event sequences. Similar to event recording feature, when event playing is triggered first time, the user is prompted to choose the virtual machine which contains the Samsung Smart TV Emulator (Figure 23 and Figure 24). If the emulator is not up then ATT will launch the emulator. Once user has started playing the test case, record button will be disabled. To perform recording the user is required to stop the playing and press record button. “Play Event” and “Stop Playing Event” button are shown in the following picture.
Reference Image feature:
- A set of reference images (screen shots) can be generated by running test once.
- Current captured images in all future execution of same tests can be compared with the generated reference images.
- Result (pass/fail) is reported after execution.
- It will be possible to change reference images in future if needed
Playing of Events has two options to execute the events sequence (at a time only one can be selected out of these two options).
- Create Reference Image
- Compare Images
“Create Ref Images” is a feature which facilitates the user to capture the image so that the user can use it for further analysis, comparison etc. When this option is selected, the created images will be used as reference images and those will be saved at the pre-define location (workspace path).
- The following steps are required to capture the image. Click on the capture image as shown in the following picture
- Play events
When Capture/Compare ref Image is added in the Test Case, users get the option to “Open Captured Image”, “Open Reference Image” and “Show in folder” option which is shown in the above picture.
“Compare image” option provides a way to the user to compare the captured image with reference image. The following steps need to be followed to compare the captured image:
- Click on the compare image.
- Play events
After playing the complete recorded test cases, it pops up the result in a message which is shown in the following picture.
Last result menu show the last result. It is stored at the default location. User can see last result using the View Menu ‣ Last result. It pops up a window which displays the last result.
ATT can also be used via command line arguments and hence can be used for automation of test cases. To use this functionality the user needs to follow some simple steps.
Open command line in Windows or terminal in Linux/Mac systems and go to the ATT package folder.
e.g.: cd /home/<username>/ATT_Linux32_Bit
User can start the ATT using following command
- Windows: AutomaticTestTool.exe
- Linux/Mac Machine: ./RunATT.sh
Command line functionality is as follows:
This command shows all the commands that can be used via command line. It generates the following output:
/help Shows help. Ignore all other input parameters. /input=<filepath> Input event file. /vmname=<Virtual_Machine_Name> Virtual Machine Name. /createref Create reference images, can't use with /compare. /compare Compare images, can't use with /createref. /result[=<filepath>] To save comparison result. If <filepath> is not defined default file path will be taken. /show Show UI of ATT. Default mode is Hide. /exit Exit from Automatic Test Tool.
User needs to provide the file path to the .stti file to run that particular test case.
- /vmname=<Virtual Machine Name>
User will provide the Emulator virtual machine name for this parameter.
User can get the virtual machine name using Oracle VM VirtualBox manager settings. Settings‣General‣Basic‣ Name
This tag is used after providing the /input and /vmname to create reference images
This tag is used for comparing captured images with the reference images. Please note that similar to the ATT functionality using UI /compare and /createref cannot be used together.
- /result=[<Desired store location for the result>]
This tag is used along with /compare tag. User can either provide the file path to store the result file in .csv format or can just type /result in which case a default result file will be stored at the location of the .stti file.
Example: /result="/home/$USER/TestCaseResults/TestCase1Results/csv". Default file path for /input=" /home/$USER/ATT_Package/ATT/TestCase1/TestCase1.stti" is /home/$USER/ATT_Package/ATT/TestCase1/Result/TestCase1_<time_stamp>.csv. For instance a result file generated at 2:30pm on 21st December 2013 will have the following name TestCase1_20131221_14h30m35s.csv
|Test Suite Group||TSG|