Test Edge Drivers using SmartThings Test Suite
Objective
Learn how to identify and resolve issues when deploying Edge Drivers using SmartThings Test Suite.
Overview
SmartThings Test Suite is a tool for testing IoT device integrations within the SmartThings platform. This solution provides a seamless certification process, allowing developers of SmartThings hub connected devices to submit their products for certification without the need for manual testing. This accelerates the certification timeline and offers a more cost-effective path to certifying these devices.
The key feature of this self-testing tool is it contains an automated testing suite that covers critical certification criteria, ranging from functionality to performance tests. The tool also provides real-time feedback that gives detailed information on the device's compliance status, allowing for quick identification and resolution of any issues. Lastly, it has an intuitive and user-friendly interface that ensures a seamless experience for developers of all levels.
Set up your environment
You will need the following:
- Host PC running on Windows 10 (or higher) or Ubuntu 20.04 (x64)
- Visual Studio Code (latest version recommended)
- Devices connected on the same network:
- Android mobile device with SmartThings app installed (with Android 10 or higher)
- SmartThings Station (or SmartThings Hub) onboarded with Samsung account
- SmartThings v4 Multipurpose Sensor or Lightify Tunable White 60 Light Bulb
NoteMake sure that your devices are connected to your SmartThings app.
Sample Code
Here is a sample code for this Code Lab. Download it and start your learning experience!
Test Suite Sample Code (185.4 KB)
Install SmartThings CLI
You need to install SmartThings CLI as this is the main tool for developing apps and drivers for SmartThings Edge Drivers.
To install SmartThings CLI, open a web browser and download the smartthings.msi
installer from the latest release.
NoteFor other operating systems, download the appropriate zipped binary and install it on your system path.
Open the SmartThings CLI Setup in the downloaded file, then click Next.
Accept the license agreement terms, then click Next.
Select the destination path for installation and click Next.
To begin the installation process, click Install.
NoteThe Windows installer may display a warning titled Windows protected your PC. To continue the installation, click More info > Run anyway.
Complete the setup by clicking Finish.
To verify if SmartThings CLI is installed correctly, open the Command Prompt and run this command:
smartthings --version
View and run available commands for SmartThings CLI with this command:
smartthings --help
For a full list of commands, visit the SmartThings CLI Commands.
NoteThe SmartThings CLI supports an automatic login flow that launches a browser window, prompting the user to log in with Samsung account and grant the CLI permissions to access the user's account.
Start your project
After downloading and extracting the sample code containing the project files, click File > Open Folder in Visual Studio Code to open it.
Locate the sample code file directory and click Select Folder.
Once finished, the project files are seen on the Explorer menu.
Open your Command Prompt or Terminal and follow the corresponding instructions depending on your device availability. Make sure that the path directory in your CLI contains the project file.
In the Terminal, type the following command to build and upload your Edge Driver package to the SmartThings Cloud:
smartthings edge:drivers:package drivers/codelab-zigbee-contact
Create a new channel for your Edge Driver and enter the following channel details:
smartthings edge:channels:create
Channel name: SmartThings Test Suite Demo
Channel description: Channel for SDC2024
Channel terms of service URL: www.smartthings.com
Enroll your hub in your newly created channel and select the corresponding channel and hub:
smartthings edge:channels:enroll
Assign your driver to the created channel:
smartthings edge:channels:assign
Install the created Edge Driver from your channel to your own hub:
smartthings edge:drivers:install
Confirm that the correct version of the driver is present in your hub:
smartthings edge:drivers:installed
Select the Edge Driver for this device:
smartthings edge:drivers:switch
In the Terminal, type the following command to build and upload your Edge Driver package to the SmartThings Cloud:
smartthings edge:drivers:package drivers/codelab-zigbee-switch
Create a new channel for your Edge Driver and enter the following channel details:
smartthings edge:channels:create
Channel name: SmartThings Test Suite Demo
Channel description: Channel for SDC2024
Channel terms of service URL: www.smartthings.com
Enroll your hub in your newly created channel and select the corresponding channel and hub:
smartthings edge:channels:enroll
Assign your driver to the created channel:
smartthings edge:channels:assign
Install the created Edge Driver from your channel to your own hub:
smartthings edge:drivers:install
Confirm that the correct version of the driver is present in your hub:
smartthings edge:drivers:installed
Select the Edge Driver for this device:
smartthings edge:drivers:switch
Test your device
On your web browser, go to SmartThings Test Suite, login to your Samsung account and follow the corresponding instructions depending on your device availability.
On the Test Suite, look for your device, click Menu icon > Prepare new test.
Under the Compatible capabilities, select all capabilities except for Battery, and click Start.
During the test execution, perform the indicated user actions for every test case if there are any. It might cause incorrect test results if user actions are not performed.
TipYou may view the real-time sensor states of the device in the SmartThings mobile app.
View the Test Summary after the test. It returns a failed test that you are going to resolve in the next step.
On the Test Suite, look for your device, click Menu icon > Prepare new test.
Under the Compatible capabilities, select all capabilities.
After you've selected the capabilities, click Start.
WarningDuring the test execution, observe the behavior of the bulb. It might cause incorrect test results if automated tests are interrupted.
View the Test Summary after the test. It returns a failed test that you are going to resolve in the next step.
Resolve test failures
The test logs contain basic information about the test results and specific test cases, providing technical context to users for efficient troubleshooting.
Download the test logs by navigating to the bottom page of the Test Summary > Show full test details > Download log.
To understand the test logs, its structure follows this schema:
{Execution Timestamp} (Device Node Path) (Node Type) [Node State] : {Execution Message}
In the downloaded test log, two test cases failed with the following error:
[FAILED]: Initialize states Following states were not set correctly [contact:any other state than "Open" on: main:contactSensor]
[FAILED]: Send command and Validate Some events didn't happen: [contact:"Closed" on: main:contactSensor] Some states aren't final: [contact:"Closed" on: main:contactSensor]
In the first error log, it appears that the Test Suite cannot change the device's state to anything other than an open state. In the second error log, the Test Suite tries to change its state to closed, but to no avail.
It is confirmed in the capability definition that the contact sensor has only two states: open and closed. Therefore, the device is constantly in an open state and unable to change to a closed state.
With these information, you can start troubleshooting by going to drivers > codelab-zigbee-contact > multi-sensor > init.lua and look for incorrect code implementation with these keywords: Open
, Closed
, contactSensor
.
It can be seen in the zone_status_change_handler
and zone_status_handler
functions that there are code blocks on comment. This might be a result of someone developing this code have changed this part of code for debugging and forgot to uncomment this part.
Uncomment this code block from zone_status_change_handler
function:
if not device.preferences["certifiedpreferences.garageSensor"] then
contactSensor_defaults.ias_zone_status_change_handler(driver, device, zb_rx)
end
Uncomment this code block from zone_status_handler
function:
if not device.preferences["certifiedpreferences.garageSensor"] then
contactSensor_defaults.ias_zone_status_attr_handler(driver, device, zone_status, zb_rx)
end
Remove this line of code from zone_status_change_handler
and zone_status_handler
functions:
device:emit_event_for_endpoint(zb_rx.address_header.src_endpoint_value, capabilities.contactSensor.contact.open())
Save the file and update the driver by invoking the same CLI commands that were also used during the configuration of custom Edge Drivers:
smartthings edge:drivers:package drivers/codelab-zigbee-contact
smartthings edge:channels:assign
smartthings edge:drivers:install
Again, go to SmartThings Test Suite, select your device, click Menu > Prepare new test.
Ensure that all Compatible capabilities are selected, with the exception for Battery.
Again, start the test and perform indicated user actions for every test case if there are any.
Now, all tests are passed!
The test logs contain basic information about the test results and specific test cases, providing technical context to users for efficient troubleshooting.
Download the test logs by navigating to the bottom page of the Test Summary > Show full test details > Download log.
To understand the test logs, its structure follows this schema:
{Execution Timestamp} (Device Node Path) (Node Type) [Node State] : {Execution Message}
In the downloaded test log, one test case failed with the following error:
[FAILED]: Send command and Validate Some events didn't happen: [colorTemperature:maximum (7500K) on: main:colorTemperature] Some states aren't final: [colorTemperature:maximum (7500K) on: main:colorTemperature]
In the error log, it directs to an issue for setting an incorrect maximum colorTemperature
value. The configuration on your Edge Driver is set to 7500K. You can start to troubleshoot by looking for the bulb's color temperature rating either from the device packaging or the device manufacturer website.
With these information, you can start troubleshooting by going to drivers > codelab-zigbee-switch > profiles > rgbw-bulb.yml and look for lines that declares the colorTemperature
value.
Change the colorTemperature
range declaration:
range: [ 2700, 6500 ]
Save the file and update the driver by invoking the same CLI commands that were also used during the configuration of custom Edge Drivers:
smartthings edge:drivers:package drivers/codelab-zigbee-switch
smartthings edge:channels:assign
smartthings edge:drivers:install
Again, go to SmartThings Test Suite, select your device, click Menu > Prepare new test.
Under the Compatible capabilities, select all capabilities.
Again, start the test and perform indicated user actions for every test case if there are any.
Now, all tests are passed!
You're done!
Congratulations! You have successfully achieved the goal of this Code Lab. Now, you can test your Edge Driver for SmartThings devices using SmartThings Test Suite! If you're having trouble, you may download this file:
Test Suite Complete Code (184.6 KB)
To learn more about SmartThings Test Suite, visit: