Voice Interaction API

Samsung smart TVs allow developers to use voice commands such as Navigation, Search, Selection and Media Control to control their application. The Voice Assistant in the TV can be Bixby or other assistants. Regardless of which assistant is used, the application will interact with it via the Voice Interaction APIs. For best results, we recommend the application to be implemented with all the features that are described in this document.


Declare Privilege

To use the Voice Interaction API, the following privileges must be declared in the manifest file of the application. If they are not declared, all usage of these APIs are blocked, and security exceptions are thrown. To use these privileges, the minimum ‘required_version’ is 6.0.

  • http://developer.samsung.com/privilege/voicecontrol
  • http://developer.samsung.com/privilege/microphone
<?xml version="1.0" encoding="UTF-8"?>
<widget xmlns="http://www.w3.org/ns/widgets" xmlns:tizen="http://tizen.org/ns/widgets" id="http://yourdomain/VoiceInteractionWebSample" version="1.0.0" viewmodes="maximized">
    <access origin="*" subdomains="true"></access>
    <tizen:application id="AbCDEfgHrJ.VoiceInteractionWebSample" package="AbCDEfgHrJ" required_version="6.0"/>
    <content src="index.html"/>
    <feature name="http://tizen.org/feature/screen.size.normal.1080.1920"/>
    <icon src="icon.png"/>
    <name>VoiceInteractionWebSample</name>
    <tizen:privilege name="http://developer.samsung.com/privilege/voicecontrol"/>
    <tizen:privilege name="http://developer.samsung.com/privilege/microphone"/>
    <tizen:profile name="tv-samsung"/>
    <tizen:setting screen-orientation="landscape" context-menu="enable" encryption="disable" install-location="auto" hwkey-event="enable"/>
</widget>

Tizen Configuration Editor


Using webapis.voiceinteraction

The application must be implemented with callback functions to let the Voice Assistant send the commands to it. After getting ready, it is necessary to set the callbacks with the function webapis.voiceinteraction.setCallback, and call the function webapis.voiceinteraction.listen to start the interaction with the Voice Assistant. Except for the onupdatestate, onupdatecontentcontext callback, the return value is a Boolean value to specify whether the callback is processed or not. If the application returns the false value or there is no registered callback function for the utterance, the Samsung TV performs it’s default action for each callback. Please refer to the tvVoiceInterface project code for the usage.

Sample Application

var funcObject = {
  onupdatestate: function() {
    return "List";
  },
  onnavigation: function(vn) {
    console.log("onnavigation" + vn);
    return true;
  },
  onselection: function(voiceSelection) {
    console.log("OnSelection" + voiceSelection);
    return true;
  }
};


window.onload(function() {
  try {
    var version = webapis.voiceinteraction.getVersion();
    console.log("VoiceInteraction Version : " + version);
    webapis.voiceinteraction.setCallback(funcObject);
    webapis.voiceinteraction.listen();
  } catch (e) {
    console.log("exception [" + e.code + "] name: " + e.name + " message: " + e.message);
  }
});

Samsung Smart TV Web API

Application Status

Required Tizen Version : 6.0

Developers must specify the current application status to the onupdatestate function to get a proper voice control from the Voice Assistant. This function is called right before processing every utterance, so that the Voice Assistant can be aware of the application status dynamically. Depending on the status, the callback function called could be different. Therefore, it is important to return the real status of the application.

VoiceApplicationState Description Feature
"None" Application is int its default status. Same as "Player" status
"List" The application is showing something in list. During this status, the utterances "Play this" or "Play" will call onselection(0)
The utterance "Previous" will call onnavigation("NAV_PREVIOUS")
"Player The application is playing a piece of content. During this status, the utterance "Play this" will call onplay()
webapis.voiceinteraction.setCallback({
  onupdatestate: function() {
    console.log("Assistant tries to get app state");
    return "List";
  }
});

Changing Application Status

Required Tizen Version : 6.0

To support the utterance of shortcut commands, the application must have the onchangeappstate callback.

Sample Utterances TV Default Actions Callbacks VoiceApplicationState
"Go to Home" Launches the Samsung TV FirstScreen boolean ? onchangeappstate ( VoiceApplicationState state ) "Home"
"Go to Settings" Launches the Samsung TV Menu boolean ? onchangeappstate ( VoiceApplicationState state ) "Setting"
"Go to Search" Launches the Samsung TV Search Application boolean ? onchangeappstate ( VoiceApplicationState state ) "Search"
webapis.voiceinteraction.setCallback({
  onupdatestate: function() {
    console.log("Assistant tries to get app state");
    return "None";
  },
  onchangeappstate: function(state) {
    console.log("onchangeappstate : " + state);
    var bSupport = true;
    switch (state) {
      case "Home":
        //Go to App's Home
        break;
      default:
        bSupport = false;
        break;
    }
    return bSupport;
  }
});

Media Control

Required Tizen Version : 6.0

To support the media control via voice, the application must have the following callbacks. Otherwise, the Key Event will be generated.

Sample Utterances TV Default actions Callbacks Param
"Play" Generates the Remote Control Key, "MediaPlay" boolean ? onplay ( )
"Stop" Generates the Remote Control Key, "MediaStop" boolean ? onstop ( )
"Fast forward" Generates the Remote Control Key, "MediaFastForward" boolean ? onfastforward ( )
"Rewind" Generates the Remote Control Key, "MediaRewind" boolean ? onrewind ( )
"Pause" Generates the Remote Control Key, "MediaPlayPause" boolean ? onpause ( )
"Exit" Generates the Remote Control Key, "Exit" boolean ? onexit ( )
"Play the previous video" - boolean ? onchangeprevioustrack ( )
"Play the next video" - boolean ? onchangenexttrack ( )
"Restart the video" - boolean ? onrestart ( )
"Go back 10 minutes" - boolean ? onskipbackward ( long offsetSeconds ) The relative position in seconds
"Go forward by 5 minutes" - boolean ? onskipforward ( long offsetSeconds ) The relative position in seconds
"Play from 1 hour 28 minutes" - boolean ? onsetplayposition ( long position ) The absolute position in seconds
"Turn on the subtitles" - boolean ? onchangesubtitlemode ( MediaFunctionMode mode ) "MEDIA_FUNCTION_ON" / "MEDIA_FUNCTION_OFF"
"Activate shuffle" - boolean ? onchangeshufflemode ( MediaFunctionMode mode ) "MEDIA_FUNCTION_ON" / "MEDIA_FUNCTION_OFF"
"Screen fill on" - boolean ? onchangescreenfitmode ( MediaFunctionMode mode ) "MEDIA_FUNCTION_ON" / "MEDIA_FUNCTION_OFF"
"Zoom in" - boolean ? onzoom ( MediaZoomMode zoom ) "MEDIA_ZOOM_IN" / "MEDIA_ZOOM_OUT"
"Rotate screen anticlockwise" - boolean ? onrotate ( MediaRotateMode direction ) "MEDIA_ROTATE_LEFT" / "MEDIA_ROTATE_RIGHT"
"Enable the 360 view" - boolean ? onchange360mode ( MediaFunctionMode mode ) "MEDIA_FUNCTION_ON" / "MEDIA_FUNCTION_OFF"
"Repeat once" - boolean ? onchangerepeatmode ( MediaRepeatMode mode ) "MEDIA_REPEAT_OFF" / "MEDIA_REPEAT_ONE" / "MEDIA_REPEAT_ALL"
"Continue watching on (App Title)" Launches the application boolean ? oncontinuewatching ( )
webapis.voiceinteraction.setCallback({
  onupdatestate: function() {
    console.log("Assistant tries to get app state");
    return "Player";
  },
  onplay: function() {
    console.log("OnPlay called");
    return true;
  },
  onstop: function() {
    console.log("OnStop called");
    return true;
  },
  onchangeprevioustrack: function() {
    console.log("onchangeprevioustrack");
    return true;
  },
  onskipbackward: function(offsetSeconds) {
    console.log("onskipbackward : " + offsetSeconds);
    return true;
  },
  onsetplayposition: function(position) {
    console.log("onsetplayposition : " + position);
    return true;
  },
  onchangesubtitlemode: function(mode) {
    console.log("onchangesubtitlemode");
    switch (mode) {
      case "MEDIA_FUNCTION_ON":
        console.log("Function ON");
        break;
      default:
        console.log("Function OFF");
        break;
    }
    return true;
  },
  oncontinuewatching: function() {
    console.log("oncontinuewatching");
    return true;
  }
});   

Samsung Smart TV Remote Control

Required Tizen Version : 6.0

To support navigation and selection controls via voice, the application should have the following callbacks. Otherwise, the Key Event will be generated.

Sample Utterances TV Default Actions Callbacks Param
"Go to left"
"Move to right"
Generates Remote Control Key Events, "ArrowRight", "ArrowLeft", "ArrowUp", "ArrowDown" boolean ? onnavigation ( VoiceNavigation voiceNavigation ) "NAV_LEFT", "NAV_RIGHT", "NAV_UP", "NAV_DOWN"
"Show me more"
"Next page"
- boolean ? onnavigation ( VoiceNavigation voiceNavigation ) "NAV_SHOW_MORE", "NAV_PREVIOUS", "NAV_NEXT"
"Select this" Generates Remote Control Key Events, "Enter" boolean ? onselection ( long voiceSelection ) 0
"Select the first one"
"Select the second one"
"Select the last one"
- boolean ? onselection ( long voiceSelection ) 1 ~ 60 (Ordinal Index) -1 (Last)
webapis.voiceinteraction.setCallback({
  onupdatestate: function() {
    console.log("Assistant tries to get app state");
    return "List";
  },
  onnavigation: function(voiceNavigation) {
    var bSupport = true;
    console.log("onnavigation : " + voiceNavigation);
    switch (voiceNavigation) {
      case "NAV_PREVIOUS ":
        //Previous
        break;
      case "NAV_NEXT":
        //Next
        break;
      case "NAV_LEFT":
        //Left
        break;
      case "NAV_RIGHT":
        //Right
        break;
      case "NAV_UP":
        //Up
        break;
      case "NAV_DOWN":
        //Down
        break;
      case "NAV_SHOW_MORE":
        //ShowMore
        break;
      default:
        bSupport = false;
        break;
    }
    return bSupport;
  },
  onselection: function(voiceSelection) {
    var bSupport = true;
    console.log("onselection : " + voiceSelection);
    switch (voiceSelection) {
      case -1:
        //Select the last item of this page
        break;
      case 0:
        //Select this item on focus
        break;
      default: {
        if (voiceSelection >= 1) {
          //Select the (voiceSelection)th item
          //index of the first item is 1
          console.log("For Ordinal : " + voiceSelection);
        } else {
          bSupport = false;
        }
      }
      break;
    }
    return bSupport;
  }
});

Samsung Smart TV Remote Control

Required Tizen Version : 6.0

For a contents search provided by the application, it can receive a search utterance to show the search results. In order to support the search via voice, the application must be registered to Samsung TV's companion search application and have the following callbacks. The OnSearch parameter has a specific format. To learn more about it, please refer to the “Appendix B List of Search Term Fields”.

Sample Utterance TV Default Action Callback Param
"Search for action movie" Launches Samsung TV Search application boolean ? onsearch ( VoiceSearchTerm voiceSearchTerm ) The object contains the information via Voice
webapis.voiceinteraction.setCallback({
  onupdatestate: function() {
    console.log("Assistant tries to get app state");
    return "Home";
  },
  onsearch: function(voiceSearchTerm) {
    console.log("OnSearch : " + JSON.stringify(voiceSearchTerm));
    var title = webapis.voiceinteraction.getDataFromSearchTerm(voiceSearchTerm, "SEARCH_TERM_TITLE");
    var genre = webapis.voiceinteraction.getDataFromSearchTerm(voiceSearchTerm, "SEARCH_TERM_GENRE");
    console.log("Request to search " + title + ", " + genre);
    return true;
  }
}); 

Title Selection

Required Tizen Version : 6.5

Available Locale : ko-KR, en-US
*Other locale will be supported in upcoming releases.

Title selection refers to an utterance in the format "Select {Title Name}". To support the title selection via voice, the following two callbacks are required : onrequestcontentcontext and ontitleselection. The onrequestcontentcontext callback provides the information regarding the content of the screen (such as title and position) to the Samsung TV. This method will be invoked at the beginning of every utterance. The response consists of positionX(int), positionY(int), title(string), alias(string array) and bFocused(bool). The title property is used for ASR conversion and the other properties are used to set the priority of each title. The OnTitleSelection callback receives the title name from the utterance.

Sample Utterances TV Default Actions Callbacks Param
- - DOMString? onrequestcontentcontext() -
"Select (Title Name)" - boolean? ontitleselection(DOMString title) Title Name
    webapis.voiceinteraction.setCallback({
        onupdatestate : function () {
          console.log("Assistant tries to get app state");
          return "List";
        },
        onrequestcontentcontext : function () {
          console.log("onrequestcontentcontext ");
          var result = [];
          try {
              var item = {
                             "positionX" : 0,
                             "positionY" : 0,
                             "title" : "Title Text1",
                             "alias" : ["Title1", "My Text1"],
                             "bFocused" : true
                         };
              result.push(item);
              var item2 = {
                             "positionX" : 1,
                             "positionY" : 0,
                             "title" : "Title Text2",
                             "alias" : ["Title2", "My Text2"],
                             "bFocused" : false
                         };
              result.push(item2);
              result.push(webapis.voiceinteraction.buildVoiceInteractionContentContextItem(2,0,"Title Text3", ["Title3", "My Text3"], false));
            } catch (e) {
              console.log("exception [" + e.code + "] name: " + e.name + " message: " + e.message);
            }
            return webapis.voiceinteraction.buildVoiceInteractionContentContextResponse(result);
        },
        ontitleselection : function (title) {
          console.log("ontitleselection" + title);
          return true;
        }
    });
  

Applying a Developer Key

Some callbacks are not called by default such as onsearch. To receive all the callback signals on the development stage, request a key from Samsung and apply the key to the Manifest of your Tizen Application. The following example shows how to add the "http://developer.samsung.com/tizen/metadata/support-vif-dev-feature" key to the manifest file.

config.xml

<?xml version="1.0" encoding="UTF-8"?>
<widget xmlns="http://www.w3.org/ns/widgets" xmlns:tizen="http://tizen.org/ns/widgets" id="http://yourdomain/VoiceInteractionWebSample" version="1.0.0" viewmodes="maximized">
    <access origin="*" subdomains="true"></access>
    <tizen:application id="AbCDEfgHrJ.VoiceInteractionWebSample" package="AbCDEfgHrJ" required_version="6.0"/>
    <content src="index.html"/>
    <feature name="http://tizen.org/feature/screen.size.normal.1080.1920"/>
    <icon src="icon.png"/>
        <tizen:metadata key="http://developer.samsung.com/tizen/metadata/support-vif-dev-feature" value="6926643b2761dc5cdbe394f4800300b29d106b5717c11368b103699efe27b7b3"/>
    <name>VoiceInteractionWebSample</name>
    <tizen:privilege name="http://developer.samsung.com/privilege/voicecontrol"/>
    <tizen:privilege name="http://developer.samsung.com/privilege/microphone"/>
    <tizen:profile name="tv-samsung"/>
    <tizen:setting screen-orientation="landscape" context-menu="enable" encryption="disable" install-location="auto" hwkey-event="enable"/>
</widget>
 

Tizen Configuration Editor


Appendix A. List of VoiceInteraction Callbacks by Voice Assistants

Example Utterances TV Default Actions Callbacks Param
For every utterance Regards its status as not ready for interacting with the Voice Assistant VoiceApplicationState ? onupdatestate ( )
"Go to Home" Launches the Samsung TV FirstScreen boolean ? onchangeappstate ( VoiceApplicationState state ) "Home"
"Go to Settings" Launches the Samsung TV Menu boolean ? onchangeappstate ( VoiceApplicationState state ) "Setting"
"Go to Search" Launches the Samsung TV Search Application boolean ? onchangeappstate ( VoiceApplicationState state ) "Search"
"Play" Generates the Remote Control Key, "MediaPlay" boolean ? onplay ( )
"Stop" Generates the Remote Control Key, "MediaStop" boolean ? onstop ( )
"Fastforward" Generates the Remote Control Key, "MediaFastForward" boolean ? onfastforward ( )
"Rewind" Generates the Remote Control Key, "MediaRewind" boolean ? onrewind ( )
"Pause" Generates the Remote Control Key, "MediaPlayPause" boolean ? onpause ( )
"Exit" Generates the Remote Control Key, "Exit" boolean ? onexit ( )
"Play the previous video" - boolean ? onchangeprevioustrack ( )
"Play the next video" - boolean ? onchangenexttrack ( )
"Restart the video" - boolean ? onrestart ( )
"Go back 10 minutes" - boolean ? onskipbackward ( long offsetSeconds ) The relative position in seconds
"Go forward by 5 minutes" - boolean ? onskipforward ( long offsetSeconds ) The relative position in seconds
"Play from 1 hour 28 minutes" - boolean ? onsetplayposition ( long position ) The absolute position in seconds
"Turn on the subtitles" - boolean ? onchangesubtitlemode ( MediaFunctionMode mode ) "MEDIA_FUNCTION_ON" / "MEDIA_FUNCTION_OFF"
"Activate shuffle" - boolean ? onchangeshufflemode ( MediaFunctionMode mode ) "MEDIA_FUNCTION_ON" / "MEDIA_FUNCTION_OFF"
"Screen fill on" - boolean ? onchangescreenfitmode ( MediaFunctionMode mode ) "MEDIA_FUNCTION_ON" / "MEDIA_FUNCTION_OFF"
"Zoom in" - boolean ? onzoom ( MediaZoomMode zoom ) "MEDIA_ZOOM_IN" / "MEDIA_ZOOM_OUT"
"Rotate screen anticlockwise" - boolean ? onrotate ( MediaRotateMode direction ) "MEDIA_ROTATE_LEFT" / "MEDIA_ROTATE_RIGHT"
"Enable the 360 view" - boolean ? onchange360mode ( MediaFunctionMode mode ) "MEDIA_FUNCTION_ON" / "MEDIA_FUNCTION_OFF"
"Repeat once" - boolean ? onchangerepeatmode ( MediaRepeatMode mode ) "MEDIA_REPEAT_OFF" / "MEDIA_REPEAT_ONE" / "MEDIA_REPEAT_ALL"
"Continue watching on (App Title)" Launches the application boolean ? oncontinuewatching ( )
"Go to left"
"Move to right"
Generates Remote Control Key Events, "ArrowRight", "ArrowLeft", "ArrowUp", "ArrowDown" boolean ? onnavigation ( VoiceNavigation voiceNavigation ) "NAV_LEFT", "NAV_RIGHT", "NAV_UP", "NAV_DOWN"
"Show me more"
"Next page"
- boolean ? onnavigation ( VoiceNavigation voiceNavigation ) "NAV_SHOW_MORE", "NAV_PREVIOUS", "NAV_NEXT"
"Select this" Generates Remote Control Key Events, "Enter" boolean ? onselection ( long voiceSelection ) 0
"Select the first one"
"Select the second one"
"Select the last one"
- boolean ? onselection ( long voiceSelection ) 1 ~ 60 (Ordinal Index) 0 (This) -1 (Last)
"Search for action movie" Launches Samsung TV Search application boolean ? onsearch ( VoiceSearchTerm voiceSearchTerm ) The object contains the information via Voice
"Select (Title Name)" - DOMString? onrequestcontentcontext() boolean? ontitleselection(DOMString title) Title Text

Appendix B. List of Search Term Fields

Fields of Search Term

VoiceSearchTermField Description Example
SEARCH_TERM_UTTERANCE The field for the full utterance "Search for action movies"
SEARCH_TERM_TITLE The field for the title Title Name
SEARCH_TERM_GENRE The field for the genre "Action"
SEARCH_TERM_CAST The field for the cast Actor Name
SEARCH_TERM_CONTENT_TYPE The field for the content type ("Movie”/“TV”) "Movie"
SEARCH_TERM_RELEASE_DATE_FROM The field for the start date time(ISO 8601) "2019-04-01T00:00:01+00:00"
SEARCH_TERM_RELEASE_DATE_TO The field for the end date time(ISO 8601) "2019-04-01T00:00:01+00:00"

Getting Search Term Field Values

webapis.voiceinteraction.setCallback({
  onupdatestate: function() {
    console.log("Assistant tries to get app state");
    return "Home";
  },
  onsearch: function(voiceSearchTerm) {
    console.log("OnSearch : " + JSON.stringify(voiceSearchTerm));
    try {
      var title = webapis.voiceinteraction.getDataFromSearchTerm(voiceSearchTerm, "SEARCH_TERM_TITLE");
      var genre = webapis.voiceinteraction.getDataFromSearchTerm(voiceSearchTerm, "SEARCH_TERM_GENRE");
      console.log("Request to search " + title + ", " + genre);
    } catch (e) {
      console.log("exception [" + e.code + "] name: " + e.name + " message: " + e.message);
    }
    return true;
  }
});