top

Audio in C++

This tutorial performs a step by step decomposition of NaCl Audio demo application in C++.

Introduction

This article will discuss the structure of the basic audio playing NaCl application written in the C++ programming language. The source code can be found in Tizen Studio or you can click here. The downloaded zip file can be imported into Tizen Studio or extracted and compiled using a separately downloaded NaCl toolchain. Build instructions are available in the demo package.

Note

Please note that you need at least pepper_42 toolchain version to compile this sample application.

We recommend to go through this tutorial along with analyzing the downloaded source code. This analysis will only focus on the native part of the NaCl application, the web part (.html and .js code) will be omitted because it’s trivial and almost identical for every demo presented on this website. If you want to learn more about NaCl or how to implement a basic NaCl application from scratch, read Getting started with NaCl and How to create a sample NaCl application. We also suggest to see Hello World in C++.

There is also a cheat sheet for this demo available in Tizen Studio that explains the application functionality by highlighting most important code fragments. Cheat sheets are available from the Help menu.



The analyzed application is a simple audio playback application that can play a few different audio files, in the NaCl embed element object on the web page.

Code Analysis

In order to play audio samples, the application has to:

  • initialize the instance,
  • initialize audio context,
  • create audio control routines,
  • implement audio callback loop.

Before implementing any functions we should take a look at AudioInstance class member variables definition. These are as follows:

  • audio_ - instance of pp::Audio interface,
  • header_ - pointer to the header of the WAVE file,
  • file_data_bytes_ - data of the WAVE file,
  • sound_instances_ - vector of sound instances read from WAVE file,
  • active_sounds_- array of active sound instances,
  • file_names_ - vector of WAVE file names.

Implementation of the instance happens in the AudioInstance constructor. It initializes all necessary class members, initializes the Logger helper class and callback factory. Code for the AudioInstance constructor is presented below:

AudioInstance::AudioInstance(PP_Instance instance) :
    pp::Instance(instance),
    file_number_(0),
    active_sounds_number_(0),
    callback_factory_(this) {
  file_names_.reserve(kNumberOfInputSounds);
  Logger::InitializeInstance(this);
}

The Init() method, called right after the constructor, takes care of initializing audio context. Its code is presented below:

bool AudioInstance::Init(uint32_t /*argc*/, const char* /*argn*/[],
    const char* /*argv*/[]) {
  // getting sample frame count from the browser
  sample_frame_count_ =
    pp::AudioConfig::RecommendSampleFrameCount(this,
      PP_AUDIOSAMPLERATE_44100,
      SAMPLE_FRAME_COUNT);
  // creating an audio resource with configuration using 44,1kHz samplerate
  // and sample frame count form the browser
  pp::AudioConfig audio_config = pp::AudioConfig(this,
      PP_AUDIOSAMPLERATE_44100,
      sample_frame_count_);
  audio_ = pp::Audio(this,  // pointer to pp::Instance
      audio_config,
      AudioCallback,
      this); // argument of type void* for the AudioCallback call
  return true;
}

The first step performed by this function is to get recommended sample count in one frame for given audio parameters, which is used as audio buffer size in audio callback loop. This is done by calling RecommendSampleFrameCount() on the pp::Audio interface and then pp::AudioConfig object is initialized with this count. AudioConfig object represents the configuration of the audio playback. Using this configuration the pp::Audio object can be created and then it is used directly to play audio files. The initialization of pp::Audio takes as parameters the previously created configuration and also the AudioCallback() function pointer. AudioCallback() function is the main audio logic function, it performs mixing of audio data from multiple files and is called repeatedly during audio playback. After these steps the pp::Audiointerface is ready to be used.

Audio Control Routines

To establish proper audio control routines certain events need to be handled. On the web page there is a set of buttons and fields that allow controlling audio playback, they look like this:

There are four types of buttons available to the user:

  • Load button - sends a message via PostMessage() to the NaCl plugin that starts file loading and contains the path of the .wav file to be loaded
  • Play button - sends a message via PostMessage() to the NaCl plugin that starts the playback
  • Pause button - sends a message via PostMessage() to the NaCl plugin that pauses the playback
  • Stop button - sends a message via PostMessage() to the NaCl plugin that stops the playback

Each button sends a message to the NaCl plugin upon being pressed. Let’s examine theHandleMessage() method in the NaCl plugin which handles receiving those messages. Its code is presented below:

void AudioInstance::HandleMessage(const pp::Var& var_message) {
  if (var_message.is_string()) {
    const std::string message = var_message.AsString();
    if (StartsWith(message, kPlayCommand)) {
      // start playing
      Play(GetIdFromMessage(message, kPlayCommand));
    } else if (StartsWith(message, kPauseCommand)) {
      // pause
      Pause(GetIdFromMessage(message, kPauseCommand));
    } else if (StartsWith(message, kStopCommand)) {
      // stop
      Stop(GetIdFromMessage(message, kStopCommand));
    } else if (StartsWith(message, kLoadCommand)) {
      // load file
      PrepareReadingFile(message.substr(strlen(kLoadCommand)));
    }
  }
}

As you can see each of the received messages is directed to the specific method which is called with an argument that is either a path to the file in case of load button being pressed or ID of the audio file that should be controlled. Next we will examine these methods in detail.

Reading Audio Files

The PrepareReadingFile() method is responsible for loading the .wav file. Further details on how to load files in NaCl plugin see URLLoader in C++ article. When the file is loaded to a data buffer, the ReadWAVE() method is called, with the buffer as an argument. It is responsible for interpreting the data of the WAVE file. Code of that function is presented below:

void AudioInstance::ReadWAVE(const std::string& data) {
  Logger::Log("Interpreting WAVE data of file %s",
      file_names_[file_number_].c_str());

  // clean and create buffer for interpreting data read from file
  file_data_bytes_.reset(new char[data.size() + 1]);

  // copying data read from file to buffer
  std::copy(data.begin(), data.end(), file_data_bytes_.get());
  file_data_bytes_[data.size()] = '\0';

  // reading header data
  header_ = reinterpret_cast<WAVEFileHeader*>(file_data_bytes_.get());

  // check for malformed header
  std::string header_error = CheckForWAVEHeaderErrors(*header_);
  if (!header_error.empty()) {
    Logger::Error("Unsupported file %s - bad header:\n%s",
        file_names_[file_number_].c_str(), header_error.c_str());

    PostMessage(CreateCommandMessage(kErrorMessage, file_number_));
    return;
  }
  std::unique_ptr<uint16_t> sample_data(reinterpret_cast<uint16_t*>(
      file_data_bytes_.get() + sizeof(WAVEFileHeader)));

  // create new sound instance, fill it's fields and add it
  // to a sound instances vector
  std::shared_ptr<SoundInstance> instance(new SoundInstance(file_number_,
      header_->num_channels, (data.size() - sizeof(WAVEFileHeader)) /
      sizeof(uint16_t), &sample_data));
  sound_instances_.insert(std::pair<int, std::shared_ptr<SoundInstance> >(
      instance->id_, instance));

  // depending on current file reading send appropriate control message
  PostMessage(CreateCommandMessage(kFileLoadedMessage, file_number_));

  Logger::Log("File %s data loaded",
      file_names_[file_number_].c_str());
}

This method starts by copying the input string to the vector of chars, then pointer to the data vector is cast to WAVEFileHeader structure pointer. This structure is used to determine whether the header is correctly formatted. If it isn’t, an error message is printed. When the header check succeeds, the address of the beginning of the rest of the data array is cast to the smart pointer of 16-bit unsigned integer. With these two pointers in place we can create a SoundInstance object which holds all of the audio data needed for playback. When the SoundInstance is created it is immediately pushed to the sound_instances_collection from where it’s used for playback.

Audio Control

The play routine allows starting or resuming the sound playback, it is handled by the Play() method which starts audio playback and marks the chosen sound as active. Its code is presented below:

void AudioInstance::Play(int number) {
  // start playing if nothing is active
  if (active_sounds_number_ == 0) {
    audio_.StartPlayback();
    Logger::Log("Playing started");
  }
  assert((number - 1 >= 0) || (number - 1 < kNumberOfInputSounds));
  active_sounds_[number - 1] = true;
  ++active_sounds_number_;

  // send message to browser, that playing started
  PostMessage(CreateCommandMessage(kPlayMessage, number));
}

A check is performed whether all sound files are inactive. If it’s true, the StartPlayback() method is called on the audio interface, which begins the audio callback loop. Then the sound of the provided number is marked as active and number of active sounds is increased. This starts the playback of the sound from the audio file of the provided number. On completion a message is sent to JavaScript which informs that the playback was started.

Pause() and Stop() methods look very similar to each other and perform almost the same operations. In these methods the number of active sounds is decreased and the chosen sound is marked as inactive. As a result the file will stop playing. Next, if there are no active sound files, an audio callback loop is stopped by calling StopPlayback() on the audio interface. There is one additional step in Stop()method which resets the sample_data_offset_ which results in playing the sample from the beginning the next time the play is requested. Both functions on completion send a message to JavaScript which informs it that the playback was stopped/paused. Code is presented below:

void AudioInstance::Pause(int number) {
  // set clicked sound as not active
  --active_sounds_number_;
  assert((number - 1 >= 0) || (number - 1 < kNumberOfInputSounds));
  active_sounds_[number - 1] = false;

  // stop playing if nothing is active
  if (active_sounds_number_ == 0) {
    audio_.StopPlayback();
    Logger::Log("Playing stopped");
  }
  // send message to browser, that playing is paused
  PostMessage(CreateCommandMessage(kPauseMessage, number));
}

void AudioInstance::Stop(int number) {
  // set clicked sound as not active
  assert((number - 1 >= 0) || (number - 1 < kNumberOfInputSounds));
  if (active_sounds_[number - 1]) {
    --active_sounds_number_;
    active_sounds_[number - 1] = false;
  }
  // stop playing if nothing is active
  if (active_sounds_number_ == 0) {
    audio_.StopPlayback();
    Logger::Log("Playing stopped");
  }

  // set offset to 0 to start playing from the beginning
  // the next time play is requested
  sound_instances_[number]->sample_data_offset_ = 0;

  // send message to browser, that playing stopped
  PostMessage(CreateCommandMessage(kStopMessage, number));
}

Audio Callback Loop

The last thing to implement is the the audio callback loop. It is responsible for filling the audio buffer with sample data and mixing data from multiple sources in real time. This function is called periodically by the browser. Size of the buffer is decided when creating pp::Audio interface. Each iteration fills the buffer with the next chunk of data. The code of the function is presented below:

void AudioInstance::AudioCallback(void* samples, uint32_t buffer_size,
    void* data) {

  // instance pointer is passed when creating Audio resource
  AudioInstance* audio_instance = reinterpret_cast<AudioInstance*>(data);

  // browser audio buffer to fill
  int16_t* buff = reinterpret_cast<int16_t*>(samples);
  memset(buff, 0, buffer_size);

  assert(audio_instance->active_sounds_number_ > 0);
  // compute volume of sound instances
  const double volume = 1.0 / sqrt(audio_instance->active_sounds_number_);
  // Make sure we can't write outside the buffer
  assert(buffer_size >= (sizeof(*buff) * MAX_CHANNELS_NUMBER *
      audio_instance->sample_frame_count_));

  for (SoundInstances::iterator it =
      audio_instance->sound_instances_.begin(); it !=
          audio_instance->sound_instances_.end(); ++it) {
    std::shared_ptr<SoundInstance> instance(it->second);

    // main loop for writing samples to audio buffer
    for (size_t i = 0;  i < audio_instance->sample_frame_count_; ++i) {
      // if there are samples to play
      if (audio_instance->active_sounds_[instance->id_ - 1] &&
        instance->sample_data_offset_ < instance->sample_data_size_) {
        audio_instance->SafeAdd(&buff[2 * i], volume *
            (int16_t)instance->sample_data_[instance->sample_data_offset_]);
        // for mono sound each sample is passed to both channels, but for
        // stereo samples are written successively to all channels
        if (instance->channels_ == 2) {
          ++instance->sample_data_offset_;
        }
        audio_instance->SafeAdd(&buff[2 * i + 1],  volume *
            (int16_t)instance->sample_data_[instance->sample_data_offset_]);
        ++instance->sample_data_offset_;
      }
    }

    // playing finished so stop it
    if (instance->sample_data_offset_ == instance->sample_data_size_) {
      // Normally we should avoid Pepper API calls in audio callback,
      // as it may result in an audio callback thread being swapped out
      // which leads to audio dropouts. Here the Pepper API is called
      // when audio data ends, so a dropout wouldn't bother us.
      //
      audio_instance->PostMessage(audio_instance->CreateCommandMessage(
          kReturnStopMessage, instance->id_));
    }
  }
}

As the first step, the function performs necessary casting of the pointers to the AudioInstance and to the buffer we need to fill with audio data, which is filled with zeroes. Next, volume modifier is calculated which is equal to the inversed square root of active channel count. This will ensure that the overall volume remain the same, regardless of the number of played sounds.

Then the process of filling the buffer starts. For each SoundInstance in sound_instances_map a loop is started. In this loop:

  • the next chunk of data is written to the audio buffer, differently for mono and stereo sounds,
  • a check is performed whether the sound reached its end.
Note

Adding to the buffer is done by using SafeAdd() function, which clamps the values in the buffer to the possible minimum and maximum value of single sample.

In each data filling loop iteration data offset is incremented in each SoundInstance, which allows resuming buffer filling on the next AudioCallback()call.

If the audio file reached its end, a message is sent to the JavaScript informing it that playing for that SoundInstance has to be stopped.

Summary

This application can be compiled using Tizen Studio or manually with the make program. The application compiled manually can be run in Google Chrome. To run it on Smart TV Emulator you have to build it with Tizen Studio.

When the NaCl module loads, the Load buttons beside sample sound file names will be enabled and you will be able to play provided sample sound files.