1# Audio Decoding 2 3You can call the native APIs provided by the AudioCodec module to decode audio, that is, to decode media data into PCM streams. 4 5For details about the supported decoding capabilities, see [AVCodec Supported Formats](avcodec-support-formats.md#audio-decoding). 6 7**Usage Scenario** 8 9- Audio playback 10 11 Decode audio and transmit the data to the speaker for playing. 12- Audio rendering 13 14 Decode audio and transmit the data to the audio processing module for audio rendering. 15- Audio editing 16 17 Decode audio and transmit the data for audio editing (for example, adjusting the playback speed of a channel). Audio editing is performed based on PCM streams. 18> **NOTE** 19> 20> Streams generated in the MP3 audio encoding process cannot be directly decoded through the MP3 audio decoding process. The following process is recommended: PCM stream -> MP3 audio encoding -> muxing -> demuxing -> MP3 audio decoding. 21 22## How to Develop 23 24Read [AudioCodec](../../reference/apis-avcodec-kit/_audio_codec.md) for the API reference. 25 26Refer to the code snippet below to complete the entire audio decoding process, including creating a decoder, setting decoding parameters (such as the sampling rate, bit rate, and number of audio channels), and starting, refreshing, resetting, and destroying the decoder. 27 28During application development, you must call the APIs in the defined sequence. Otherwise, an exception or undefined behavior may occur. 29 30The figure below shows the call relationship of audio decoding. 31 32- The dotted line indicates an optional operation. 33 34- The solid line indicates a mandatory operation. 35 36 37 38### Linking the Dynamic Libraries in the CMake Script 39 40```cmake 41target_link_libraries(sample PUBLIC libnative_media_codecbase.so) 42target_link_libraries(sample PUBLIC libnative_media_core.so) 43target_link_libraries(sample PUBLIC libnative_media_acodec.so) 44``` 45 46### How to Develop 47 481. Add the header files. 49 50 ```cpp 51 #include <multimedia/player_framework/native_avcodec_audiocodec.h> 52 #include <multimedia/native_audio_channel_layout.h> 53 #include <multimedia/player_framework/native_avcapability.h> 54 #include <multimedia/player_framework/native_avcodec_base.h> 55 #include <multimedia/player_framework/native_avformat.h> 56 #include <multimedia/player_framework/native_avbuffer.h> 57 ``` 58 592. Create a decoder instance. In the code snippet below, **OH_AVCodec *** is the pointer to the decoder instance created. 60 61 ```cpp 62 // Namespace of the C++ standard library. 63 using namespace std; 64 // Create a decoder by name. 65 OH_AVCapability *capability = OH_AVCodec_GetCapability(OH_AVCODEC_MIMETYPE_AUDIO_MPEG, false); 66 const char *name = OH_AVCapability_GetName(capability); 67 OH_AVCodec *audioDec_ = OH_AudioCodec_CreateByName(name); 68 ``` 69 70 ```cpp 71 // Specify whether encoding is used. The value false means decoding. 72 bool isEncoder = false; 73 // Create a decoder by MIME type. 74 OH_AVCodec *audioDec_ = OH_AudioCodec_CreateByMime(OH_AVCODEC_MIMETYPE_AUDIO_MPEG, isEncoder); 75 ``` 76 77 ```cpp 78 // Initialize the queues. 79 class ADecBufferSignal { 80 public: 81 std::mutex inMutex_; 82 std::mutex outMutex_; 83 std::mutex startMutex_; 84 std::condition_variable inCond_; 85 std::condition_variable outCond_; 86 std::condition_variable startCond_; 87 std::queue<uint32_t> inQueue_; 88 std::queue<uint32_t> outQueue_; 89 std::queue<OH_AVBuffer *> inBufferQueue_; 90 std::queue<OH_AVBuffer *> outBufferQueue_; 91 }; 92 ADecBufferSignal *signal_; 93 ``` 94 953. Call **OH_AudioCodec_RegisterCallback()** to register callback functions. 96 97 Register the **OH_AVCodecCallback** struct that defines the following callback function pointers: 98 99 - **OH_AVCodecOnError**, a callback used to report a codec operation error 100 - **OH_AVCodecOnStreamChanged**, a callback used to report a codec stream change, for example, audio channel change 101 - **OH_AVCodecOnNeedInputBuffer**, a callback used to report input data required, which means that the decoder is ready for receiving data 102 - **OH_AVCodecOnNewOutputBuffer**, a callback used to report output data generated, which means that decoding is complete 103 104 You need to process the callback functions to ensure that the decoder runs properly. 105 106 ```cpp 107 // Implement the OH_AVCodecOnError callback function. 108 static void OnError(OH_AVCodec *codec, int32_t errorCode, void *userData) 109 { 110 (void)codec; 111 (void)errorCode; 112 (void)userData; 113 } 114 // Implement the OH_AVCodecOnStreamChanged callback function. 115 static void OnOutputFormatChanged(OH_AVCodec *codec, OH_AVFormat *format, void *userData) 116 { 117 (void)codec; 118 (void)format; 119 (void)userData; 120 } 121 // Implement the OH_AVCodecOnNeedInputBuffer callback function. 122 static void OnInputBufferAvailable(OH_AVCodec *codec, uint32_t index, OH_AVBuffer *data, void *userData) 123 { 124 (void)codec; 125 ADecBufferSignal *signal = static_cast<ADecBufferSignal *>(userData); 126 unique_lock<mutex> lock(signal->inMutex_); 127 signal->inQueue_.push(index); 128 signal->inBufferQueue_.push(data); 129 signal->inCond_.notify_all(); 130 // The input stream is sent to inBufferQueue_. 131 } 132 // Implement the OH_AVCodecOnNewOutputBuffer callback function. 133 static void OnOutputBufferAvailable(OH_AVCodec *codec, uint32_t index, OH_AVBuffer *data, void *userData) 134 { 135 (void)codec; 136 ADecBufferSignal *signal = static_cast<ADecBufferSignal *>(userData); 137 unique_lock<mutex> lock(signal->outMutex_); 138 signal->outQueue_.push(index); 139 signal->outBufferQueue_.push(data); 140 signal->outCond_.notify_all(); 141 // The index of the output buffer is sent to outQueue_. 142 // The decoded data is sent to outBufferQueue_. 143 } 144 signal_ = new ADecBufferSignal(); 145 OH_AVCodecCallback cb_ = {&OnError, &OnOutputFormatChanged, &OnInputBufferAvailable, &OnOutputBufferAvailable}; 146 int32_t ret = OH_AudioCodec_RegisterCallback(audioDec_, cb_, signal_); 147 if (ret != AVCS_ERR_OK) { 148 // Exception handling. 149 } 150 ``` 151 1524. (Optional) Call **OH_AudioCodec_SetDecryptionConfig** to set the decryption configuration. Call this API after the media key system information is obtained but before **Prepare()** is called. For details about how to obtain such information, see step 4 in [Media Data Demuxing](audio-video-demuxer.md). For details about DRM APIs, see [DRM](../../reference/apis-drm-kit/_drm.md). 153 154 Add the header files. 155 156 ```c++ 157 #include <multimedia/drm_framework/native_mediakeysystem.h> 158 #include <multimedia/drm_framework/native_mediakeysession.h> 159 #include <multimedia/drm_framework/native_drm_err.h> 160 #include <multimedia/drm_framework/native_drm_common.h> 161 ``` 162 Link the dynamic library in the CMake script. 163 164 ``` cmake 165 target_link_libraries(sample PUBLIC libnative_drm.so) 166 ``` 167 168 The following is the sample code: 169 ```c++ 170 // Create a media key system based on the media key system information. The following uses com.clearplay.drm as an example. 171 MediaKeySystem *system = nullptr; 172 int32_t ret = OH_MediaKeySystem_Create("com.clearplay.drm", &system); 173 if (system == nullptr) { 174 printf("create media key system failed"); 175 return; 176 } 177 178 // Create a media key session. 179 MediaKeySession *session = nullptr; 180 DRM_ContentProtectionLevel contentProtectionLevel = CONTENT_PROTECTION_LEVEL_SW_CRYPTO; 181 ret = OH_MediaKeySystem_CreateMediaKeySession(system, &contentProtectionLevel, &session); 182 if (ret != DRM_OK) { 183 // If the creation fails, refer to the DRM interface document and check logs. 184 printf("create media key session failed."); 185 return; 186 } 187 if (session == nullptr) { 188 printf("media key session is nullptr."); 189 return; 190 } 191 // Generate a media key request and set the response to the media key request. 192 // Set the decryption configuration, that is, set the decryption session and secure channel flag to the decoder. (Currently, the secure channel is not supported for audio decryption and therefore the secure channel flag should be set to false.) 193 bool secureAudio = false; 194 ret = OH_AudioCodec_SetDecryptionConfig(audioDec_, session, secureAudio); 195 ``` 196 1975. Call **OH_AudioCodec_Configure()** to configure the decoder. 198 199 Key values of configuration options are described as follows: 200 201 | | Description | AAC | FLAC| Vorbis | MPEG | G711mu | AMR (AMR-NB and AMR-WB) | APE | 202 | ---------------------------- | :----------------------------------------------------------: | :--------------------------------: | :--: | :--------------------------------: | :--: | :-----------------: | :-------------------------------: | :-------------------------------: | 203 | OH_MD_KEY_AUD_SAMPLE_RATE | Sampling rate | Mandatory | Mandatory| Mandatory | Mandatory| Mandatory | Mandatory | Mandatory | 204 | OH_MD_KEY_AUD_CHANNEL_COUNT | Number of audio channels | Mandatory | Mandatory| Mandatory | Mandatory| Mandatory | Mandatory | Mandatory | 205 | OH_MD_KEY_MAX_INPUT_SIZE | Maximum input size | Optional | Optional| Optional | Optional| Optional | Optional | Optional | 206 | OH_MD_KEY_AAC_IS_ADTS | ADTS or not | Optional (Default value: 1 latm) | - | - | - | - | - | - | 207 | MD_KEY_AUDIO_SAMPLE_FORMAT | Output audio stream format | Optional (SAMPLE_S16LE, SAMPLE_F32LE)| - | Optional (SAMPLE_S16LE, SAMPLE_F32LE)| Optional| Optional (default: SAMPLE_S16LE)| Optional (SAMPLE_S16LE, SAMPLE_F32LE)| Optional | 208 | MD_KEY_BITRATE | Optional | Optional | Optional| Optional | Optional| Optional | Optional | Optional | 209 | MD_KEY_IDENTIFICATION_HEADER | ID Header | - | - | Mandatory (Either this parameter or **MD_KEY_CODEC_CONFIG** must be set.) | - | - | - | - | 210 | MD_KEY_SETUP_HEADER | Setup Header | - | - | Mandatory (Either this parameter or **MD_KEY_CODEC_CONFIG** must be set.) | - | - | - | - | 211 | MD_KEY_CODEC_CONFIG | MD_KEY_SETUP_HEADERID Header+Common Header+Setup Header stitching| - | | Mandatory (Either this parameter or the combination of **MD_KEY_IDENTIFICATION_HEADER** and **MD_KEY_SETUP_HEADER** must be selected.) | - | - | - | - | 212 213 The sample below lists the value range of each audio decoding type. 214 | Audio Decoding Type| Sampling Rate (Hz) | Audio Channel Count| 215 | ----------- | ---------------------------------------------------------------------------------------------- | :----: | 216 | AAC | 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000, 64000, 88200, 96000 | 1–8 | 217 | FLAC | 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000, 64000, 88200, 96000, 192000 | 1–8 | 218 | Vorbis | 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000, 64000, 88200, 96000, 176400, 192000| 1–8 | 219 | MPEG (MP3) | 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000 | 1–2 | 220 | G711mu | 8000 | 1 | 221 | AMR (amrnb) | 8000 | 1 | 222 | AMR (amrwb) | 16000 | 1 | 223 | APE | 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000, 64000, 88200, 96000, 176400, 192000| 1–2 | 224 <!--RP4--> 225 <!--RP4End--> 226 227 ```cpp 228 // Set the decoding resolution. 229 int32_t ret; 230 // (Mandatory) Configure the audio sampling rate. 231 constexpr uint32_t DEFAULT_SAMPLERATE = 44100; 232 // (Optional) Configure the audio bit rate. 233 constexpr uint32_t DEFAULT_BITRATE = 32000; 234 // (Mandatory) Configure the number of audio channels. 235 constexpr uint32_t DEFAULT_CHANNEL_COUNT = 2; 236 // (Optional) Configure the maximum input length. 237 constexpr uint32_t DEFAULT_MAX_INPUT_SIZE = 1152; 238 // Configure whether to use ADTS decoding (optional for AAC decoding). 239 constexpr uint32_t DEFAULT_AAC_TYPE = 1; 240 OH_AVFormat *format = OH_AVFormat_Create(); 241 // Set the format. 242 OH_AVFormat_SetIntValue(format, OH_MD_KEY_AUD_SAMPLE_RATE, DEFAULT_SAMPLERATE); 243 OH_AVFormat_SetIntValue(format, OH_MD_KEY_BITRATE, DEFAULT_BITRATE); 244 OH_AVFormat_SetIntValue(format, OH_MD_KEY_AUD_CHANNEL_COUNT, DEFAULT_CHANNEL_COUNT); 245 OH_AVFormat_SetIntValue(format, OH_MD_KEY_MAX_INPUT_SIZE, DEFAULT_MAX_INPUT_SIZE); 246 OH_AVFormat_SetIntValue(format, OH_MD_KEY_AAC_IS_ADTS, DEFAULT_AAC_TYPE); 247 // Configure the decoder. 248 ret = OH_AudioCodec_Configure(audioDec_, format); 249 if (ret != AV_ERR_OK) { 250 // Exception handling. 251 } 252 ``` 253 2546. Call **OH_AudioCodec_Prepare()** to prepare internal resources for the decoder. 255 256 ```cpp 257 ret = OH_AudioCodec_Prepare(audioDec_); 258 if (ret != AV_ERR_OK) { 259 // Exception handling. 260 } 261 ``` 262 2637. Call **OH_AudioCodec_Start()** to start the decoder. 264 265 ```c++ 266 unique_ptr<ifstream> inputFile_ = make_unique<ifstream>(); 267 unique_ptr<ofstream> outFile_ = make_unique<ofstream>(); 268 // Open the path of the binary file to be decoded. 269 inputFile_->open(inputFilePath.data(), ios::in | ios::binary); 270 // Configure the path of the output file. 271 outFile_->open(outputFilePath.data(), ios::out | ios::binary); 272 // Start decoding. 273 ret = OH_AudioCodec_Start(audioDec_); 274 if (ret != AV_ERR_OK) { 275 // Exception handling. 276 } 277 ``` 278 2798. (Optional) Call **OH_AVCencInfo_SetAVBuffer()** to set the Common Encryption Scheme (CENC) information. 280 281 If the content being played is DRM encrypted and demuxing is performed by the upper-layer application, call **OH_AVCencInfo_SetAVBuffer()** to set the CENC information to the AVBuffer so that the media data can be decrypted in the AVBuffer. 282 283 Add the header file. 284 285 ```c++ 286 #include <multimedia/player_framework/native_cencinfo.h> 287 ``` 288 Link the dynamic library in the CMake script. 289 290 ``` cmake 291 target_link_libraries(sample PUBLIC libnative_media_avcencinfo.so) 292 ``` 293 294 The following is the sample code: 295 ```c++ 296 auto buffer = signal_->inBufferQueue_.front(); 297 uint32_t keyIdLen = DRM_KEY_ID_SIZE; 298 uint8_t keyId[] = { 299 0xd4, 0xb2, 0x01, 0xe4, 0x61, 0xc8, 0x98, 0x96, 300 0xcf, 0x05, 0x22, 0x39, 0x8d, 0x09, 0xe6, 0x28}; 301 uint32_t ivLen = DRM_KEY_IV_SIZE; 302 uint8_t iv[] = { 303 0xbf, 0x77, 0xed, 0x51, 0x81, 0xde, 0x36, 0x3e, 304 0x52, 0xf7, 0x20, 0x4f, 0x72, 0x14, 0xa3, 0x95}; 305 uint32_t encryptedBlockCount = 0; 306 uint32_t skippedBlockCount = 0; 307 uint32_t firstEncryptedOffset = 0; 308 uint32_t subsampleCount = 1; 309 DrmSubsample subsamples[1] = { {0x10, 0x16} }; 310 // Create a CencInfo instance. 311 OH_AVCencInfo *cencInfo = OH_AVCencInfo_Create(); 312 if (cencInfo == nullptr) { 313 // Exception handling. 314 } 315 // Set the decryption algorithm. 316 OH_AVErrCode errNo = OH_AVCencInfo_SetAlgorithm(cencInfo, DRM_ALG_CENC_AES_CTR); 317 if (errNo != AV_ERR_OK) { 318 // Exception handling. 319 } 320 // Set KeyId and Iv. 321 errNo = OH_AVCencInfo_SetKeyIdAndIv(cencInfo, keyId, keyIdLen, iv, ivLen); 322 if (errNo != AV_ERR_OK) { 323 // Exception handling. 324 } 325 // Set the sample information. 326 errNo = OH_AVCencInfo_SetSubsampleInfo(cencInfo, encryptedBlockCount, skippedBlockCount, firstEncryptedOffset, 327 subsampleCount, subsamples); 328 if (errNo != AV_ERR_OK) { 329 // Exception handling. 330 } 331 // Set the mode. KeyId, Iv, and SubSamples have been set. 332 errNo = OH_AVCencInfo_SetMode(cencInfo, DRM_CENC_INFO_KEY_IV_SUBSAMPLES_SET); 333 if (errNo != AV_ERR_OK) { 334 // Exception handling. 335 } 336 // Set CencInfo to the AVBuffer. 337 errNo = OH_AVCencInfo_SetAVBuffer(cencInfo, buffer); 338 if (errNo != AV_ERR_OK) { 339 // Exception handling. 340 } 341 // Destroy the CencInfo instance. 342 errNo = OH_AVCencInfo_Destroy(cencInfo); 343 if (errNo != AV_ERR_OK) { 344 // Exception handling. 345 } 346 ``` 347 3489. Call **OH_AudioCodec_PushInputBuffer()** to write the data to decode. 349 350 To indicate the End of Stream (EOS), pass in the **AVCODEC_BUFFER_FLAGS_EOS** flag. 351 352 ```c++ 353 uint32_t index = signal_->inQueue_.front(); 354 auto buffer = signal_->inBufferQueue_.front(); 355 int64_t size; 356 int64_t pts; 357 // size is the length of each frame of the data to decode. pts is the timestamp of each frame and is used to indicate when the audio should be played. 358 // The values of size and pts are obtained from an audio and video resource file or data stream to decode. 359 // In the case of an audio and video resource file, the values are obtained from the buffer in the decapsulated OH_AVDemuxer_ReadSampleBuffer. 360 // In the case of a data stream, the values are obtained from the data stream provider. 361 // In this example, the values of size and pts are obtained from the test file. 362 inputFile_.read(reinterpret_cast<char *>(&size), sizeof(size)); 363 inputFile_.read(reinterpret_cast<char *>(&pts), sizeof(pts)); 364 inputFile_.read((char *)OH_AVBuffer_GetAddr(buffer), size); 365 OH_AVCodecBufferAttr attr = {0}; 366 if (inputFile_->eof()) { 367 attr.size = 0; 368 attr.flags = AVCODEC_BUFFER_FLAGS_EOS; 369 } else { 370 attr.size = size; 371 attr.flags = AVCODEC_BUFFER_FLAGS_NONE; 372 } 373 attr.pts = pts; 374 OH_AVBuffer_SetBufferAttr(buffer, &attr); 375 int32_t ret = OH_AudioCodec_PushInputBuffer(audioDec_, index); 376 if (ret != AV_ERR_OK) { 377 // Exception handling. 378 } 379 ``` 380 38110. Call **OH_AudioCodec_FreeOutputBuffer()** to output decoded PCM streams. 382 383 <!--RP3--> 384 ```c++ 385 uint32_t index = signal_->outQueue_.front(); 386 OH_AVBuffer *data = signal_->outBufferQueue_.front(); 387 // Obtain the buffer attributes. 388 OH_AVCodecBufferAttr attr = {0}; 389 ret = OH_AVBuffer_GetBufferAttr(data, &attr); 390 if (ret != AV_ERR_OK) { 391 // Exception handling. 392 } 393 // Write the decoded data (specified by data) to the output file. 394 pcmOutputFile_.write(reinterpret_cast<char *>(OH_AVBuffer_GetAddr(data)), attr.size); 395 ret = OH_AudioCodec_FreeOutputBuffer(audioDec_, index); 396 if (ret != AV_ERR_OK) { 397 // Exception handling. 398 } 399 if (attr.flags == AVCODEC_BUFFER_FLAGS_EOS) { 400 // End 401 } 402 ``` 403 <!--RP3End--> 404 40511. (Optional) Call **OH_AudioCodec_Flush()** to refresh the decoder. 406 407 After **OH_AudioCodec_Flush()** is called, the decoder remains in the running state, but the current queue is cleared and the buffer storing the decoded data is freed. To continue decoding, you must call **OH_AudioCodec_Start()** again. 408 409 You need to call **OH_AudioCodec_Start()** in the following cases: 410 411 * The EOS of the file is reached. 412 * An error with **OH_AudioCodec_IsValid** set to **true** (indicating that the execution can continue) occurs. 413 414 ```c++ 415 // Refresh the decoder. 416 ret = OH_AudioCodec_Flush(audioDec_); 417 if (ret != AV_ERR_OK) { 418 // Exception handling. 419 } 420 // Start decoding again. 421 ret = OH_AudioCodec_Start(audioDec_); 422 if (ret != AV_ERR_OK) { 423 // Exception handling. 424 } 425 ``` 426 42712. (Optional) Call **OH_AudioCodec_Reset()** to reset the decoder. 428 429 After **OH_AudioCodec_Reset()** is called, the decoder returns to the initialized state. To continue decoding, you must call **OH_AudioCodec_Configure()** and then **OH_AudioCodec_Start()**. 430 431 ```c++ 432 // Reset the decoder. 433 ret = OH_AudioCodec_Reset(audioDec_); 434 if (ret != AV_ERR_OK) { 435 // Exception handling. 436 } 437 // Reconfigure the decoder. 438 ret = OH_AudioCodec_Configure(audioDec_, format); 439 if (ret != AV_ERR_OK) { 440 // Exception handling. 441 } 442 ``` 443 44413. Call **OH_AudioCodec_Stop()** to stop the decoder. 445 446 ```c++ 447 // Stop the decoder. 448 ret = OH_AudioCodec_Stop(audioDec_); 449 if (ret != AV_ERR_OK) { 450 // Exception handling. 451 } 452 ``` 453 45414. Call **OH_AudioCodec_Destroy()** to destroy the decoder instance and release resources. 455 456 > **NOTE** 457 > 458 > You only need to call this API once. 459 460 ```c++ 461 // Call OH_AudioCodec_Destroy to destroy the decoder. 462 ret = OH_AudioCodec_Destroy(audioDec_); 463 if (ret != AV_ERR_OK) { 464 // Exception handling. 465 } else { 466 audioDec_ = NULL; // The decoder cannot be destroyed repeatedly. 467 } 468 ``` 469 470
