1# AI Framework Development Guide 2 3## **Overview** 4 5### Introduction 6 7The AI subsystem is the part of OpenHarmony that provides native distributed AI capabilities. At the heart of the subsystem is a unified AI engine framework, which implements quick integration of AI algorithm plug-ins. 8The framework consists of the plug-in management, module management, and communication management modules, fulfilling lifecycle management and on-demand deployment of AI algorithms. Specifically, plug-in management implements lifecycle management, on-demand deployment, and quick integration of AI algorithm plug-ins; module management implements task scheduling and client instance management; communication management manages inter-process communication (IPC) between the client and server and data transmission between the AI engine and plug-ins. Under this framework, AI algorithm APIs will be standardized to facilitate distributed calling of AI capabilities. In addition, unified inference APIs will be provided to adapt to different inference framework hierarchies. 9The following figure shows the AI engine framework. 10 11 12 **Figure 1** AI engine framework 13  14 15### Setting Up the Environment 16 17 181. Prepare a development board, which can be Hi3516D V300 or Hi3518E V300. 19 202. [Download the source code.](../get-code/sourcecode-acquire.md) 21 22## Technical Specifications 23 24### Code Management 25 26The AI engine framework consists of three modules: **client**, **server**, and **common**. The client module provides the server connection management function. An OpenHarmony SDK needs to encapsulate and call the public APIs provided by the client in the algorithm's external APIs. The server module provides functions such as plug-in loading and task management. Plug-ins are integrated using the plug-in APIs provided by the server. The common module provides platform-related operation methods, engine protocols, and tool classes for other modules. 27 28 29The following figure shows the code dependency between modules of the AI engine framework. 30 31 32 **Figure 2** Code dependency 33 34  35 36 37#### Recommendation: Develop plug-ins and OpenHarmony SDKs in the directories specified by the AI engine. 38 39In the overall planning of the AI engine framework, OpenHarmony SDKs are a part of the client, and plug-ins are called by the server and are considered a part of the server. Therefore, the following directories have been planned for plug-in and OpenHarmony SDK development in the AI engine framework: 40 41- SDK code directory: //foundation/ai/engine/services/client/algorithm_sdk 42 43 Example 1: //foundation/ai/engine/services/client/algorithm_sdk/cv 44 45 Example 2: //foundation/ai/engine/services/client/algorithm_sdk/nlu 46 47- Plug-in code directory: //foundation/ai/engine/services/server/plugin 48 49 Example 1: //foundation/ai/engine/services/server/plugin/cv 50 51 Example 2: //foundation/ai/engine/services/server/plugin/nlu 52 53 54#### Rule: Store all external APIs provided by plug-ins in the **interfaces/kits** directory of the AI subsystem. 55 56The AI subsystem exposes its capabilities through external APIs of OpenHarmony SDKs. According to API management requirements of OpenHarmony, store all external APIs of the SDK in the **interfaces/kits** directory of the subsystem. Currently, the external APIs of plug-ins of the AI subsystem are stored in the **//foundation/ai/engine/interfaces/kits** directory. You can add a sub-directory for each newly added plug-in in this directory. For example, if you add a CV plug-in, then store its external APIs in the **//foundation/ai/engine/interfaces/kits/cv** directory. 57 58 59#### Rule: Make sure that plug-in build results are stored in the **/usr/lib** directory. 60 61Plug-in loading on the server uses the dlopen mode and can only be performed in the** /usr/lib** directory. Therefore, when compiling the **.so** file of a plug-in, set the output directory as **/usr/lib** in the build configuration file. 62 63### Naming rule 64 65#### Rule: Name an SDK in the format of **domain_keyword<_other information 1_other information 2_...>_sdk.so**. 66 67You are advised to use the commonly known abbreviations for domains. For example, use **cv** for image and video, **asr** for voice recognition, and **translation** for text translation. Add one if there is no available abbreviation for a domain. Use keywords that accurately describe the algorithm capability of the plug-in. For example, use **keyword\_spotting** for wakeup keyword spotting (KWS). Add other information, such as the supported chip type and applicable region, between **keyword** and **sdk**, with each of them separated by an underscore (\_). Note that the name of an SDK must end with **\_sdk**. 68 69For example, if the SDK for the KWS plug-in supports only the Kirin 9000 chipset and is applicable only in China, then name the SDK as follows: **asr\_keyword\_spotting\_kirin9000\_china\_sdk.so**. 70 71 72#### Rule: Name a plug-in in the format of **domain_keyword<_other information 1_other information 2_...>.so**. 73 74There is a one-to-one mapping between plug-ins and SDKs. Therefore, the definitions and requirements of terms such as the domain, keyword, and other information in plug-in names are the same as those in SDK names. The only difference is that the name of the SDK ends with **\_sdk** additionally. For example, if the plug-in is named **asr\_keyword\_spotting.so**, the corresponding SDK is named **asr\_keyword\_spotting\_sdk.so**. 75 76For example, if the SDK for the KWS plug-in supports only the Kirin 9000 chipset and is applicable only in China, then name the plug-in as follows: **asr\_keyword\_spotting\_kirin9000\_china.so**. 77 78### API Development 79 80#### Rule: Encapsulate the external APIs provided by the client based on the algorithm call sequence. For the SDK of an asynchronous plug-in, implement the **IClientCb** API provided by the client. 81 82The external APIs provided by the client of the AI engine include **AieClientInit**, **AieClientPrepare**, **AieClientSyncProcess**, **AieClientAsyncProcess**, **AieClientRelease**, **AieClientDestroy**, **AieClientSetOption**, and **AieClientGetOption**. The SDK needs to encapsulate at least the following five APIs in sequence: **AieClientInit**, **AieClientPrepare**, **AieClientSyncProcess** (or **AieClientAsyncProcess**), **AieClientRelease**, and **AieClientDestroy**. Otherwise, a call failure or memory leakage may occur. For example, if the **AieClientPrepare** API is omitted during encapsulation, the server cannot load the plug-in. As a result, APIs that follow it cannot be called. 83 84For an asynchronous plug-in, the SDK needs to implement the **IClientCb** API to receive the algorithm inference result from the client and return the result to the third-party caller. 85 86 87#### Rule: Save all common data related to client interaction in the SDK during API implementation. 88 89The client of the AI engine uses the singleton pattern for API implementation. If the client is connecting to multiple SDKs, each SDK needs to store all common data exchanged with the client so that they can connect to the server to perform operations such as task inference and return the result. Common data usually includes **clientInfo**, **algorithmInfo**, and **configInfo**, which are defined in the SDK's member variables. 90 91 92#### Recommendation: Enable the SDK to implement the **IServiceDeadCb** API defined by the client. 93 94The processes running on the server are system resident processes. The server provides services for clients by way of system capabilities. The **IServiceDeadCb** API is called if a server process is abnormally killed. The SDK can implement related operations in this API, for example, stopping process call or restarting the server. 95 96The following is an example of **IServiceDeadCb** API implementation: 97 98 99``` 100class ServiceDeadCb : public IServiceDeadCb { 101public: 102ServiceDeadCb() = default; 103~ServiceDeadCb() override = default; 104void OnServiceDead() override 105{ 106printf("[ServiceDeadCb]OnServiceDead Callback happens"); 107} 108}; 109``` 110 111As shown above, the SDK can implement its own operations in the **OnServiceDead()** function, for example, stopping API call. 112 113 114#### Rule: Convert dedicated algorithm data into common data of the AI engine if the SDK and plug-ins need to use the codec module. 115 116For plug-ins, inference data is transmitted by the third-party caller to them through the client and server. The required data type varies according to algorithms. For example, the CV algorithm requires image data, and the ASR algorithm requires audio data. To address this issue, the AI engine provides the codec capabilities to convert different types of data into common data that can be used by it. 117 118The encoded data is as follows: 119 120 121``` 122struct DataInfo { 123unsigned char *data; 124int length; 125} DataInfo; 126 127``` 128 129As shown above, **DataInfo** consists of two variables: a pointer to the data memory, and the data length. 130 131To use the APIs of the AI engine framework, you need to: 132 1331. Add the dependency header file utils/encdec/include/encdec.h. 134 1352. Add the dependency items in the **build.gn** file. 136 137Add **//foundation/ai/engine/services/common** to **include\_dirs**. 138 139Add **//foundation/ai/engine/services/common/utils/encdec:encdec** to **deps**. 140 1413. Convert different types of data through codec. The following is an example: 142 143 144``` 145// Example function for encoding: arg1, arg2, and arg3 are variables to be encoded, and dataInfo is the encoding result. 146retCode = ProcessEncode(dataInfo, arg1, arg2, arg3) // The number of parameters can be flexible. 147// Example function for decoding: dataInfo is the data to be decoded, and arg1, arg2, and arg3 are the decoding result. 148retCode = ProcessDecode(dataInfo, arg1, arg2, arg3) // The number of parameters can be flexible. 149``` 150 151Note: 152 153- The sequence of parameters must be the same during encoding and decoding. 154 155- After encoding, the memory used by **dataInfo** needs to be manually released by the caller. 156 157- The memory is managed and released separately on the server and the client. 158 159- If a pointer contains the shared memory, no extra processing is required. 160 161- If other types of pointers are used, you need to dereference them before using **ProcessEncode** or **ProcessDecode**. 162 163- The codec module has not been adapted to the **class** data type and therefore it is not recommended. 164 165 166#### Rule: Release the memory used by the encoded or decoded parameters in the SDK. Otherwise, a memory leakage occurs. 167 168Encoding is essentially a process of encapsulating different types of data in the same memory space and then encapsulating the start address and length of the memory into the body. A plug-in is unable to release the memory that has been allocated to output parameter data, which is returned to the SDK through encoding. To obtain the data, the SDK first needs to release the memory. 169 170The following is an example of releasing the memory: 171 172 173``` 174DataInfo outputInfo = { 175.data = nullptr, 176.length = 0, 177}; 178AieClientPrepare(clientInfo_, algorithmInfo_, inputInfo, outputInfo, nullptr); 179if (outputInfo.data != nullptr) { 180free(outputInfo.data); 181outputInfo.data = nullptr; 182outputInfo.length = 0; 183} 184``` 185 186 187#### Rule: Enable plug-ins to implement the **IPlugin** API defined by the server and use the **PLUGIN\_INTERFACE\_IMPL** statement to provide the function pointer for external systems. 188 189The server manages a variety of plug-ins, and the API implementation logic varies according to plug-ins. To unify the plug-in loading process, the AI engine provides the **IPlugin** API. In the runtime environment, a plug-in is loaded as a dynamic link library (DLL) by the AI engine framework in dlopen mode. Therefore, the plug-in needs to use the **PLUGIN\_INTERFACE\_IMPL** statement to expose the function pointer. Otherwise, the plug-in cannot be properly loaded. 190 191 192#### Rule: Use the unified data channel provided by the AI engine for plug-ins. 193 194The AI engine provides a unified data channel between the server and plug-ins to send inference requests from the SDK and returned results from plug-ins. Plug-ins need to obtain the request data and encapsulate the inference result over the data channel when calling the inference API. 195 196The following is an example of using the data channel: 197 198 199``` 200int SyncProcess(IRequest *request, IResponse *&response) 201{ 202HILOGI("[IvpPlugin]Begin SyncProcess"); 203if (request == nullptr) { 204HILOGE("[IvpPlugin]SyncProcess request is nullptr"); 205return RETCODE_NULL_PARAM; 206} 207DataInfo inputInfo = request->GetMsg(); 208if (inputInfo.data == nullptr) { 209HILOGE("[IvpPlugin]InputInfo data is nullptr"); 210return RETCODE_NULL_PARAM; 211} 212 213... 214 215response = IResponse::Create(request); 216response->SetResult(outputInfo); 217return RETCODE_SUCCESS; 218} 219``` 220 221In the example, the request and response are the data body sent over the data channel. The server encapsulates data in the request and sends it to the plug-in. After completing algorithm processing, the plug-in encapsulates the result into the response and returns it to the server over the data channel. 222 223## Development Guidelines 224 225### Developing the SDK 226 227The function of the SDK header file is implemented by mapping SDK API calls to client API calls. The following table lists the APIs provided by the client. 228 229 230 **Table 1** Client APIs 231 232| API| Description| Parameters| 233| -------- | -------- | -------- | 234| int **AieClientInit**(const ConfigInfo &configInfo,<br> ClientInfo &clientInfo, const AlgorithmInfo<br> &algorithmInfo, IServiceDeadCb \*cb) | **Function**: Links and initializes the engine service and activates IPC call.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **configInfo**: engine-related initial configuration data. This parameter must not be null.<br>**clientInfo**: engine client information. This parameter must not be null.<br>**algorithmInfo**: information about the called algorithm. This parameter must not be null.<br>**cb**: death callback object. This parameter can be null. | 235| int **AieClientPrepare**(const ClientInfo &clientInfo<br>, const AlgorithmInfo &algorithmInfo, const DataInfo<br> &inputInfo, DataInfo &outputInfo, IClientCb \*cb) | **Function**: Loads an algorithm plug-in.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **clientInfo**: engine client information. This parameter must not be null.<br>**algorithmInfo**: information about the called algorithm. This parameter must not be null.<br>**inputInfo**: input information specified for algorithm plug-in loading. This parameter can be null.<br>**outputInfo**: information returned after algorithm plug-in loading, if any. This parameter can be null.<br>**cb**: return result of the asynchronous algorithm. This parameter must not be null for the asynchronous algorithm. For the synchronous algorithm, this parameter must be null. | 236| int **AieClientAsyncProcess**(const ClientInfo &clientInfo,<br> const AlgorithmInfo &algorithmInfo, const DataInfo<br> &inputInfo) | **Function**: Executes an asynchronous algorithm.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **clientInfo**: engine client information. This parameter must not be null.<br>**algorithmInfo**: information about the called algorithm. This parameter must not be null.<br>**inputInfo**: input information specified for algorithm operations. This parameter can be null. | 237| int **AieClientSyncProcess**(const ClientInfo &clientInfo,<br> const AlgorithmInfo &algorithmInfo, const<br> DataInfo &inputInfo, DataInfo &outputInfo) | **Function**: Executes a synchronous algorithm.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **clientInfo**: engine client information. This parameter must not be null.<br>**algorithmInfo**: information about the called algorithm. This parameter must not be null.<br>**inputInfo**: input information specified for algorithm operations. This parameter can be null.<br>**outputInfo**: output information in the return result of the synchronous algorithm. This parameter can be null. | 238| int **AieClientRelease**(const ClientInfo &clientInfo,<br> const AlgorithmInfo &algorithmInfo, const<br> DataInfo &inputInfo) | **Function**: Uninstalls an algorithm plug-in.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **clientInfo**: engine client information. This parameter must not be null.<br>**algorithmInfo**: information about the algorithm plug-in to be uninstalled. This parameter must not be null.<br>**inputInfo**: input information specified for algorithm plug-in uninstallation. This parameter can be null. | 239| int **AieClientDestroy**(ClientInfo &clientInfo) | **Function**: Disconnects from the server and releases the cache.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **clientInfo**: engine client information. This parameter must not be null. | 240| int **AieClientSetOption**(const ClientInfo &clientInfo,<br> int optionType, const DataInfo &inputInfo) | **Function**: Sets configuration items. You can use this API to pass algorithm's extended information to plug-ins.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **clientInfo**: engine client information. This parameter must not be null.<br>**optionType**: algorithm used for obtaining the configuration item information. An algorithm plug-in can use this parameter as needed. This parameter must not be null.<br>**inputInfo**: Indicates algorithm parameter information. An algorithm plug-in can use this parameter as needed. This parameter can be null. | 241| int **AieClientGetOption**(const ClientInfo &clientInfo,<br> int optionType, const DataInfo &inputInfo,<br> DataInfo &outputInfo) | **Function**: Obtains configuration item information based on the specified **optionType** and **inputInfo**.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **clientInfo**: engine client information. This parameter must not be null.<br>**optionType**: algorithm used for obtaining the configuration item information. This parameter must not be null.<br>**inputInfo**: input information specified for obtaining configuration item information of the algorithm. This parameter can be null.<br>**outputInfo**: configuration item information in the return result. This parameter can be null. | 242 243 244The following table describes the data structure of **ConfigInfo**, **ClientInfo**, **AlgorithmInfo**, and **DataInfo**. 245 246 247 **Table 2** Data structure of ConfigInfo, ClientInfo, AlgorithmInfo, and DataInfo 248 249| Data Structure| Description| Attributes| 250| -------- | -------- | -------- | 251| ConfigInfo | Algorithm configuration item information| **const char \*description**: body of configuration item information. | 252| ClientInfo | Client information.| **long long clientVersion**: client version number. This parameter is not used currently.<br>**int clientId**: client ID.<br>**int sessionId**: session ID.<br>**uid\_t serverUid**: server UID.<br>**uid\_t clientUid**: client UID.<br>**int extendLen**: length of the extended information (**extendMsg**).<br>**unsigned char \*extendMsg**: body of the extended information. | 253| AlgorithmInfo | Algorithm information| **long long clientVersion**: client version number. This parameter is not used currently.<br>**bool isAsync**: whether asynchronous execution is used.<br>**int algorithmType**: algorithm type ID allocated by the AI engine framework based on the plug-in loading sequence.<br>**long long algorithmVersion**: algorithm version number.<br>**bool isCloud**: whether to migrate data to the cloud. This parameter is not used currently.<br>**int operateId**: execution ID. This parameter is not used currently.<br>**int requestId**: request ID, which identifies each request and corresponds to the execution result.<br>**int extendLen**: length of the extended information (**extendMsg**).<br>**unsigned char \*extendMsg**: body of the extended information. | 254| DataInfo | Algorithm input parameter configuration information (**inputInfo**)<br>and output parameter configuration information (**outputInfo**)| **unsigned char \*data**: data subject.<br>**int length**: data length. | 255 256 257For details about the development process, see the development example of the [KWS SDK](#kws-sdk). 258 259### Developing a Plug-in 260The AI engine framework has defined a set of algorithm plug-in access specifications. Each plug-in needs to implement specified APIs to implement functions such as obtaining the plug-in version information and algorithm inference type, executing synchronous and asynchronous algorithms, loading algorithm plug-ins, uninstalling algorithm plug-ins, setting algorithm configuration information, and obtaining specified algorithm configuration information. Specifically, implement the **SyncProcess** API for the synchronous algorithm and the **AsyncProcess** API for the asynchronous algorithm. 261 262 263The following table describes the **IPlugin** APIs. 264 265 266 **Table 3** IPlugin APIs 267 268| API| Description| Parameters| 269| -------- | -------- | -------- | 270| const long long GetVersion() const; | **Function**: Obtains the plug-in version information.<br>**Return value**: version number (in the **long long** format)| - | 271| const char \*GetInferMode() const; | **Function**: Obtains the algorithm inference type.<br>**Return value**: "SYNC" or "ASYNC"| - | 272| int SyncProcess(IRequest \*request,<br>IResponse \*&response); | **Function**: Executes a synchronous algorithm.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **request**: used to pass the request content to the algorithm plug-in over the data channel between the engine server and the plug-in. This parameter must not be null.<br>**response**: used to receive the synchronous algorithm execution result returned by the algorithm plug-in over the data channel between the engine server and the plug-in. This parameter must not be null.| 273| int AsyncProcess(IRequest \*request,<br>IPluginAlgorithmCallback \*callback); | **Function**: Executes an asynchronous algorithm.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **request**: used to pass the request content to the algorithm plug-in over the data channel between the engine server and the plug-in. This parameter must not be null.<br>**callback**: used to return the asynchronous algorithm execution result to the engine server. This parameter must not be null.| 274| int Prepare(long long transactionId,<br>const DataInfo &inputInfo, DataInfo<br>&outputInfo); | **Function**: Loads an algorithm plug-in.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **transactionId**: transaction ID, which is used to identify the client and session. This parameter must not be null.<br>**inputInfo**: input information specified for algorithm plug-in loading. This parameter can be null.<br>**outputInfo**: output information in the return result of algorithm plug-in loading. This parameter can be null.| 275| int Release(bool isFullUnload, long long<br>transactionId, const DataInfo &inputInfo); | **Function**: Uninstalls an algorithm plug-in.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **isFullUnload**: whether a plug-in is called by only one client. A plug-in can be uninstalled only when it is called by only one client. This parameter must not be null.<br>**transactionId**: transaction ID, which is used to identify the client and session. This parameter must not be null.<br>**inputInfo**: input information specified for algorithm plug-in uninstallation. This parameter can be null.| 276| int SetOption(int optionType, const<br>DataInfo &inputInfo); | **Function**: Sets configuration items. You can use this API to pass algorithm's extended information to plug-ins.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **optionType**: algorithm used for obtaining the configuration item information. An algorithm plug-in can use this parameter as needed. This parameter must not be null.<br>**inputInfo**: algorithm parameter information. An algorithm plug-in can use this parameter as needed. This parameter can be null.| 277| int GetOption(int optionType, const<br>DataInfo &inputInfo, DataInfo<br>&outputInfo); | **Function**: Obtains configuration item information based on the specified **optionType** and **inputInfo**.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **optionType**: algorithm used for obtaining the configuration item information. This parameter must not be null.<br>**inputInfo**: input information specified for obtaining configuration item information of the algorithm. This parameter can be null.<br>**outputInfo**: configuration item information in the return result. This parameter can be null.| 278 279 280Algorithm plug-in APIs including **Prepare**, **SyncProcess**, **AsyncProcess**, **Release**, **SetOption**, and **GetOption** are in 1:1 mapping with the client APIs including **AieClientPrepare**, **AieClientSyncProcess**, **AieClientAsyncProcess**, **AieClientRelease**, **AieClientSetOption**, and **AieClientGetOption**. The **GetInferMode** API is used to return the algorithm execution type, which can be synchronous or asynchronous. 281 282 283The following table describes the **IPluginCallback** APIs. 284 285 286 **Table 4** IPluginCallback APIs 287 288| API| Description| Parameters| 289| -------- | -------- | -------- | 290| void OnEvent(PluginEvent event,<br>IResponse \*response); | **Function**: Returns the asynchronous algorithm execution result.| **event**: used to enumerate the algorithm execution result. The value can be **ON\_PLUGIN\_SUCCEED** or **ON\_PLUGIN\_FAIL**.<br>**response**: used to return the algorithm execution result.| 291 292 293The **Request** and **Response** classes define the requests and responses used for communication between the AI engine server and algorithm plug-ins. A request encapsulates the request content and input data of the caller. The plug-in returns the calculation result to the AI engine server through a response. 294 295 296The following table describes the attributes of the **Request** class. 297 298 299 **Table 5** Attributes of the Request class 300 301| Attribute| Description| Default Value| 302| -------- | -------- | -------- | 303| innerSequenceId_ | **Type**: long long<br>**Function**: reserved| 0 | 304| requestId_ | **Type**: int<br>**Function**: Indicates the request sequence, which is used to bind the return result.| 0 | 305| operationId_ | **Type**: int<br>**Function**: reserved| 0 | 306| transactionId_ | Type: long long<br>**Function**: Indicates the transaction ID, which is the combination of **clientId** and **sessionId**.| 0 | 307| algoPluginType_ | **Type**: int<br>**Function**: Indicates the algorithm type ID allocated by the AI engine framework based on the plug-in loading sequence.| 0 | 308| msg_ | **Type**: DataInfo<br>**Function**: Stores the input parameters for calling the algorithm API.| .data = nullptr<br>.length = 0 | 309 310 311The following table describes the attributes of the **Response** class. 312 313 314 **Table 6** Attributes of the Response class 315 316| Attribute| Description| Default Value| 317| -------- | -------- | -------- | 318| innerSequenceId_ | **Type**: long long<br>**Function**: reserved| 0 | 319| requestId_ | **Type**: int<br>**Function**: Indicates the request sequence, which is used to bind the return result.| 0 | 320| retCode__ | **Type**: int<br>**Function**: Indicates the inference result code of the asynchronous algorithm.| 0 | 321| retDesc_ | **Type**: string<br>**Function**: reserved| - | 322| transactionId_ | Type: long long<br>**Function**: Indicates the transaction ID, which is the combination of **clientId** and **sessionId**.| 0 | 323| algoPluginType_ | **Type**: int<br>**Function**: Indicates the algorithm type ID allocated by the AI engine framework based on the plug-in loading sequence.| INVALID_ALGO_PLUGIN_TYPE(-1) | 324| result_ | **Type**: DataInfo<br>**Function**: Stores the inference result of the asynchronous algorithm.| .data = nullptr<br>.length = 0 | 325 326 327For details about the development process, see the development example of the [KWS plug-in](#kws-plug-in). 328 329### Preparing the Configuration File 330 331The SDK identifies the plug-in type based on **algorithmVersion** and **algorithmType** in the **AlgorithmInfo** structure so it can call the plug-in capabilities. Therefore, you need to develop the plug-in configuration file as follows: 332 333 3341. Add the plug-in configuration file to the **//foundation/ai/engine/services/common/protocol/plugin\_config/plugin\_config\_ini/** directory. 335 3362. Add the algorithm type to the **aie\_algorithm_type.h** file in the **//foundation/ai/engine/services/common/protocol/plugin\_config/** directory. 337 3383. Add the name of the KWS algorithm and its sequence number in **ALGORITHM\_TYPE\_ID\_LIST** to the **aie\_plugin\_info.h** file in the **//foundation/ai/engine/services/server/plugin\_manager/include/** directory. 339 340For details about the development process, see the development example of the [configuration file](#kws-configuration-file). 341 342## Development Example 343 344### KWS SDK 345 346 3471. Add the API definition of the KWS SDK to the **//foundation/ai/engine /interfaces/kits** directory. This API can be called by third-party applications. The following code snippet is an example API for the KWS SDK. The reference code is available at the **//foundation/ai/engine /interfaces/kits/asr/keyword\_spotting** directory. 348 349 ``` 350 class KWSSdk { 351 public: 352 KWSSdk(); 353 virtual ~KWSSdk(); 354 355 // Create a KWS SDK instance. 356 int32_t Create(); 357 358 // Synchronously execute the KWS task. 359 int32_t SyncExecute(const Array<int16_t> &audioInput); 360 361 // Set the KWS callback. 362 int32_t SetCallback(const std::shared_ptr<KWSCallback> &callback); 363 364 // Destroy the KWS SDK instance to release the session engaged with the plug-in. 365 int32_t Destroy(); 366 }; 367 ``` 368 3692. Add the API implementation of the SDK to the **//foundation/ai/engine/services/client/algorithm\_sdk** directory and call the APIs provided by the client to use the algorithm plug-in capabilities. The following code snippet is an example implementation of the **create** method in the API of the KWS SDK. For more details, see the reference code at **//foundation/ai/engine/services/client/algorithm\_sdk/asr/keyword\_spotting**. 370 371 ``` 372 int32_t KWSSdk::KWSSdkImpl::Create() 373 { 374 if (kwsHandle_ != INVALID_KWS_HANDLE) { 375 HILOGE("[KWSSdkImpl]The SDK has been created"); 376 return KWS_RETCODE_FAILURE; 377 } 378 if (InitComponents() != RETCODE_SUCCESS) { 379 HILOGE("[KWSSdkImpl]Fail to init sdk components"); 380 return KWS_RETCODE_FAILURE; 381 } 382 // Call the AieClientInit API provided by the client to initialize the engine service and activate IPC call. 383 int32_t retCode = AieClientInit(configInfo_, clientInfo_, algorithmInfo_, nullptr); 384 if (retCode != RETCODE_SUCCESS) { 385 HILOGE("[KWSSdkImpl]AieClientInit failed. Error code[%d]", retCode); 386 return KWS_RETCODE_FAILURE; 387 } 388 if (clientInfo_.clientId == INVALID_CLIENT_ID) { 389 HILOGE("[KWSSdkImpl]Fail to allocate client id"); 390 return KWS_RETCODE_FAILURE; 391 } 392 DataInfo inputInfo = { 393 .data = nullptr, 394 .length = 0, 395 }; 396 DataInfo outputInfo = { 397 .data = nullptr, 398 .length = 0, 399 }; 400 // Call the AieClientPrepare API provided by the client to load the algorithm plug-in. 401 retCode = AieClientPrepare(clientInfo_, algorithmInfo_, inputInfo, outputInfo, nullptr); 402 if (retCode != RETCODE_SUCCESS) { 403 HILOGE("[KWSSdkImpl]AieclientPrepare failed. Error code[%d]", retCode); 404 return KWS_RETCODE_FAILURE; 405 } 406 if (outputInfo.data == nullptr || outputInfo.length <= 0) { 407 HILOGE("[KWSSdkImpl]The data or length of output info is invalid"); 408 return KWS_RETCODE_FAILURE; 409 } 410 MallocPointerGuard<unsigned char> pointerGuard(outputInfo.data); 411 retCode = PluginHelper::UnSerializeHandle(outputInfo, kwsHandle_); 412 if (retCode != RETCODE_SUCCESS) { 413 HILOGE("[KWSSdkImpl]Get handle from inputInfo failed"); 414 return KWS_RETCODE_FAILURE; 415 } 416 return KWS_RETCODE_SUCCESS; 417 } 418 ``` 419 420 The preceding code is the specific API implementation. The **create** function in the API of the KWS SDK calls the open **AieClientInit** and **AieClientPrepare** APIs provided by the client to connect to the server and load the algorithm model. 421 422 > **NOTE** 423 > 424 > The sequence for the SDK to call client APIs of the AI engine is as follows: AieClientInit -> AieClientPrepare -> AieClientSyncProcess/AieClientAsyncProcess -> AieClientRelease -> AieClientDestroy. An exception will be thrown if the call sequence is violated. 425 426### KWS Plug-in 427Add the API definition (that is, **IPlugin**) of the KWS plug-in to the **//foundation/ai/engine/services/server/plugin** directory. This API is used to call AI capabilities. The following code snippet is an example API implementation of the KWS plug-in. The reference code is available at the **//foundation/ai/engine/services/server/plugin/asr/keyword\_spotting** directory. 428 429 ``` 430 #include "plugin/i_plugin.h 431 class KWSPlugin : public IPlugin { 432 public: 433 KWSPlugin(); 434 ~KWSPlugin(); 435 const long long GetVersion() const override; 436 const char* GetName() const override; 437 const char* GetInferMode() const override; 438 int32_t Prepare(long long transactionId, const DataInfo &amp;inputInfo, DataInfo &amp;outputInfo) override; 439 int32_t SetOption(int optionType, const DataInfo &amp;inputInfo) override; 440 int32_t GetOption(int optionType, const DataInfo &amp;inputInfo, DataInfo &amp;outputInfo) override; 441 int32_t SyncProcess(IRequest *request, IResponse *&amp;response) override; 442 int32_t AsyncProcess(IRequest *request, IPluginCallback*callback) override; 443 int32_t Release(bool isFullUnload, long long transactionId, const DataInfo &amp;inputInfo) override; 444 }; 445 ``` 446The preceding code implements the **IPlugin** APIs provided by the server. The following table shows the mapping between the client APIs and the plug-in APIs. 447 448**Table 7** Mapping between the client APIs and the plug-in APIs 449 450| Client API| Plug-in API| Description| 451| -------- | -------- | -------- | 452| AieClientPrepare | Prepare | Initializes the inference algorithm plug-in. For KWS, this API loads the KWS model from the fixed location (**/sdcard/wenwen\_inst.wk**) to the memory.| 453| AieClientSyncProcess | SyncProcess | Executes the inference algorithm synchronously. For KWS, this API synchronously executes the audio inference algorithm to determine whether the specified wakeup keyword exists in the audio.| 454| AieClientAsyncProcess | AsyncProcess | Executes the inference algorithm asynchronously. Currently, this API is not used in KWS. However, you can implement the API based on your use case.| 455| AieClientSetOption | SetOption | Sets algorithm-related configuration items, such as the confidence threshold and delay. Currently, this API is not used in KWS. However, you can implement the API based on your use case.| 456| AieClientGetOption | GetOption | Obtains algorithm-related configuration items. For KWS, this API can obtain the input and output scale of the KWS model. The input scale is the MFCC feature (fixed value: **4000**) required by the KWS model, and the output scale is the confidence (fixed value: **2**) of the result.| 457| AieClientRelease | Release | Releases the algorithm model. For KWS, this API releases the specified algorithm model and clears the dynamic memory in the feature processor.| 458 459 Note: 460 461 1. The **AieClientInit** and **AieClientDestroy** APIs are used to connect to and disconnect from the server, respectively. They are not called in the plug-in algorithm and therefore do not need to be defined in the plug-in. 462 463 2. The KWS plug-in needs to use the **PLUGIN\_INTERFACE\_IMPL** statement to expose the function pointer. Otherwise, the plug-in cannot be properly loaded. 464 465 466 ``` 467 PLUGIN_INTERFACE_IMPL(KWSPlugin); 468 ``` 469 470### KWS Configuration File 471 4721. Add the KWS configuration file to the **//foundation/ai/engine/services/common/protocol/plugin\_config/plugin\_config\_ini/** directory. 473 474 ``` 475 [base] 476 supported_boards = hi3516dv300 477 related_sessions = asr_keyword_spotting+20001002 478 479 // Naming rule: [algorithm name+algorithm version], for example, [asr\_keyword\_spotting+20001002] 480 [asr_keyword_spotting+20001002] 481 AID = asr_keyword_spotting 482 VersionCode = 20001002 483 VersionName = 2.00.01.002 484 XPU = NNIE 485 District = China 486 // Location of the complied **.so** file of the plug-in 487 FullPath = /usr/lib/libasr_keyword_spotting.so 488 Chipset = ALL 489 ChkSum = '' 490 Key = '' 491 ``` 492 4932. Add the type ID of the KWS algorithm to the **aie\_algorithm_type.h** file in the **//foundation/ai/engine/services/common/protocol/plugin\_config/** directory. 494 495 ``` 496 // Make sure that the type ID maps to the sequence number in **ALGORITHM\_TYPE\_ID\_LIST**. 497 const int ALGORITHM_TYPE_KWS = 3; 498 ``` 499 5003. Add the name of the KWS algorithm and its sequence number in **ALGORITHM\_TYPE\_ID\_LIST** to the **aie\_plugin\_info.h** file in the **//foundation/ai/engine/services/server/plugin\_manager/include/** directory. 501 502 ``` 503 const std::string ALGORITHM_ID_SAMPLE_1 = "sample_plugin_1"; 504 const std::string ALGORITHM_ID_SAMPLE_2 = "sample_plugin_2"; 505 const std::string ALGORITHM_ID_IVP = "cv_human_detect"; 506 // Add the name of the KWS algorithm: asr\_keyword\_spotting. 507 // Name the algorithm variable in the same way as the algorithm type ID in ALGORITHM\_TYPE\_ID\_LIST, for example, ALGORITHM_ID_KWS. 508 const std::string ALGORITHM_ID_KWS = "asr_keyword_spotting"; 509 const std::string ALGORITHM_ID_IC = "cv_image_classification"; 510 const std::string ALGORITHM_ID_INVALID = "invalid algorithm id"; 511 512 const std::vector<std::string> ALGORITHM_TYPE_ID_LIST = { 513 ALGORITHM_ID_SAMPLE_1, 514 ALGORITHM_ID_SAMPLE_2, 515 ALGORITHM_ID_IVP, 516 // Add the sequence number of the KWS algorithm to ALGORITHM\_TYPE\_ID\_LIST, so that the name of the KWS algorithm can be obtained based on the sequence number. 517 // Make sure that the algorithm name maps to the sequence number in ALGORITHM\_TYPE\_ID\_LIST. 518 ALGORITHM_ID_KWS, 519 ALGORITHM_ID_IC, 520 }; 521 ``` 522
