1# Neural Network Runtime Device Access 2 3## Overview 4 5### Function Introduction 6 7Neural Network Runtime (NNRt) functions as a bridge to connect the upper-layer AI inference framework and underlying acceleration chips, implementing cross-chip inference computing of AI models. 8 9NNRt opens HDIs for chip vendors to connect dedicated acceleration chips to NNRt to interconnect with the OpenHarmony ecosystem. 10 11### Basic Concepts 12Before you get started, it would be helpful for you to have a basic understanding of the following concepts: 13 14- Hardware Device Interface (HDI): defines APIs for cross-process communication between services in OpenHarmony. 15- Interface Description Language (IDL): defines the HDI language format. 16 17### Constraints 18- System version: OpenHarmony trunk version. 19- Development environment: Ubuntu 18.04 or later. 20- Access device: a chip with AI computing capabilities. 21 22### Working Principles 23NNRt connects to device chips through HDIs, which implement cross-process communication between services. 24 25**Figure 1** NNRt architecture 26 27 28 29The NNRt architecture consists of three layers: AI applications at the application layer, AI inference framework and NNRt at the system layer, and device services at the chip layer. To use a dedicated acceleration chip model for inference, an AI application needs to call the underlying dedicated AI acceleration chip through the AI inference framework and NNRt. NNRt is responsible for adapting to various underlying dedicated acceleration chips. It opens standard and unified HDIs for various AI acceleration chips to access the OpenHarmony ecosystem. In addition, NNRt opens standard and unified APIs to connect to various upper-layer AI inference frameworks. 30 31During program running, the AI application, AI inference framework, and NNRt reside in the user process, and the underlying AI chip device services reside in the HDI service process. NNRt implements the HDI client and the service side implements HDI Service to fulfill cross-process communication through the Hardware Driver Foundation (HDF) subsystem. 32 33## How to Develop 34 35### Application Scenario 36Suppose you are connecting an AI acceleration chip, for example, RK3568, to NNRt. The following exemplifies how to connect the RK3568 chip to NNRt through HDI V2.0 to implement AI model inference. The process is similar for HDI V1.0. 37> **NOTE**<br>In this tutorial, the purpose of connecting the RK3568 chip to NNRt is to utilize the runtime and CPU operators of MindSpore Lite, instead of implementing the CPU driver. Therefore, the RK3568 chip depends on the dynamic libraries and header files of MindSpore Lite. In practice, the RK3568 chip does not depend on any library or header file of MindSpore Lite. 38 39### Development Flowchart 40The following figure shows the process of connecting a dedicated AI acceleration chip to NNRt. 41 42**Figure 2** Process of connecting a dedicated AI acceleration chip to NNRt 43 44 45 46### Development Procedure 47The following uses the RK3568 chip as an example to describe the development procedure. 48#### Generating the HDI Header File 49Download the OpenHarmony source code from the open source community, build the `drivers_interface` component, and generate the HDI header file. 50 511. [Download the source code](../get-code/sourcecode-acquire.md). 52 532. Go to the root directory of OpenHarmony source code, and compile the IDL API file of NNRt. 54 ```shell 55 ./build.sh --product-name rk3568 –ccache --build-target drivers_interface_nnrt 56 ``` 57 58 After the compilation is complete, an HDI header file of the C++ language is generated in the `out/rk3568/gen/drivers/interface/nnrt/v2_0` directory. To generate a header file of the C language, run the following command to set the language field in the `drivers/interface/nnrt/v2_0/BUILD.gn` file before starting compilation: 59 60 ```shell 61 language = "c" 62 ``` 63 64 The directory of the generated header file is as follows: 65 ```text 66 out/rk3568/gen/drivers/interface/nnrt 67 └── v2_0 68 ├── drivers_interface_nnrt__libnnrt_proxy_2.0_external_deps_temp.json 69 ├── drivers_interface_nnrt__libnnrt_stub_2.0_external_deps_temp.json 70 ├── innrt_device.h # Header file of the HDI 71 ├── iprepared_model.h # Header file of the AI model 72 ├── libnnrt_proxy_2.0__notice.d 73 ├── libnnrt_stub_2.0__notice.d 74 ├── model_types.cpp # Implementation file for AI model structure definition 75 ├── model_types.h # Header file for AI model structure definition 76 ├── nnrt_device_driver.cpp # Device driver implementation example 77 ├── nnrt_device_proxy.cpp 78 ├── nnrt_device_proxy.h 79 ├── nnrt_device_service.cpp # Implementation file for device services 80 ├── nnrt_device_service.h # Header file for device services 81 ├── nnrt_device_stub.cpp 82 ├── nnrt_device_stub.h 83 ├── nnrt_types.cpp # Implementation file for data type definition 84 ├── nnrt_types.h # Header file for data type definitions 85 ├── node_attr_types.cpp # Implementation file for AI model operator attribute definition 86 ├── node_attr_types.h # Header file for AI model operator attribute definition 87 ├── prepared_model_proxy.cpp 88 ├── prepared_model_proxy.h 89 ├── prepared_model_service.cpp # Implementation file for AI model services 90 ├── prepared_model_service.h # Header file for AI model services 91 ├── prepared_model_stub.cpp 92 └── prepared_model_stub.h 93 ``` 94 95#### Implementing the HDI Service 96 971. Go to the root directory of OpenHarmony source code, and create the `nnrt` folder in the `drivers/peripheral` directory for HDI service development. The directory structure is as follows: 98 ```text 99 drivers/peripheral/nnrt 100 ├── bundle.json 101 ├── v2_0 102 ├── BUILD.gn # Code build script 103 └── hdi_cpu_service # Customized directory 104 ├── BUILD.gn # Code build script 105 ├── include 106 │ ├── nnrt_device_service.h # Header file for device services 107 │ ├── node_functions.h # Optional, depending on the actual implementation 108 │ ├── node_registry.h # Optional, depending on the actual implementation 109 │ └── prepared_model_service.h # Header file for AI model services 110 └── src 111 ├── nnrt_device_driver.cpp # Implementation file for the device driver 112 ├── nnrt_device_service.cpp # Implementation file for device services 113 ├── nnrt_device_stub.cpp # Optional, depending on the actual implementation 114 ├── node_attr_types.cpp # Optional, depending on the actual implementation 115 ├── node_functions.cpp # Optional, depending on the actual implementation 116 ├── node_registry.cpp # Optional, depending on the actual implementation 117 └── prepared_model_service.cpp # Implementation file for AI model services 118 ``` 119 1202. Implement the device driver. Unless otherwise required, you can directly use the `nnrt_device_driver.cpp` file generated through IDL file compilation. 121 1223. Implement service APIs by referring to the `nnrt_device_service.cpp` and `prepared_model_service.cpp` files. For details about the API definitions, see [NNRt HDI Definitions](https://gitee.com/openharmony/drivers_interface/tree/master/nnrt). 123 1244. Compile the implementation files for device drivers and services as shared libraries. 125 126 Create the `BUILD.gn` file with the following content in the `drivers/peripheral/nnrt/v2_0/hdi_cpu_service/` directory. For details about how to set related parameters, see [Compilation and Building](https://gitee.com/openharmony/build). 127 128 ```shell 129 import("//build/ohos.gni") 130 import("//drivers/hdf_core/adapter/uhdf2/uhdf.gni") 131 132 ohos_shared_library("libnnrt_service_2.0") { 133 include_dirs = [] 134 sources = [ 135 "src/nnrt_device_service.cpp", 136 "src/node_functions.cpp", 137 "src/node_registry.cpp", 138 "src/prepared_model_service.cpp", 139 "src/shared_buffer_parser.cpp", 140 "src/validation.cpp", 141 ] 142 143 external_deps = [ 144 "c_utils:utils", 145 "drivers_interface_nnrt:libnnrt_stub_2.0", 146 "hdf_core:libhdf_utils", 147 "hilog_native:libhilog", 148 "ipc:ipc_core", 149 ] 150 151 install_images = [ chipset_base_dir ] 152 subsystem_name = "hdf" 153 part_name = "drivers_peripheral_nnrt" 154 } 155 156 ohos_shared_library("libnnrt_driver") { 157 include_dirs = [] 158 sources = [ "src/nnr_device_driver.cpp" ] 159 deps = [ ":libnnrt_service_2.0" ] 160 161 external_deps = [ 162 "c_utils:utils", 163 "drivers_interface_nnrt:libnnrt_stub_2.0", 164 "hdf_core:libhdf_host", 165 "hdf_core:libhdf_ipc_adapter", 166 "hdf_core:libhdf_utils", 167 "hdf_core:libhdi", 168 "hilog_native:libhilog", 169 "ipc:ipc_core", 170 ] 171 172 install_images = [ chipset_base_dir ] 173 subsystem_name = "hdf" 174 part_name = "drivers_peripheral_nnrt" 175 } 176 177 group("hdf_nnrt_service") { 178 deps = [ 179 ":libnnrt_driver", 180 ":libnnrt_service_2.0", 181 ] 182 } 183 ``` 184 185 Add `group("hdf_nnrt_service")` to the `drivers/peripheral/nnrt/v2_0/BUILD.gn` file so that it can be referenced at a higher directory level. 186 ```shell 187 if (defined(ohos_lite)) { 188 group("nnrt_entry") { 189 deps = [] 190 } 191 } else { 192 group("nnrt_entry") { 193 deps = [ "./hdi_cpu_service:hdf_nnrt_service" ] 194 } 195 } 196 ``` 197 198 Create the `drivers/peripheral/nnrt/bundle.json` file to define the new `drivers_peripheral_nnrt` component. 199 ```json 200 { 201 "name": "drivers_peripheral_nnrt", 202 "description": "Neural network runtime device driver", 203 "version": "4.0", 204 "license": "Apache License 2.0", 205 "component": { 206 "name": "drivers_peripheral_nnrt", 207 "subsystem": "hdf", 208 "syscap": [""], 209 "adapter_system_type": ["standard"], 210 "rom": "1024KB", 211 "ram": "2048KB", 212 "deps": { 213 "components": [ 214 "c_utils", 215 "hdf_core", 216 "hilog_native", 217 "ipc" 218 ], 219 "third_part": [ 220 "bounds_checking_function" 221 ] 222 }, 223 "build": { 224 "sub_component": [ 225 "//drivers/peripheral/nnrt/v2_0:nnrt_entry" 226 ], 227 "test": [ 228 ], 229 "inner_kits": [ 230 ] 231 } 232 } 233 } 234 ``` 235 236#### Declaring the HDI Service 237 238 In the `uhdf` directory, declare the user-mode driver and services in the `.hcs` file of the corresponding product. For example, for the RK3568 chip, add the following configuration to the `vendor/hihope/rk3568/hdf_config/uhdf/device_info.hcs` file: 239 ```text 240 nnrt :: host { 241 hostName = "nnrt_host"; 242 priority = 50; 243 uid = ""; 244 gid = ""; 245 caps = ["DAC_OVERRIDE", "DAC_READ_SEARCH"]; 246 nnrt_device :: device { 247 device0 :: deviceNode { 248 policy = 2; 249 priority = 100; 250 moduleName = "libnnrt_driver.z.so"; 251 serviceName = "nnrt_device_service"; 252 } 253 } 254 } 255 ``` 256> **NOTE**<br>After modifying the `.hcs` file, you need to delete the `out` folder and compile the file again for the modification to take effect. 257 258#### Configuring the User ID and Group ID of the Host Process 259 In the scenario of creating an nnrt_host process, you need to configure the user ID and group ID of the corresponding process. The user ID is configured in the `base/startup/init/services/etc/passwd` file, and the group ID is configured in the `base/startup/init/services/etc/group` file. 260 ```text 261 # Add the user ID in base/startup/init/services/etc/passwd. 262 nnrt_host:x:3311:3311:::/bin/false 263 264 # Add the group ID in base/startup/init/services/etc/group. 265 nnrt_host:x:3311: 266 ``` 267 268#### Configuring SELinux 269 270The SELinux feature has been enabled for the OpenHarmony. You need to configure SELinux rules for the new processes and services so that they can run the host process to access certain resources, release HDI services, etc. 271 2721. Add the following configuration to the `base/security/selinux/sepolicy/ohos_policy/drivers/adapter/public/hdf_service_contexts` file: 273 ```text 274 # Add the security context configuration. 275 nnrt_device_service u:object_r:hdf_nnrt_device_service:s0 276 ``` 277 > **NOTE**<br>`nnrt_host` indicates the process name configured in [Declaring the HDI Service](#declaring-the-hdi-service). 278 2792. Add the following configuration to the `base/security/selinux/sepolicy/ohos_policy/drivers/adapter/public/hdf_service.te` file: 280 ```text 281 # Add the security context configuration. 282 type hdf_nnrt_device_service, hdf_service_attr; 283 ``` 284 2853. Add the following configuration to the `base/security/selinux/sepolicy/ohos_policy/drivers/adapter/public/hdfdomain.te` file: 286 ```text 287 # Add the security context configuration. 288 neverallow { domain -hdfdomain -sadomain } { hdfdomain -nnrt_host -allocator_host -hdf_public_domain }:binder call; 289 ``` 290 2914. Add the following configuration to the `base/security/selinux/sepolicy/ohos_policy/drivers/adapter/public/type.te` file: 292 ```text 293 # Add the security context configuration. 294 type nnrt_host, hdfdomain, domain; 295 ``` 296 2975. Add the following configuration to the `base/security/selinux/sepolicy/ohos_policy/drivers/adapter/vendor/hdf_devmgr.te` file: 298 ```text 299 # Add the security context configuration. 300 allow hdf_devmgr nnrt_host:binder { call transfer }; 301 allow hdf_devmgr nnrt_host:dir { search }; 302 allow hdf_devmgr nnrt_host:file { open read write }; 303 allow hdf_devmgr nnrt_host:process { getattr }; 304 ``` 305 3066. Add the following configuration to the `base/security/selinux/sepolicy/ohos_policy/drivers/adapter/vendor/init.te` file: 307 ```text 308 # Add the security context configuration. 309 allow init nnrt_host:process { rlimitinh siginh transition }; 310 ``` 311 3127. Modify the configuration in the `base/security/selinux/sepolicy/ohos_policy/startup/init/public/chipset_init.te` file. 313 314 Find the line containing `chipset_init`. 315 ```text 316 allow chipset_init { light_host input_user_host wifi_host camera_host power_host audio_host }:process { rlimitinh siginh transition }; 317 ``` 318 Add `nnrt_host` to the `host` list. 319 ```text 320 allow chipset_init { light_host input_user_host wifi_host camera_host power_host audio_host nnrt_host }:process { rlimitinh siginh transition }; 321 ``` 322 3238. Create the `nnrt_host.te` configuration file. 324 ```shell 325 # Create the nnrt folder. 326 mkdir base/security/selinux/sepolicy/ohos_policy/drivers/peripheral/nnrt 327 328 # Create the vendor folder. 329 mkdir base/security/selinux/sepolicy/ohos_policy/drivers/peripheral/nnrt/vendor 330 331 # Create the `nnrt_host.te` file. 332 touch base/security/selinux/sepolicy/ohos_policy/drivers/peripheral/nnrt/vendor/nnrt_host.te 333 ``` 334 3359. Write the following required permissions to the `nnrt_host.te` file: 336 ```text 337 allow nnrt_host dev_hdf_kevent:chr_file { ioctl }; 338 allow nnrt_host hilog_param:file { read open map }; 339 allow nnrt_host sh:binder { transfer }; 340 allow nnrt_host samgr:binder { call }; 341 allow nnrt_host dev_ashmem_file:chr_file { open }; 342 allow nnrt_host dev_unix_socket:dir { search }; 343 allow nnrt_host hdf_device_manager:hdf_devmgr_class { get }; 344 allow nnrt_host hdf_nnrt_device_service:hdf_devmgr_class { add get }; 345 allow nnrt_host dev_console_file:chr_file { read write }; 346 allow nnrt_host debug_param:file { read open map }; 347 allow nnrt_host sa_device_service_manager:samgr_class { get }; 348 allow nnrt_host hdf_devmgr:binder { call transfer }; 349 allow nnrt_host hdf_nnrt_device_service:binder { call }; 350 allow nnrt_host sysfs_devices_system_cpu:file { read open getattr }; 351 allow sh hdf_nnrt_device_service:hdf_devmgr_class { add get }; 352 allow sh hdf_hci_interface_service:hdf_devmgr_class { get }; 353 allow sh nnrt_host:dir { getattr search }; 354 allow sh nnrt_host:file { open read }; 355 allow sh nnrt_host:process { getattr }; 356 allow sh nnrt_host:binder { call }; 357 allow sh nnrt_host:fd { use }; 358 ``` 359 36010. Configure access permissions because SELinux uses the trustlist access permission mechanism. Upon service startup, run the `dmesg` command to view the AVC alarm, 361which provides a list of missing permissions. For details about the SELinux configuration, see [security_selinux](https://gitee.com/openharmony/security_selinux/blob/master/README-en.md). 362 ```shell 363 hdc_std shell 364 dmesg | grep nnrt 365 ``` 366 367#### Configuring the Component Build Entry 368Access the `chipset_common.json` file. 369```shell 370vim //productdefine/common/inherit/chipset_common.json 371``` 372Add the following configuration to `"subsystems"`, `"subsystem":"hdf"`, and `"components"`: 373```shell 374{ 375 "component": "drivers_peripheral_nnrt", 376 "features": [] 377} 378``` 379 380#### Deleting the out Directory and Building the Entire System 381```shell 382# Delete the out directory. 383rm -rf ./out 384 385# Build the entire system. 386./build.sh --product-name rk3568 –ccache --jobs=4 387``` 388 389 390### Commissioning and Verification 391On completion of service development, you can use XTS to verify its basic functions and compatibility. The operation procedure is as follows: 392 3931. Compile the **hats** test cases of NNRt in the `test/xts/hats/ai/nnrt/hdi` directory. 394 ```shell 395 # Go to the hats directory. 396 cd test/xts/hats 397 398 # Compile the `hats` test cases. 399 ./build.sh suite=hats system_size=standard product_name=rk3568 400 401 # Return to the root directory. 402 cd - 403 ``` 404 The test case executable file `HatsHdfNnrtFunctionTest` is exported to the `out/rk3568/suites/hats/testcases/` directory. 405 4062. Push the test case executable file to the `/data/local/tmp/` directory of the RK3568 device. 407 ```shell 408 # Push the test case executable file to the device. In this example, the executable file is HatsHdfNnrtFunctionTest. 409 hdc_std file send out/rk3568/suites/hats/testcases/HartsHdfNnrtFunctionTest /data/local/tmp/ 410 411 # Grant required permissions to the test case executable file. 412 hdc_std shell "chmod +x /data/local/tmp/HatsHdfNnrtFunctionTest" 413 ``` 414 4153. Execute the test cases and view the result. 416 ```shell 417 # Execute the test cases. 418 hdc_std shell "/data/local/tmp/HatsHdfNnrtFunctionTest" 419 ``` 420 421 According to the following example test report, all of the 47 test cases are successfully executed, indicating the service has passed the compatibility test. 422 ```text 423 ... 424 [----------] Global test environment tear-down 425 Gtest xml output finished 426 [==========] 47 tests from 3 test suites ran. (515 ms total) 427 [ PASSED ] 47 tests. 428 ``` 429 430### Development Example 431For the complete demo code, see [NNRt Service Implementation Example](https://gitee.com/openharmony/ai_neural_network_runtime/tree/master/example/drivers). 432 4331. Go to the root directory of OpenHarmony source code and create the `nnrt` folder in the `drivers/peripheral` directory. Then, copy the `example/driver/nnrt/v2_0` folder from the `foundation/ai/neural_network_runtime` directory of NNRt source code to the created `nnrt` folder. 434 ```shell 435 cp -r example/drivers/nnrt/v2_0 drivers/peripheral/nnrt 436 ``` 437 4382. Add the `bundle.json` file to the `drivers/peripheral/nnrt` directory. For details about how to write the `bundle.json` file, see [Implementing the HDI Service](#implementing-the-hdi-service). 439 4403. Add the dependency files of MindSpore Lite because the demo depends on the CPU operator of MindSpore Lite. 441 - Run the following command in the root directory of the OpenHarmony source code to build the MindSpore Lite dynamic library: The MindSpore source code is stored in `third_party/mindspore` in the root directory of the OpenHarmony source code. 442 ```shell 443 # Build the MindSpore Lite dynamic libraries. 444 ./build.sh --product-name rk3568 -ccaache --jobs 4 --build-target mindspore_lib 445 ``` 446 - Create the `mindspore` folder in the `drivers/peripheral/nnrt/v2_0` directory to store the dynamic libraries and header files of MindSpore Lite. 447 ```shell 448 mkdir drivers/peripheral/nnrt/v2_0/mindspore 449 ``` 450 - Copy the `mindspore-src/source/include` directory in the MindSpore source code to the `drivers/peripheral/nnrt/v2_0/mindspore` directory. 451 ```shell 452 cp third_party/mindspore/mindspore-src/source/include drivers/peripheral/nnrt/v2_0/mindspore 453 ``` 454 - Create and copy the `schema` file of MindSpore Lite. 455 ```shell 456 # Create the mindspore_schema folder. 457 mkdir drivers/peripheral/nnrt/v2_0/hdi_cpu_service/include/mindspore_schema 458 459 # Copy the MindSpore schema file from the third_party directory. 460 cp third_party/mindspore/mindspore-src/source/mindspore/lite/schema/* drivers/peripheral/nnrt/v2_0/hdi_cpu_service/include/mindspore_schema/ 461 ``` 462 - Copy the MindSpore Lite dynamic library to the `mindspore` directory. 463 ```shell 464 # Create the `mindspore` folder in the `drivers/peripheral/nnrt/v2_0/mindspore` directory. 465 mkdir drivers/peripheral/nnrt/v2_0/mindspore/mindspore 466 467 # Copy the MindSpore dynamic libraries from the `out` folder to the `drivers/peripheral/nnrt/v2_0/mindspore/mindspore` directory. 468 cp out/rk3568/package/phone/system/lib/libmindspore-lite.so drivers/peripheral/nnrt/v2_0/mindspore/mindspore/ 469 ``` 4704. Follow the [development procedure](#development-procedure) to complete other configurations. 471
