README.md
1# Intelligent Voice Framework
2
3## Overview
4
5### Introduction
6
7The intelligent voice framework consists of the intelligent voice service framework and intelligent voice driver. It implements voice enrollment and voice wakeup.
8
9**Figure 1** Architecture of the intelligent voice framework
10
11
12
13The intelligent voice service framework provides the following features:
14
15- System event monitoring: monitoring system events such as unlocking upon power-on and screen-on/off
16- Concurrency policy: intelligent voice service concurrency management
17
18- Intelligent voice service: voice enrollment, voice wakeup, and more
19- Sound trigger: Digital Signal Processor (DSP) model loading, DSP algorithm enabling/disabling, and DSP event processing
20
21The intelligent voice driver provides the following features:
22
23- Engine algorithm: intelligent voice algorithm engine and event reporting
24
25- Device driver: DSP model loading/unloading, DSP algorithm enabling/disabling, event reporting, and hardware-related channel configuration
26
27### Basic Concepts
28- Voice enrollment: process of converting a wakeup word spoken by a user into an acoustic model and a voiceprint feature, which will be used for comparison during voice wakeup
29- Voice wakeup: process of checking whether the current speaker is a registered user and if yes, waking up the system
30- DSP chip: chip that implements digital signal processing
31
32### Directory Structure
33
34The structure of the repository directory is as follows:
35
36```shell
37/foundation/ai/intelligent_voice_framework # Service code of the intelligent audio framework
38├── frameworks # Framework code
39│ ├── native # Internal API implementation
40│ └── js # External API implementation
41├── interfaces # API code
42│ ├── inner_api # Internal APIs
43│ └── kits # External APIs
44├── sa_profile # Service configuration profile
45├├── services # Service code
46├── LICENSE # License file
47├── tests # Developer test
48└── utils # Public functions
49```
50
51### Constraints
52
53Currently, the intelligent voice framework supports the enrollment and wakeup of only one wakeup word.
54
55
56## Available APIs
57### APIs Used for Voice Enrollment
58
59| API | Description |
60| ------------------------------------------------------------ | ------------------ |
61| createEnrollIntelligentVoiceEngine(descriptor: EnrollIntelligentVoiceEngineDescriptor): EnrollIntelligentVoiceEngine | Creates an enrollment engine. |
62| init(config: EnrollEngineConfig): EnrollIntelligentVoiceEngineCallbackInfo | Initializes this enrollment engine. |
63| start(isLast: boolean): EnrollIntelligentVoiceEngineCallbackInfo | Starts enrollment. |
64| stop(): void | Stops enrollment. |
65| commit(): EnrollIntelligentVoiceEngineCallbackInfo | Commits the enrollment data. |
66| setWakeupHapInfo(info: WakeupHapInfo): void | Sets the wakeup application information.|
67| setSensibility(sensibility: SensibilityType): void | Sets the sensitivity. |
68| release(): void | Releases this enrollment engine. |
69
70### APIs Used for Voice Wakeup
71
72| API | Description |
73| ------------------------------------------------------------ | ------------------ |
74| createWakeupIntelligentVoiceEngine(descriptor: WakeupIntelligentVoiceEngineDescriptor): WakeupIntelligentVoiceEngine | Creates a wakeup engine. |
75| setWakeupHapInfo(info: WakeupHapInfo): void | Sets the wakeup application information.|
76| setSensibility(sensibility: SensibilityType): void | Sets the sensitivity. |
77| on(type: 'wakeupIntelligentVoiceEvent', callback: Callback<WakeupIntelligentVoiceEngineCallbackInfo>): void | Subscribes to wakeup events. |
78| release(): void | Releases this wakeup engine. |
79
80
81## How to Develop
82
83### Voice Enrollment
84
85The voice enrollment process is an interaction process initiated by a user through the enrollment page of an application. The main process is as follows:
861. A user starts enrollment (creating and initializing the enrollment engine), and the enrollment page is displayed.
872. The enrollment page asks the user to speak a wakeup word, and the user speaks the wakeup word (starting enrollment). The enrollment page asks the user to speak the wakeup word again several times.
883. After the enrollment data is committed, the enrollment process is complete.
89The code snippet is as follows:
90
91```js
92// Import the intelligentVoice module.
93import intelligentVoice from '@ohos.ai.intelligentVoice';
94
95// Obtain the intelligent audio management service.
96var manager = intellVoice.getIntelligentVoiceManager();
97if (manager == null) {
98 console.error("Get IntelligentVoiceManager failed.");
99} else {
100 console.info("Get IntelligentVoiceManager success.");
101 return;
102}
103
104// Create an enrollment engine.
105var engine = null;
106let engineDescriptor = {
107 wakeupPhrase: '', // Set a wakeup word.
108}
109await intellVoice.createEnrollIntelligentVoiceEngine(engineDescriptor).then((data) => {
110 engine = data;
111 console.info('Create EnrollIntelligentVoice Engine finish');
112}).catch((err) => {
113 console.error('Create EnrollIntelligentVoice Engine failed, err: ' + err.message);
114});
115if (engine == null) {
116 console.error('Create EnrollIntelligentVoice Engine failed');
117 return;
118}
119
120// Initialize the enrollment engine.
121let config = {
122 language: "zh", // Chinese
123 area: "CN", // China
124}
125engine.init(config).then((data) => {
126 console.info('Init EnrollIntelligentVoice Engine finish');
127}).catch((err) => {
128 console.info('Init EnrollIntelligentVoice Engine failed, err: '+ err.message);
129});
130
131// Start enrollment.
132let isLast = true; // The value true means that this is the last time to start enrollment, and false means the opposite. The value true is used here.
133engine.start(isLast).then((data) => {
134 console.info('Start enrollment finish');
135}).catch((err) => {
136 console.info('Start enrollment failed, err: '+ err.message);
137});
138
139// Commit the enrollment data.
140engine.commit().then((data) => {
141 console.info('Commit enroll result finish');
142}).catch((err) => {
143 console.info('Commit enroll result failed, err: '+ err.message);
144});
145
146// Deliver the voice wakeup application information.
147let info = {
148 bundleName: "demo", // Bundle name of the application. demo here is for reference only. Set this parameter based on your application.
149 abilityName: "demo", // Ability name of the application. demo here is for reference only. Set this parameter based on your application.
150}
151engine.setWakeupHapInfo(info).then((data) => {
152 console.info('Set wakeup hap info finish');
153}).catch((err) => {
154 console.info('Set wakeup hap info failed, err: '+ err.message);
155});
156
157// Release the enrollment engine.
158engine.release().then((data) => {
159 console.info('Release EnrollIntelligentVoice engine success.');
160}).catch((err) => {
161 console.info('Release EnrollIntelligentVoice engine failed, err: '+ err.message);
162});
163```
164
165### Voice Wakeup
166
167Voice wakeup is controlled by the intelligent voice framework. Upper-layer applications only need to create a wakeup engine by calling **createWakeupIntelligentVoiceEngine** and then subscribe to wakeup events.
168
169```js
170// Obtain the wakeup engine.
171var engine = null;
172let engineDescriptor = {
173 needApAlgEngine: true, // Specify whether the framework needs to provide the AP algorithm engine.
174 wakeupPhrase: '', // Set a wakeup word.
175}
176await intellVoice.createWakeupIntelligentVoiceEngine(engineDescriptor).then((data) => {
177 engine = data;
178 console.info('Create WakeupIntelligentVoice Engine finish');
179}).catch((err) => {
180 console.error('Create WakeupIntelligentVoice Engine failed, err: ' + err.message);
181});
182if (engine == null) {
183 console.error('Create WakeupIntelligentVoice Engine failed');
184 return;
185}
186
187// Subscribe to wakeup events.
188engine.on('wakeupIntelligentVoiceEvent',(callback) => {
189 console.info('wakeupIntelligentVoiceEvent CallBackInfo:')
190 for (let prop in callback) {
191 console.info('wakeupIntelligentVoiceEvent prop: ' + prop);
192 console.info('wakeupIntelligentVoiceEvent value: ' + callback[prop]);
193 }
194});
195```
196
197## Repositories Involved
198
199intelligent_voice_framework
200
README_zh.md
1# 智能语音部件
2
3## 概述
4
5### 功能简介
6
7智能语音组件包括智能语音服务框架和智能语音驱动，主要实现了语音注册及语音唤醒相关功能。
8
9**图 1** 智能语音组件架构图
10
11
12
13智能语音服务框架支持如下功能：
14系统事件监测：开机解锁、亮灭屏等系统事件监测
15并发策略：智能语音业务并发管理
16智能语音业务：语音注册、语音唤醒等智能语音业务处理
17声音触发器：DSP模型加载、DSP算法启停、DSP事件处理
18
19智能语音驱动支持如下功能：
20引擎算法：智能语音算法引擎以及事件上报
21设备驱动：DSP模型加载卸载、算法启停、事件上报以及硬件相关通路配置
22
23### 基本概念
24- 语音注册：将用户说的唤醒词转换为声学模型以及声纹特征，以便后续的语音唤醒。
25- 语音唤醒：判断当前说话人是否为已注册的特定用户。
26- DSP：数字信号处理器（Digital Signal Processors），DSP芯片即指能够实现数字信号处理技术的芯片。
27
28### 目录结构
29
30仓目录结构如下：
31
32```shell
33/foundation/ai/intelligent_voice_framework # 智能音频组件业务代码
34├── frameworks # 框架代码
35│ ├── native # 内部接口实现
36│ └── js # 外部接口实现
37├── interfaces # 接口代码
38│ ├── inner_api # 内部接口
39│ └── kits # 外部接口
40├── sa_profile # 服务配置文件
41├── services # 服务代码
42├── LICENSE # 证书文件
43├── tests # 开发者测试
44└── utils # 公共函数
45```
46
47### 约束与限制
48
49- 智能语音服务当前只支持一个唤醒词的注册以及唤醒。
50
51## 编译构建
52
53在OpenHarmony源码根目录下，调用以下指令，单独编译intelligent_voice_framework。
54```shell
55./build.sh --product-name rk3568 --ccache --build-target intelligent_voice_framework --jobs 4
56```
57> **说明：**
58--product-name：产品名称，例如Hi3516DV300、rk3568等。
59--ccache：编译时使用缓存功能。
60--build-target: 编译的部件名称。
61--jobs：编译的线程数，可加速编译。
62
63
64## 接口说明
65### 语音注册接口说明
66
67|接口名称|接口描述|
68|---|---|
69|createEnrollIntelligentVoiceEngine(descriptor: EnrollIntelligentVoiceEngineDescriptor): EnrollIntelligentVoiceEngine|创建注册引擎。|
70|init(config: EnrollEngineConfig): EnrollIntelligentVoiceEngineCallbackInfo|初始化注册引擎。|
71|start(isLast: boolean): EnrollIntelligentVoiceEngineCallbackInfo|启动注册。|
72|stop(): void|停止注册。|
73|commit(): EnrollIntelligentVoiceEngineCallbackInfo|确认注册结果。|
74|setWakeupHapInfo(info: WakeupHapInfo): void|设置唤醒应用信息。|
75|setSensibility(sensibility: SensibilityType): void|设置灵敏度。|
76|release(): void|释放注册引擎。|
77
78### 语音唤醒接口说明
79
80|接口名称|接口描述|
81|---|---|
82|createWakeupIntelligentVoiceEngine(descriptor: WakeupIntelligentVoiceEngineDescriptor): WakeupIntelligentVoiceEngine|创建唤醒引擎。|
83|setWakeupHapInfo(info: WakeupHapInfo): void|设置唤醒应用信息。|
84|setSensibility(sensibility: SensibilityType): void|设置灵敏度。|
85|on(type: 'wakeupIntelligentVoiceEvent', callback: Callback<WakeupIntelligentVoiceEngineCallbackInfo>): void|订阅唤醒事件。|
86|release(): void|释放唤醒引擎。|
87
88
89## 开发实例
90
91### 语音注册
92
93语音注册流程是用户通过应用的注册界面主动发起的交互流程，主要流程如下：
941. 用户启动注册（创建注册引擎并初始化注册引擎）后，进入注册界面。
952. 界面提示用户说出唤醒词，用户根据提示说出相应唤醒词（启动注册），界面会让用户再次重复说出唤醒词，直到最后一次。
963. 注册完成（确认注册结果）后，注册流程全部完成。
97示例代码如下：
98
99```js
100// 引入智能音频
101import intelligentVoice from '@ohos.ai.intelligentVoice';
102
103// 获取智能音频管理服务
104var manager = intellVoice.getIntelligentVoiceManager();
105if (manager == null) {
106 console.error("Get IntelligentVoiceManager failed.");
107} else {
108 console.info("Get IntelligentVoiceManager success.");
109 return;
110}
111
112// 创建注册引擎
113var engine = null;
114let engineDescriptor = {
115 wakeupPhrase: '', // 设置唤醒词
116}
117await intellVoice.createEnrollIntelligentVoiceEngine(engineDescriptor).then((data) => {
118 engine = data;
119 console.info('Create EnrollIntelligentVoice Engine finish');
120}).catch((err) => {
121 console.error('Create EnrollIntelligentVoice Engine failed, err: ' + err.message);
122});
123if (engine == null) {
124 console.error('Create EnrollIntelligentVoice Engine failed');
125 return;
126}
127
128// 初始化注册引擎
129let config = {
130 language: "zh", // 中文
131 area: "CN", // 中国
132}
133engine.init(config).then((data) => {
134 console.info('Init EnrollIntelligentVoice Engine finish');
135}).catch((err) => {
136 console.info('Init EnrollIntelligentVoice Engine failed, err: '+ err.message);
137});
138
139// 启动注册
140let isLast = true; // true: 最后一次启动，false: 非最后一次启动，实例为true
141engine.start(isLast).then((data) => {
142 console.info('Start enrollment finish');
143}).catch((err) => {
144 console.info('Start enrollment failed, err: '+ err.message);
145});
146
147// 确认注册结果
148engine.commit().then((data) => {
149 console.info('Commit enroll result finish');
150}).catch((err) => {
151 console.info('Commit enroll result failed, err: '+ err.message);
152});
153
154// 下发语音唤醒应用信息
155let info = {
156 bundleName: "demo", // 应用的bundle name，demo只是个参考例子，具体填写由应用确定
157 abilityName: "demo", // 应用的ability name，demo只是个参考例子，具体填写由应用确定
158}
159engine.setWakeupHapInfo(info).then((data) => {
160 console.info('Set wakeup hap info finish');
161}).catch((err) => {
162 console.info('Set wakeup hap info failed, err: '+ err.message);
163});
164
165// 释放注册引擎
166engine.release().then((data) => {
167 console.info('Release EnrollIntelligentVoice engine success.');
168}).catch((err) => {
169 console.info('Release EnrollIntelligentVoice engine failed, err: '+ err.message);
170});
171```
172
173### 语音唤醒
174
175语言唤醒由智能语音组件控制，上层应用只需要调用`createWakeupIntelligentVoiceEngine`获取唤醒引擎后注册唤醒事件回调即可。
176
177```js
178// 获取唤醒引擎
179var engine = null;
180let engineDescriptor = {
181 needApAlgEngine: true, // 是否需要框架提供ap侧算法引擎
182 wakeupPhrase: '', // 设置唤醒词
183}
184await intellVoice.createWakeupIntelligentVoiceEngine(engineDescriptor).then((data) => {
185 engine = data;
186 console.info('Create WakeupIntelligentVoice Engine finish');
187}).catch((err) => {
188 console.error('Create WakeupIntelligentVoice Engine failed, err: ' + err.message);
189});
190if (engine == null) {
191 console.error('Create WakeupIntelligentVoice Engine failed');
192 return;
193}
194
195// 注册事件回调
196engine.on('wakeupIntelligentVoiceEvent',(callback) => {
197 console.info('wakeupIntelligentVoiceEvent CallBackInfo:')
198 for (let prop in callback) {
199 console.info('wakeupIntelligentVoiceEvent prop: ' + prop);
200 console.info('wakeupIntelligentVoiceEvent value: ' + callback[prop]);
201 }
202});
203```
204
205## 相关仓
206
207intelligent_voice_framework
208
