1# Intelligent Voice Framework
2
3## Overview
4
5### Introduction
6
7The intelligent voice framework consists of the intelligent voice service framework and intelligent voice driver. It implements voice enrollment and voice wakeup.
8
9**Figure 1** Architecture of the intelligent voice framework
10
11![image.png](figures/en-intelligent-voice-framework.png)
12
13The intelligent voice service framework provides the following features:
14
15- System event monitoring: monitoring system events such as unlocking upon power-on and screen-on/off
16- Concurrency policy: intelligent voice service concurrency management
17
18- Intelligent voice service: voice enrollment, voice wakeup, and more
19- Sound trigger: Digital Signal Processor (DSP) model loading, DSP algorithm enabling/disabling, and DSP event processing
20
21The intelligent voice driver provides the following features:
22
23- Engine algorithm: intelligent voice algorithm engine and event reporting
24
25- Device driver: DSP model loading/unloading, DSP algorithm enabling/disabling, event reporting, and hardware-related channel configuration
26
27### Basic Concepts
28- Voice enrollment: process of converting a wakeup word spoken by a user into an acoustic model and a voiceprint feature, which will be used for comparison during voice wakeup
29- Voice wakeup: process of checking whether the current speaker is a registered user and if yes, waking up the system
30- DSP chip: chip that implements digital signal processing
31
32### Directory Structure
33
34The structure of the repository directory is as follows:
35
36```shell
37/foundation/ai/intelligent_voice_framework  # Service code of the intelligent audio framework
38├── frameworks                                      # Framework code
39│   ├── native                                      # Internal API implementation
40│   └── js                                          # External API implementation
41├── interfaces                                      # API code
42│   ├── inner_api                                   # Internal APIs
43│   └── kits                                        # External APIs
44├── sa_profile                                      # Service configuration profile
45├├── services                                        # Service code
46├── LICENSE                                         # License file
47├── tests                                           # Developer test
48└── utils                                           # Public functions
49```
50
51### Constraints
52
53Currently, the intelligent voice framework supports the enrollment and wakeup of only one wakeup word.
54
55
56## Available APIs
57### APIs Used for Voice Enrollment
58
59| API                                                    | Description          |
60| ------------------------------------------------------------ | ------------------ |
61| createEnrollIntelligentVoiceEngine(descriptor: EnrollIntelligentVoiceEngineDescriptor): EnrollIntelligentVoiceEngine | Creates an enrollment engine.    |
62| init(config: EnrollEngineConfig): EnrollIntelligentVoiceEngineCallbackInfo | Initializes this enrollment engine.  |
63| start(isLast: boolean): EnrollIntelligentVoiceEngineCallbackInfo | Starts enrollment.        |
64| stop(): void                                                 | Stops enrollment.        |
65| commit(): EnrollIntelligentVoiceEngineCallbackInfo           | Commits the enrollment data.    |
66| setWakeupHapInfo(info: WakeupHapInfo): void                  | Sets the wakeup application information.|
67| setSensibility(sensibility: SensibilityType): void           | Sets the sensitivity.      |
68| release(): void                                              | Releases this enrollment engine.    |
69
70### APIs Used for Voice Wakeup
71
72| API                                                    | Description          |
73| ------------------------------------------------------------ | ------------------ |
74| createWakeupIntelligentVoiceEngine(descriptor: WakeupIntelligentVoiceEngineDescriptor): WakeupIntelligentVoiceEngine | Creates a wakeup engine.    |
75| setWakeupHapInfo(info: WakeupHapInfo): void                  | Sets the wakeup application information.|
76| setSensibility(sensibility: SensibilityType): void           | Sets the sensitivity.      |
77| on(type: 'wakeupIntelligentVoiceEvent', callback: Callback<WakeupIntelligentVoiceEngineCallbackInfo>): void | Subscribes to wakeup events.    |
78| release(): void                                              | Releases this wakeup engine.    |
79
80
81## How to Develop
82
83### Voice Enrollment
84
85The voice enrollment process is an interaction process initiated by a user through the enrollment page of an application. The main process is as follows:
861. A user starts enrollment (creating and initializing the enrollment engine), and the enrollment page is displayed.
872. The enrollment page asks the user to speak a wakeup word, and the user speaks the wakeup word (starting enrollment). The enrollment page asks the user to speak the wakeup word again several times.
883. After the enrollment data is committed, the enrollment process is complete.
89The code snippet is as follows:
90
91```js
92// Import the intelligentVoice module.
93import intelligentVoice from '@ohos.ai.intelligentVoice';
94
95// Obtain the intelligent audio management service.
96var manager = intellVoice.getIntelligentVoiceManager();
97if (manager == null) {
98    console.error("Get IntelligentVoiceManager failed.");
99} else {
100    console.info("Get IntelligentVoiceManager success.");
101    return;
102}
103
104// Create an enrollment engine.
105var engine = null;
106let engineDescriptor = {
107    wakeupPhrase: '',                            // Set a wakeup word.
108}
109await intellVoice.createEnrollIntelligentVoiceEngine(engineDescriptor).then((data) => {
110    engine = data;
111    console.info('Create EnrollIntelligentVoice Engine finish');
112}).catch((err) => {
113    console.error('Create EnrollIntelligentVoice Engine failed, err: ' + err.message);
114});
115if (engine == null) {
116    console.error('Create EnrollIntelligentVoice Engine failed');
117    return;
118}
119
120// Initialize the enrollment engine.
121let config = {
122    language: "zh", // Chinese
123    area: "CN", // China
124}
125engine.init(config).then((data) => {
126    console.info('Init EnrollIntelligentVoice Engine finish');
127}).catch((err) => {
128    console.info('Init EnrollIntelligentVoice Engine failed, err: '+ err.message);
129});
130
131// Start enrollment.
132let isLast = true; // The value true means that this is the last time to start enrollment, and false means the opposite. The value true is used here.
133engine.start(isLast).then((data) => {
134    console.info('Start enrollment finish');
135}).catch((err) => {
136    console.info('Start enrollment failed, err: '+ err.message);
137});
138
139// Commit the enrollment data.
140engine.commit().then((data) => {
141    console.info('Commit enroll result finish');
142}).catch((err) => {
143    console.info('Commit enroll result failed, err: '+ err.message);
144});
145
146// Deliver the voice wakeup application information.
147let info = {
148    bundleName: "demo", // Bundle name of the application. demo here is for reference only. Set this parameter based on your application.
149    abilityName: "demo", // Ability name of the application. demo here is for reference only. Set this parameter based on your application.
150}
151engine.setWakeupHapInfo(info).then((data) => {
152    console.info('Set wakeup hap info finish');
153}).catch((err) => {
154    console.info('Set wakeup hap info failed, err: '+ err.message);
155});
156
157// Release the enrollment engine.
158engine.release().then((data) => {
159    console.info('Release EnrollIntelligentVoice engine success.');
160}).catch((err) => {
161    console.info('Release EnrollIntelligentVoice engine failed, err: '+ err.message);
162});
163```
164
165### Voice Wakeup
166
167Voice wakeup is controlled by the intelligent voice framework. Upper-layer applications only need to create a wakeup engine by calling **createWakeupIntelligentVoiceEngine** and then subscribe to wakeup events.
168
169```js
170// Obtain the wakeup engine.
171var engine = null;
172let engineDescriptor = {
173    needApAlgEngine: true, // Specify whether the framework needs to provide the AP algorithm engine.
174    wakeupPhrase: '', // Set a wakeup word.
175}
176await intellVoice.createWakeupIntelligentVoiceEngine(engineDescriptor).then((data) => {
177    engine = data;
178    console.info('Create WakeupIntelligentVoice Engine finish');
179}).catch((err) => {
180    console.error('Create WakeupIntelligentVoice Engine failed, err: ' + err.message);
181});
182if (engine == null) {
183    console.error('Create WakeupIntelligentVoice Engine failed');
184    return;
185}
186
187// Subscribe to wakeup events.
188engine.on('wakeupIntelligentVoiceEvent',(callback) => {
189    console.info('wakeupIntelligentVoiceEvent CallBackInfo:')
190    for (let prop in callback) {
191        console.info('wakeupIntelligentVoiceEvent prop: ' + prop);
192        console.info('wakeupIntelligentVoiceEvent value: ' + callback[prop]);
193    }
194});
195```
196
197## Repositories Involved
198
199intelligent_voice_framework
200