Audio Augmentation:VTLP

语音增强：VTLP

Vocal Tract Length Perturbation(VTLP)
语音识别时，说话人之间的声道之间形状的变化会影响语音识别的结果，因此每个说话人同一句话的发音是不同的。为去除不同人的声道对同一个词发音的影响，需要对音频进行处理，VLTP即用于去除说话人身份对发音的影响。VTLP是在对每一句语音时，随机生成一个扭曲因子用于对频谱的频率轴进行扭曲，来去除声道差别对语音识别结果的影响。

实现

nlp的语音增强库nlpaug
源代码：将音频序列进行短时傅里叶变换STFT得到时域-频域二维矩阵（数值代表振幅），然后对频域进行扰动(perturbation)，而后通过短时傅里叶逆变换ISTFT将其转回音频序列，注意，转回后的音频序列长度略有不同。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
parent_dir = os.path.join('.', 'data', 'respiratory-sound-database', 'Respiratory_Sound_Database',
                          'Respiratory_Sound_Database')

audio, sample_rate = librosa.load(os.path.join(parent_dir, 'audio_and_txt_files', '7383-3-0-0.wav'),
                                  mono=True,
                                  res_type='kaiser_fast', sr=22050, duration=20)
print(audio.shape)
aug = naa.VtlpAug(sampling_rate=sample_rate) //具体用法见nlpaug库文档
aug_data = aug.augment(audio)
print(aug_data.shape)

data_stft = librosa.stft(audio)
data_istft = librosa.istft(data_stft)
print(data_istft.shape)

x = np.random.uniform(0, 1, size=(5, 22050 * 5))
print(x.shape)
print(x[0].shape)
for i in range(x.shape[0]):
    audio = x[i, :]
    aug_data = aug.augment(audio)
    print(aug_data.shape)

结果

1
2
3
4
5
6
7
8
9
10
(88200,)
(88028,)
(88064,)
(5, 110250)
(110250,)
(109779,)
(109779,)
(109779,)
(109779,)
(109779,)

源码分析

上述调用aug.augment(audio)将会调用一下类，然后调用manipulate()方法进行VTLP增强，为方便理解，隐去其他基类调用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
import numpy as np
try:
    import librosa
except ImportError:
    # No installation required if not using this function
    pass

from nlpaug.model.audio import Audio


class Vtlp(Audio):
    # https://pdfs.semanticscholar.org/3de0/616eb3cd4554fdf9fd65c9c82f2605a17413.pdf
    def __init__(self, sampling_rate, zone=(0.2, 0.8), coverage=0.1, duration=None, factor=(0.9, 1.1), fhi=4800,
                 stateless=True):
        """
        :param int sampling_rate: Sampling rate of input audio. Mandatory if duration is provided.
        :param tuple zone: Assign a zone for augmentation. Default value is (0.2, 0.8) which means that no any
            augmentation
        will be applied in first 20% and last 20% of whole audio.
        :param float coverage: Portion of augmentation. Value should be between 0 and 1. If `1` is assigned, augment
            operation will be applied to target audio segment. For example, the audio duration is 60 seconds while
            zone and coverage are (0.2, 0.8) and 0.7 respectively. 42 seconds ((0.8-0.2)*0.7*60) audio will be
            augmented.
        :param int duration: Duration of augmentation (in second). Default value is None. If value is provided.
            `coverage` value will be ignored.
        :param int fhi: Boundary frequency. Default value is 4800.
        :param tuple factor: Warping factor
        """
        super().__init__(zone=zone, coverage=coverage, duration=duration, sampling_rate=sampling_rate,
                         stateless=stateless, factor=factor)

        try:
            import librosa
        except ModuleNotFoundError:
            raise ModuleNotFoundError('Missed librosa library. Install import librosa by `pip install librosa`')

        self.fhi = fhi

    @classmethod
    def get_scale_factors(cls, freq_dim, sampling_rate, fhi=4800, alpha=0.9):
        factors = []
        freqs = np.linspace(0, 1, freq_dim)

        scale = fhi * min(alpha, 1)
        f_boundary = scale / alpha
        half_sr = sampling_rate / 2

        for f in freqs:
            f *= sampling_rate
            if f <= f_boundary:
                factors.append(f * alpha)
            else:
                warp_freq = half_sr - (half_sr - scale) / (half_sr - scale / alpha) * (half_sr - f)
                factors.append(warp_freq)

        return np.array(factors)

    # https://github.com/YerevaNN/Spoken-language-identification/blob/master/augment_data.py#L26
    def _manipulate(self, audio, sampling_rate, factor):
        stft = librosa.core.stft(audio)
        time_dim, freq_dim = stft.shape
        data_type = type(stft[0][0])

        factors = self.get_scale_factors(freq_dim, sampling_rate, alpha=factor)
        factors *= (freq_dim - 1) / max(factors)
        new_stft = np.zeros([time_dim, freq_dim], dtype=data_type)

        for i in range(freq_dim):
            # first and last freq
            if i == 0 or i + 1 >= freq_dim:
                new_stft[:, i] += stft[:, i]
            else:
                warp_up = factors[i] - np.floor(factors[i])
                warp_down = 1 - warp_up
                pos = int(np.floor(factors[i]))

                new_stft[:, pos] += warp_down * stft[:, i]
                new_stft[:, pos+1] += warp_up * stft[:, i]

        return librosa.core.istft(new_stft)

    def get_warping_level(self):
        return np.random.uniform(self.factor[0], self.factor[1])

    def manipulate(self, data):
        if self.duration is None:
            start_pos, end_pos = self.get_augment_range_by_coverage(data)
        else:
            start_pos, end_pos = self.get_augment_range_by_duration(data)

        factor = self.get_warping_level()
        aug_data = self._manipulate(data[start_pos:end_pos], sampling_rate=self.sampling_rate, factor=factor)

        if not self.stateless:
            self.start_pos = start_pos
            self.end_pos = end_pos
            self.aug_factor = factor
            self.aug_data = aug_data

        return np.concatenate((data[:start_pos], aug_data, data[end_pos:]), axis=0).astype(type(data[0]))

        # if start_pos > 0:
        #     aug_data = np.concatenate((data[:start_pos], aug_data), axis=0)
        # if end_pos < len(data):
        #     aug_data = np.concatenate((aug_data, data[end_pos:]), axis=0)
        #
        # return aug_data.astype(type(data[0]))

参考文献

扫一扫，分享到微信

微信分享二维码

tag:

    缺失模块。
    1、请确保node版本大于6.2
    2、在博客根目录（注意不是yilia根目录）执行以下命令：
    npm i hexo-generator-json-content --save

    3、在根目录_config.yml里添加配置： 

      jsonContent:
    meta: false
    pages: false
    posts:
      title: true
      date: true
      path: true
      text: false
      raw: false
      content: false
      slug: false
      updated: false
      comments: false
      link: false
      permalink: false
      excerpt: false
      categories: false
      tags: true

做一个安静细微的人，<br> 于角落里自在开放，<br> 默默悦人，<br> 却始终不引起过分热闹的关注，<br> 保有独立而随意的品格，<br> 这就很好。<br><br> Stick to what you insist on,<br> believe what you believe!<br> Life hastily for decades,<br> do what I can!