Omitir navegación

Russian Open Speech To Text

Speech to Text Russian Open STT

Recopilación de muestras de voz procedentes de varios orígenes de audio. El conjunto de datos contiene breves clips de audio en ruso.

Probablemente, el mayor conjunto de datos de STT público en ruso hasta la fecha:

  • ~16 millones de expresiones;
  • ~20 000 horas;
  • 2,3 TB (sin comprimir en formato .wav en int16), 356 GB en .opus;
  • Ahora todos los archivos se han migrado a Opus, excepto los conjuntos de datos de validación;

El objetivo principal del conjunto de datos es entrenar modelos de conversión de voz en texto.

Composición del conjunto de datos

El tamaño del conjunto de datos se proporciona en archivos .wav.

Dataset Grabaciones de voz Horas GB Segundos/caracteres Comentario Anotación Calidad/ruido
radio_v4 (*) 7 603 192 10 430 1195 5 s/68 Radio Alinear 95 %/nítida
public_speech (*) 1 700 060 2709 301 6 s/79 Voz pública Alinear 95 %/nítida
audiobook_2 1 149 404 1511 162 5 s/56 Libros Alinear 95 %/nítida
radio_2 651 645 1439 154 8 s/110 Radio Alinear 95 %/nítida
public_youtube1120 1 410 979 1104 237 3 s/34 YouTube Subtítulos 95 %/~nítida
public_youtube700 759 483 701 75 3 s/43 YouTube Subtítulos 95 %/~nítida
tts_russian_addresses 1 741 838 754 81 2 s/20 Direcciones 4 voces de TTS 100 %/nítida
asr_public_phone_calls_2 603 797 601 66 4 s/37 Llamadas de teléfono ASR 70 %/con ruido
public_youtube1120_hq 369 245 291 31 3 s/37 YouTube HQ Subtítulos 95 %/~nítida
asr_public_phone_calls_1 233 868 211 23 3 s/29 Llamadas de teléfono ASR 70 %/con ruido
radio_v4_add (*) 92 679 157 18 6 s/80 Radio Alinear 95 %/nítida
asr_public_stories_2 78 186 78 9 4 s/43 Libros ASR 80 %/nítida
asr_public_stories_1 46 142 38 4 3 s/30 Libros ASR 80 %/nítida
public_series_1 20 243 17 2 3 s/38 YouTube Subtítulos 95 %/~nítida
asr_calls_2_val 12 950 7,7 2 2 s/34 Llamadas de teléfono Anotación manual 99 %/nítida
public_lecture_1 6803 6 1 3 s/47 Lecturas Subtítulos 95 %/nítida
buriy_audiobooks_2_val 7850 4,9 1 2 s/31 Libros Anotación manual 99 %/nítida
public_youtube700_val 7311 4,5 1 2 s/35 YouTube Anotación manual 99 %/nítida

(*) Solo se proporciona una muestra de datos con los archivos .txt.

Metodología para las anotaciones

El conjunto de datos se ha compilado a partir de orígenes públicos. Las secuencias largas están divididas en fragmentos de audio usando detección y alineación de actividad de voz. Algunos tipos de audio se han anotado automáticamente y se han comprobado estadísticamente o usando heurística.

Volúmenes de datos y frecuencia de actualización

El tamaño total del conjunto de datos es de 350 GB. El tamaño total del conjunto de datos con etiquetas compartidas públicamente es de 130 GB.

No es probable que el conjunto de datos en sí se actualice para que sea compatible con versiones anteriores. Siga el repositorio original para obtener bancos de pruebas y excluir archivos.

Pueden agregarse nuevos dominios e idiomas en el futuro.

Normalización de audio

Todos los archivos se han normalizado para facilitar y agilizar el procesamiento y el aumento en tiempo de ejecución del siguiente modo:

  • Convertido a mono, si es necesario;
  • Convertido a una frecuencia de muestreo de 16 kHz, si es necesario;
  • Almacenado como enteros de 16 bits;
  • Convertido a OPUS;

Metodología de base datos en disco

Cada archivo de audio (.wav, binario) tiene un algoritmo hash. El algoritmo hash se usa para crear una jerarquía de carpetas con el fin de optimizar el funcionamiento de fs.

target_format = 'wav' wavb = wav.tobytes() f_hash = hashlib.sha1(wavb).hexdigest() store_path = Path(root_folder, f_hash[0], f_hash[1:3], f_hash[3:15] + '.' + target_format)
Descargas

El conjunto de datos se proporciona con dos formatos:

  • Hay archivos disponibles a través de Azure Blob Storage y/o vínculos directos;
  • Los archivos originales están disponibles a través de Azure Blob Storage;

Todo está almacenado en https://azureopendatastorage.blob.core.windows.net/openstt/

Estructura de carpetas:

└── ru_open_stt_opus <= archived folders │ │ │ ├── archives │ │ ├── asr_calls_2_val.tar.gz <= tar.gz archives with opus and wav files │ │ │ ... <= see the below table for enumeration │ │ └── tts_russian_addresses_rhvoice_4voices.tar.gz │ │ │ └── manifests │ ├── asr_calls_2_val.csv <= csv files with wav_path, text_path, duration (see notebooks) │ │ ... │ └── tts_russian_addresses_rhvoice_4voices.csv └── ru_open_stt_opus_unpacked <= a separate folder for each uploaded domain ├── public_youtube1120 │ ├── 0 <= see "On disk DB methodology" for details │ ├── 1 │ │ ├── 00 │ │ │ ... │ │ └── ff │ │ ├── *.opus <= actual files │ │ └── *.txt │ │ ... │ └── f ├── public_youtube1120_hq ├── public_youtube700_val ├── asr_calls_2_val ├── radio_2 ├── private_buriy_audiobooks_2 ├── asr_public_phone_calls_2 ├── asr_public_stories_2 ├── asr_public_stories_1 ├── public_lecture_1 ├── asr_public_phone_calls_1 ├── public_series_1 └── public_youtube700
Dataset GB, .wav GB, archivo Archivar Source Manifest
Train
Muestra de radio y voz pública - 11.4 .opus y .txt - .manifest
audiobook_2 162 25,8 .opus y .txt Internet y alineación .manifest
radio_2 154 24,6 .opus y .txt Radio .manifest
public_youtube1120 237 19,0 .opus y .txt Vídeos de YouTube .manifest
asr_public_phone_calls_2 66 9,4 .opus y .txt Internet y ASR .manifest
public_youtube1120_hq 31 4,9 .opus y .txt Vídeos de YouTube .manifest
asr_public_stories_2 9 1.4 .opus y .txt Internet y alineación .manifest
tts_russian_addresses_rhvoice_4voices 80,9 12.9 .opus y .txt TTS .manifest
public_youtube700 75,0 12,2 .opus y .txt Vídeos de YouTube .manifest
asr_public_phone_calls_1 22,7 3.2 .opus y .txt Internet y ASR .manifest
asr_public_stories_1 4,1 0,7 .opus y .txt Historias públicas .manifest
public_series_1 1,9 0,3 .opus y .txt Series públicas .manifest
public_lecture_1 0,7 0,1 .opus y .txt Internet y manual .manifest
Val
asr_calls_2_val 2 0,8 .wav y .txt Internet .manifest
buriy_audiobooks_2_val 1 0,5 .wav y .txt Libros y manual .manifest
public_youtube700_val 2 0,13 .wav y .txt Vídeos de YouTube y manual .manifest
Instrucciones de descarga

Directamente

Consulte: https://github.com/snakers4/open_stt#download-instructions

Mediante montaje en Azure Blob Storage

Consulte el cuaderno ubicado en la pestaña “Acceso a datos”.

Contactos

Para obtener ayuda o realizar preguntas sobre los datos, póngase en contacto con el autor de los datos en aveysov@gmail.com.

Licencia

Esta licencia permite a los usuarios distribuir, remezclar, adaptar y desarrollar el material en cualquier soporte o formato para fines no comerciales exclusivamente, siempre y cuando la atribución se asigne al creador. Incluye los siguientes elementos:
* BY: el mérito se debe atribuir al creador.
* NC: solo se permite utilizar el trabajo para fines no comerciales.

CC-BY-NC y uso comercial disponibles después de firmar un acuerdo con los creadores del conjunto de datos.

Referencias y otras lecturas

Conjunto de datos original

  • https://github.com/snakers4/open_stt

Artículos en inglés

  • https://thegradient.pub/towards-an-imagenet-moment-for-speech-to-text/
  • https://thegradient.pub/a-speech-to-text-practitioners-criticisms-of-industry-and-academia/

Artículos en chino

  • https://www.infoq.cn/article/4u58WcFCs0RdpoXev1E2

Artículos en ruso

  • https://habr.com/ru/post/494006/
  • https://habr.com/ru/post/474462/

Access

Available inWhen to use
Azure Notebooks

Quickly explore the dataset with Jupyter notebooks hosted on Azure or your local machine.

Select your preferred service:

Azure Notebooks

Azure Notebooks

Package: Language: Python

Helper functions / dependencies

Building libsndfile

The best efficient way to read opus files in python (the we know of) that does incur any significant overhead is to use pysoundfile (a python CFFI wrapper around libsoundfile).

When this solution was being researched the community had been waiting for a major libsoundfile release for some time.

Opus support has been implemented some time ago upstream, but it has not been properly released. Therefore we opted for a custom build + monkey patching.

At the time when you read / use this - probably there will be decent / proper builds of libsndfile.

Please replace with your faviourite tool if there is one.

Typically, you need to run this in your shell with sudo access:

apt-get update
apt-get install cmake autoconf autogen automake build-essential libasound2-dev \
libflac-dev libogg-dev libtool libvorbis-dev libopus-dev pkg-config -y

cd /usr/local/lib
git clone https://github.com/erikd/libsndfile.git
cd libsndfile
git reset --hard 49b7d61
mkdir -p build && cd build

cmake .. -DBUILD_SHARED_LIBS=ON
make && make install
cmake --build .

Helper functions / dependencies

Install the following libraries (versions do not matter much):

pandas
numpy
scipy
tqdm
soundfile
librosa

Depending on how this notebook is run, this sometimes can be as easy as (if, for example your miniconda is not installed under root):

In [ ]:
!pip install numpy
!pip install tqdm
!pip install scipy
!pip install pandas
!pip install soundfile
!pip install librosa
!pip install azure-storage-blob

Manifests are just csv files with the following columns:

  • Path to audio
  • Path to text file
  • Duration

They proved to be the most simple / helpful format of accessing data.

For ease of use all the manifests are already rerooted, i.e. all paths in them are relative and you just need to add a root folder.

In [1]:
# manifest utils
import os
import numpy as np
import pandas as pd
from tqdm import tqdm
from urllib.request import urlopen



def reroot_manifest(manifest_df,
                    source_path,
                    target_path):
    if source_path != '':
        manifest_df.wav_path = manifest_df.wav_path.apply(lambda x: x.replace(source_path,
                                                                              target_path))
        manifest_df.text_path = manifest_df.text_path.apply(lambda x: x.replace(source_path,
                                                                                target_path))
    else:
        manifest_df.wav_path = manifest_df.wav_path.apply(lambda x: os.path.join(target_path, x))
        manifest_df.text_path = manifest_df.text_path.apply(lambda x: os.path.join(target_path, x))    
    return manifest_df


def save_manifest(manifest_df,
                  path,
                  domain=False):
    if domain:
        assert list(manifest_df.columns) == ['wav_path', 'text_path', 'duration', 'domain']
    else:
        assert list(manifest_df.columns) == ['wav_path', 'text_path', 'duration']

    manifest_df.reset_index(drop=True).sort_values(by='duration',
                                                   ascending=True).to_csv(path,
                                                                          sep=',',
                                                                          header=False,
                                                                          index=False)
    return True


def read_manifest(manifest_path,
                  domain=False):
    if domain:
        return pd.read_csv(manifest_path,
                        names=['wav_path',
                               'text_path',
                               'duration',
                               'domain'])
    else:
        return pd.read_csv(manifest_path,
                        names=['wav_path',
                               'text_path',
                               'duration'])


def check_files(manifest_df,
                domain=False):
    orig_len = len(manifest_df)
    if domain:
        assert list(manifest_df.columns) == ['wav_path', 'text_path', 'duration']
    else:
        assert list(manifest_df.columns) == ['wav_path', 'text_path', 'duration', 'domain']
    wav_paths = list(manifest_df.wav_path.values)
    text_path = list(manifest_df.text_path.values)

    omitted_wavs = []
    omitted_txts = []

    for wav_path, text_path in zip(wav_paths, text_path):
        if not os.path.exists(wav_path):
            print('Dropping {}'.format(wav_path))
            omitted_wavs.append(wav_path)
        if not os.path.exists(text_path):
            print('Dropping {}'.format(text_path))
            omitted_txts.append(text_path)

    manifest_df = manifest_df[~manifest_df.wav_path.isin(omitted_wavs)]
    manifest_df = manifest_df[~manifest_df.text_path.isin(omitted_txts)]
    final_len = len(manifest_df)

    if final_len != orig_len:
        print('Removed {} lines'.format(orig_len-final_len))
    return manifest_df


def plain_merge_manifests(manifest_paths,
                          MIN_DURATION=0.1,
                          MAX_DURATION=100):

    manifest_df = pd.concat([read_manifest(_)
                             for _ in manifest_paths])
    manifest_df = check_files(manifest_df)

    manifest_df_fit = manifest_df[(manifest_df.duration>=MIN_DURATION) &
                                  (manifest_df.duration<=MAX_DURATION)]

    manifest_df_non_fit = manifest_df[(manifest_df.duration<MIN_DURATION) |
                                      (manifest_df.duration>MAX_DURATION)]

    print(f'Good hours: {manifest_df_fit.duration.sum() / 3600:.2f}')
    print(f'Bad hours: {manifest_df_non_fit.duration.sum() / 3600:.2f}')

    return manifest_df_fit


def save_txt_file(wav_path, text):
    txt_path = wav_path.replace('.wav','.txt')
    with open(txt_path, "w") as text_file:
        print(text, file=text_file)
    return txt_path


def read_txt_file(text_path):
    #with open(text_path, 'r') as file:
    response = urlopen(text_path)
    file = response.readlines()
    for i in range(len(file)):
        file[i] = file[i].decode('utf8')
    return file 

def create_manifest_from_df(df, domain=False):
    if domain:
        columns = ['wav_path', 'text_path', 'duration', 'domain']
    else:
        columns = ['wav_path', 'text_path', 'duration']
    manifest = df[columns]
    return manifest


def create_txt_files(manifest_df):
    assert 'text' in manifest_df.columns
    assert 'wav_path' in manifest_df.columns
    wav_paths, texts = list(manifest_df['wav_path'].values), list(manifest_df['text'].values)
    # not using multiprocessing for simplicity
    txt_paths = [save_txt_file(*_) for _ in tqdm(zip(wav_paths, texts), total=len(wav_paths))]
    manifest_df['text_path'] = txt_paths
    return manifest_df


def replace_encoded(text):
    text = text.lower()
    if '2' in text:
        text = list(text)
        _text = []
        for i,char in enumerate(text):
            if char=='2':
                try:
                    _text.extend([_text[-1]])
                except:
                    print(''.join(text))
            else:
                _text.extend([char])
        text = ''.join(_text)
    return text
In [2]:
# reading opus files
import os
import soundfile as sf



# Fx for soundfile read/write functions
def fx_seek(self, frames, whence=os.SEEK_SET):
    self._check_if_closed()
    position = sf._snd.sf_seek(self._file, frames, whence)
    return position


def fx_get_format_from_filename(file, mode):
    format = ''
    file = getattr(file, 'name', file)
    try:
        format = os.path.splitext(file)[-1][1:]
        format = format.decode('utf-8', 'replace')
    except Exception:
        pass
    if format == 'opus':
        return 'OGG'
    if format.upper() not in sf._formats and 'r' not in mode:
        raise TypeError("No format specified and unable to get format from "
                        "file extension: {0!r}".format(file))
    return format


#sf._snd = sf._ffi.dlopen('/usr/local/lib/libsndfile/build/libsndfile.so.1.0.29')
sf._subtypes['OPUS'] = 0x0064
sf.SoundFile.seek = fx_seek
sf._get_format_from_filename = fx_get_format_from_filename


def read(file, **kwargs):
    return sf.read(file, **kwargs)


def write(file, data, samplerate, **kwargs):
    return sf.write(file, data, samplerate, **kwargs)
In [3]:
# display utils
import gc
from IPython.display import HTML, Audio, display_html
pd.set_option('display.max_colwidth', 3000)
#Prepend_path is set to read directly from Azure. To read from local replace below string with path to the downloaded dataset files
prepend_path = 'https://azureopendatastorage.blob.core.windows.net/openstt/ru_open_stt_opus_unpacked/'


def audio_player(audio_path):
    return '<audio preload="none" controls="controls"><source src="{}" type="audio/wav"></audio>'.format(audio_path)

def display_manifest(manifest_df):
    display_df = manifest_df
    display_df['wav'] = [audio_player(prepend_path+path) for path in display_df.wav_path]
    display_df['txt'] = [read_txt_file(prepend_path+path) for path in tqdm(display_df.text_path)]
    audio_style = '<style>audio {height:44px;border:0;padding:0 20px 0px;margin:-10px -20px -20px;}</style>'
    display_df = display_df[['wav','txt', 'duration']]
    display(HTML(audio_style + display_df.to_html(escape=False)))
    del display_df
    gc.collect()

Play with a dataset

Play a sample of files

On most platforms browsers usually support native audio playback.

So we can leverage HTML5 audio players to view our data.

In [4]:
manifest_df = read_manifest(prepend_path +'/manifests/public_series_1.csv')
#manifest_df = reroot_manifest(manifest_df,
                              #source_path='',
                              #target_path='../../../../../nvme/stt/data/ru_open_stt/')
In [5]:
sample = manifest_df.sample(n=20)
display_manifest(sample)
100%|██████████| 20/20 [00:07<00:00,  2.66it/s]
wav txt duration
5963 [пожалуйста прости всё в порядке\n] 2.48
19972 [хотелось бы хотя бы разок глазком на неё посмотреть раз такое дело\n] 5.68
15555 [они с егерем на след напали до инспектора не дозвониться\n] 3.84
430 [что то случилось\n] 1.36
4090 [так давай опаздываем\n] 2.16
18590 [да саид слушаю тебя троих нашли а в полётном листе\n] 4.60
17734 [надо сначала самому серьёзным человеком стать понимаешь\n] 4.32
978 [вот что случилось\n] 1.56
13269 [да паш юль пожалуйста не делай глупостей\n] 3.48
4957 [полусладкое или сухое\n] 2.32
1913 [ищи другую машину\n] 1.80
10522 [гражданин финн не зная что я полицейский\n] 3.08
9214 [ты чего трубку не берёшь я же переживаю\n] 2.88
10014 [я не окажу сопротивления я без оружия\n] 3.00
8351 [звони партнёру пусть он напишет\n] 2.80
3818 [ну что пойдём обсудим\n] 2.12
11097 [вы простите понимаете все об этом знают\n] 3.16
2989 [какие уж разводки\n] 2.00
12229 [я получается какой то диспетчер а не напарник\n] 3.28
5348 [я же тебе сказала никакой карелии\n] 2.40

Read a file

In [ ]:
!ls ru_open_stt_opus/manifests/*.csv

A couple of simplistic examples showing how to best read wav and opus files.

Scipy is the fastest for wav, pysoundfile is the best overall for opus.

In [6]:
%matplotlib inline

import librosa
from scipy.io import wavfile
from librosa import display as ldisplay
from matplotlib import pyplot as plt

Read a wav

In [7]:
manifest_df = read_manifest(prepend_path +'manifests/asr_calls_2_val.csv')
#manifest_df = reroot_manifest(manifest_df,
                              #source_path='',
                              #target_path='../../../../../nvme/stt/data/ru_open_stt/')
In [8]:
sample = manifest_df.sample(n=5)
display_manifest(sample)
100%|██████████| 5/5 [00:01<00:00,  2.61it/s]
wav txt duration
7802 [это же позитивные новости не негативные\n] 2.01
3590 [белый цветочек\n] 1.17
10594 [какое отношение имеет ваша пенсия к моему отделению\n] 3.14
4630 [есть есть видео\n] 1.35
468 [что ещё раз\n] 0.62
In [9]:
from io import BytesIO

wav_path = sample.iloc[0].wav_path
response = urlopen(prepend_path+wav_path)
data = response.read()
sr, wav = wavfile.read(BytesIO(data))
wav.astype('float32')
absmax = np.max(np.abs(wav))
wav =  wav / absmax
In [10]:
# shortest way to plot a spectrogram
D = librosa.amplitude_to_db(np.abs(librosa.stft(wav)), ref=np.max)
plt.figure(figsize=(12, 6))
ldisplay.specshow(D, y_axis='log')
plt.colorbar(format='%+2.0f dB')
plt.title('Log-frequency power spectrogram')
# shortest way to plot an envelope
plt.figure(figsize=(12, 6))
ldisplay.waveplot(wav, sr=sr, max_points=50000.0, x_axis='time', offset=0.0, max_sr=1000, ax=None)
Out[10]:
<matplotlib.collections.PolyCollection at 0x7fdf62f7e8d0>

Read opus

In [11]:
manifest_df = read_manifest(prepend_path +'manifests/asr_public_phone_calls_2.csv')
#manifest_df = reroot_manifest(manifest_df,
                              #source_path='',
                              #target_path='../../../../../nvme/stt/data/ru_open_stt/')
In [12]:
sample = manifest_df.sample(n=5)
display_manifest(sample)
100%|██████████| 5/5 [00:02<00:00,  2.24it/s]
wav txt duration
5018 [а вы кто\n] 0.96
143473 [пьеса дружбы нету\n] 1.86
272155 [не знаю где находится\n] 2.64
334225 [ты куда звонишь то куда ты звонишь ты знаешь\n] 3.12
143789 [помощник дежурного\n] 1.86
In [13]:
opus_path = sample.iloc[0].wav_path
response = urlopen(prepend_path+opus_path)
data = response.read()
wav, sr = sf.read(BytesIO(data))
wav.astype('float32')
absmax = np.max(np.abs(wav))
wav =  wav / absmax
In [14]:
# shortest way to plot a spectrogram
D = librosa.amplitude_to_db(np.abs(librosa.stft(wav)), ref=np.max)
plt.figure(figsize=(12, 6))
ldisplay.specshow(D, y_axis='log')
plt.colorbar(format='%+2.0f dB')
plt.title('Log-frequency power spectrogram')
# shortest way to plot an envelope
plt.figure(figsize=(12, 6))
ldisplay.waveplot(wav, sr=sr, max_points=50000.0, x_axis='time', offset=0.0, max_sr=1000, ax=None)
Out[14]:
<matplotlib.collections.PolyCollection at 0x7fdf62f8ee10>
In [ ]: