GDC95▮

Published · Updated

IESTACThe paper, explained

An English-Italian speech-to-text translation corpus built from free audiobooks: 131 hours of LibriVox speech, aligned sentence by sentence with the Italian edition of each book. The problem, the three-step pipeline, every tool we compared, the numbers, and how to cite it. Written by the first author.

Read the paper (PDF) Cite Code & corpus
IESTAC, the paper explained Contents 8 min left

01 // The problem

ProblemST

What is end-to-end speech-to-text translation, and why did it need a new corpus?

Say something in English and get it written down in Italian. For decades that meant three machines passing a baton. The classic way to translate speech was a cascade: an automatic speech recogniser turned the audio into English text, a machine translation system turned that text into Italian, and if you wanted it spoken again, a text-to-speech engine read it out. Three technologies, three handovers. Every handover added delay, and every mistake made upstream was inherited downstream. If the recogniser heard "I scream" instead of "ice cream", the translator faithfully translated the wrong thing.

End-to-end speech-to-text translation skips the relay. One model listens to the English audio and writes the Italian text directly: no intermediate transcript, no error propagation, lower latency. The catch is what such a model needs to learn from: thousands of hours of English audio paired with its Italian translation and, since pre-training on speech recognition helps, also with its English transcript. A triplet: audio, transcript, translation.

Fig. 1 · Cascade versus end-to-endwhy triplets

Cascaded speech translation chains three systems and propagates errors; end-to-end translation maps audio to Italian text directly. A · Cascade · three hand-overs English audiowaveform ASRspeech → EN text MTEN text → IT text TTSIT text → speech "ice cream" heard as "I scream" error propagates B · End-to-end · one model English audiowaveform Direct ST modellistens in English, writes Italian Italian textno relay training data needed: ( audio , EN transcript , IT translation ) × many thousands
The paper's job was to produce the triplets approach B needs, for a language pair that had none.

Speech translation corpora, 2020what existed

#CorpusSourcePairLicence
01MuST-CFBKTED talksEN → 8 languagesno commercial useno derivatives
02Europarl-STUPVEuropean Parliament6 languagesno commercial use
03Augmented LibriSpeechLIMSILibriVox + GutenbergEN → FRpermissive
04IESTACUppsala · this paperLibriVox + Gutenberg + Liber LiberEN → ITpublic-domain sourcesfreely released

Pairs and licences as reported in the paper's related-work section. For English-Italian with a permissive licence there was nothing before IESTAC.

That was the gap I had set out to fill in my Master's thesis in Language Technology at Uppsala University. After the thesis, my supervisor Sara Stymne and I turned the work into a paper, IESTAC: English-Italian Parallel Corpus for End-to-End Speech-to-Text Machine Translation, presented in November 2020 at the First Workshop on NLP Beyond Text at EMNLP. The paper is as much about how to fill such a gap for any language pair as about the corpus itself.

02 // The pipeline

Pipeline3 steps

How do you build a speech translation corpus from free audiobooks?

The raw material was already on the web, for free. LibriVox hosts volunteers reading public-domain books aloud. Project Gutenberg hosts the same books as text. What was missing was the Italian side, and the glue between all three. We followed Augmented LibriSpeech in three steps, and at each step we tested several tools and said which to pick.

  1. 01

    Text collection

    For each English audiobook with its e-book, find the matching Italian e-book, if one exists in the public domain.

    Compared: WikiData · WikiMedia · Google Translate

  2. 02

    Sentence alignment

    Pair each English sentence with the Italian sentence or sentences that translate it, chapter by chapter.

    Compared: hunalign ×2 · Vecalign

  3. 03

    Forced alignment

    Cut the audio exactly where each English sentence starts and ends, so every sentence owns its clip.

    Compared: Gentle · Aeneas

  4. →

    Release

    60,561 triplets, one folder per book and per chapter, the alignment carried in the file names, plus a SQL database.

    GitHub · Kaggle

03 // Text collection

Books39 pairs

How do you find the Italian translation of an English book, automatically?

We started from the LibriSpeech metadata: a list of about 1,500 Gutenberg titles that had audiobooks. Titles were messy. "Tom Swift and His Sky Racer, or, the Quickest Flight on Record" is one book with two names, so regular expressions first split such titles into every plausible variant. Then came the real question: what is this book called in Italian? We tried three answers.

  1. A

    Ask WikiDataSPARQL

    Fetch every entity of class literary work with both an English and an Italian label, keep the ones whose English label matches our list. Clean output: real published titles, with their real Italian editions.

    ?work wdt:P31/wdt:P279* wd:Q7725634 ; rdfs:label ?en, ?it .
  2. B

    Ask WikiMediaREST

    Query each English title against the WikiMedia endpoint and take the title of the corresponding Italian page. Fast, but noisy: it returned films, plays and categories that had nothing to do with books.

  3. C

    Ask Google TranslateMT

    Machine-translate the English titles into Italian and hope the publisher chose the literal translation. The noisiest of all: the output had no link to any actual publication.

  4. ?

    Score them indirectly

    No ground truth exists for which books truly match. So we scraped an index of about 3,400 freely available Italian e-books and counted which candidate titles from each method actually hit it.

Figure 1 of the paper · Italian e-books found, by methodhits on an index of ~3,400 titles

#MethodFoundShare
01WikiDataSPARQL · literary work2540.3%
02WikiMediapage title lookup2133.9%
03Google Translateliteral title1625.8%

Found by one method onlyset intersections

#SubsetBooksShare
01WikiData only812.9%
02Google Translate only58.1%
03WikiMedia only46.5%
—All three agree812.9%

Redrawn from the paper's UpSet plot. Shares are over all books found. Each method found at least four books nobody else did, so the three are complementary.

Together the three methods yielded 39 pairs of English and Italian e-books. The modest count was mostly the index's fault: 3,400 Italian titles, against the 14,845 French titles Augmented LibriSpeech could draw on. Then the housekeeping: we split each book into chapters with regular expressions, in English and in Italian, checked the chapter pairs by hand, because a missed heading merges two chapters into one, and segmented sentences with spaCy. For the first release we picked nine books by eight authors, 373 chapter pairs, about 130 hours of audio: enough to compare alignment methods properly, instead of aligning all 39 pairs, some 500 hours, with tools not yet proven.

04 // Bilingual sentence alignment

Sentences70,204

Which sentence aligner works best: hunalign or Vecalign?

Two translations of the same chapter rarely march in step. One language says in two sentences what the other says in one; a translator drops a line, adds a flourish, splits a paragraph. Sentence alignment is the craft of pairing them anyway. We pitted a classic against a newcomer, three ways.

  1. 1

    hunalign, hand-made dictionaryGale-Church

    Sentences that translate each other have similar lengths, and a bilingual dictionary helps confirm it. The dictionary was the LFAligner English-Italian lexicon, about 14,500 manually checked terms. Two passes: align, grow the dictionary from what you aligned, align again.

  2. 2

    hunalign, inferred dictionaryGale-Church

    Same aligner, a much bigger lexicon. We built a 3.1-million-sentence English-Italian corpus from Europarl, Wikipedia, GlobalVoices and OPUS Books, ran Giza++ and Moses to extract translation probabilities, and kept every pair scoring above 0.10: 692,437 terms, roughly 48× the hand-made one.

  3. 3

    Vecalign, LASER embeddingschosen

    No lengths, no dictionaries. Every sentence, and every run of up to ten consecutive sentences, is mapped into a multilingual vector space with Facebook's LASER. Sentences that mean the same thing land close together, so alignment becomes a cosine-similarity search, and one-to-many pairings fall out naturally.

  4. ?

    Judge the hard cases

    No gold standard, and random sampling is useless because most alignments are easy. So we zoomed in on the 2,030 cases where the three tools disagreed, out of 70,204 candidate alignments, and judged 200 of them by hand: correct, wrong or partial.

Table 1 of the paper · 200 difficult alignments, judged by handn = 200

#AlignerCorrectWrongPartialshare of 200
01VecalignLASER · up to 10 consecutive sentences16984.5%52.5%2613.0%
02hunalign + inferred692k terms4321.5%8944.5%6834.0%
03hunalignhand-made dictionary · 14.5k terms4221.0%10854.0%5025.0%

correctwrongpartial

A dictionary 48× bigger fixed 19 errors but added one correct alignment. Agreement between the two hunalign runs: 93.6%. Between hunalign and Vecalign: 79.2%.

Compare · pick an alignersame sentence, three verdicts

169correct
5wrong
26partial

Embeds runs of up to ten sentences. The two English sentences together land next to the single Italian one in LASER space, so the one-to-many pairing is found directly. It also returns zero-to-one alignments, which quietly drop passages the two editions do not share.

English · two sentences

I see it now.

A long room with three long rows of desks.

Italian · one sentence

Mi sembra di rivederla: una lunga sala, con tre lunghe file di piccoli scrittoi.

Both English sentences paired with the Italian one. A real alignment from the corpus.

05 // Forced alignment

Audio373 chapters

What is forced alignment, and why did we pick Aeneas over Gentle?

Now each English sentence had its Italian partner. The last step was to find, inside 373 chapter-long recordings, the exact second where each English sentence began and ended. That is forced alignment, and again there were two schools.

  1. 1

    GentleKaldi · HMM

    A pre-trained Kaldi speech recogniser, a hidden Markov model that finds where each phoneme and word sits in the audio. It returns word timings; sentence timings need a post-processing step. Its strength: it copes with large stretches of text or audio that have no counterpart.

  2. 2

    Aeneaschosen · TTS + DTW

    Read the transcript aloud with a text-to-speech engine, then use dynamic time warping to stretch and squeeze the synthetic audio until it lines up with the human recording. It gives sentence timings directly. Its weakness: the audio must match the text closely. Ours did, once we had cut the LibriVox disclaimers.

Fig. 2 · Dynamic time warpingsynthetic speech across · human reading down

Dynamic time warping: a path through a grid matches moments of synthetic speech on one axis to moments of the human recording on the other. TTS reading of the sentence → LibriVox recording →
The warp path, in orange, is the cheapest way through the grid. Wherever it bends, the two audios disagree on pace, and sentence boundaries are read off its corners.

Table 2 of the paper · 210 clips checked by earAeneas · 210 clips

Correct 204Every word of the sentence is there and whole. 97.1% of the clips.
Mild error 2A letter clipped at the start or the end of the clip.
Severe 4Clip and text off-sync or unrelated, such as a reader voicing a footnote the text does not contain. None propagated to its neighbours.

06 // Corpus statistics and structure

CorpusTable 3

How big is IESTAC, and how is it organised?

60,561triplets · audio, EN text, IT text
131.23hours of English speech
98speakers · 3.8 chapters each
373chapters · 9 books · 8 authors
7.80seconds · average clip
1,577,118EN tokens · 1,425,072 IT

Unique tokens · same stories, two vocabulariesTable 3

#SideTokensUniqueunique, relative
ITItaliantranslations · Liber Liber1,425,07263,774
ENEnglishtranscripts · Gutenberg1,577,11838,325

Italian has two thirds more distinct word forms than English for the same content: verb conjugation and adjective agreement. One reason English-to-Italian is harder than its token count suggests.

We released the corpus as a zipped folder: two parallel raw text files and nine folders, one per book, named by Gutenberg ID. Inside, one folder per chapter; inside that, the triplets. The alignment lives in the file names.

83_07_33.wavEnglish audio · 7.8 s on average
83_07_33.enEnglish source sentence
83_07_33.itItalian translation

83 Gutenberg e-book ID · 07 chapter · 33 segment · a SQL database is included for querying

What we left for later: an extrinsic test, training actual speech recognition and translation models on the corpus; automatic filtering of weaker alignments using alignment scores, character error rate or embedding similarity; and a full taxonomy of how to collect speech-to-text datasets for translation. What we delivered was a recipe anyone can follow for another language pair, with the tool choices already made and justified: WikiData first for finding books, Vecalign for sentences, Aeneas for audio.

07 // Questions

FAQs7

The short answers, for people who skipped to the end.

What is IESTAC?

IESTAC, the Italian-English Speech and Text Audiobooks Corpus, is a freely available corpus of 60,561 triplets of English audio, English transcript and Italian translation, aligned at sentence level: 131 hours of LibriVox audiobooks from 9 public-domain books. Giuseppe Della Corte and Sara Stymne, Uppsala University, built it and published the method at EMNLP 2020.

What is end-to-end speech-to-text translation?

A single model that listens to audio in one language and writes text in another, instead of chaining speech recognition, machine translation and text-to-speech. It avoids error propagation and reduces latency, but needs training data made of audio, transcript and translation triplets.

How do you find the Italian translation of an English book automatically?

The paper compared three methods: a SPARQL query on WikiData for literary works with English and Italian labels, querying the WikiMedia endpoint for the Italian page title, and machine-translating the titles with Google Translate. WikiData was the most precise and found the most unique titles, but the three were complementary; together they found 39 English-Italian book pairs.

Which sentence aligner works best, hunalign or Vecalign?

On 200 difficult alignments judged by hand, Vecalign with LASER multilingual sentence embeddings got 169 correct and 5 wrong, while hunalign, Gale-Church with a bilingual dictionary, got about 42 correct and 89 to 108 wrong, even with a 692,437-term inferred dictionary. Vecalign handles one-to-many alignments far better.

What is forced alignment, and why Aeneas instead of Gentle?

Forced alignment finds the time interval in an audio file for each sentence of its transcript. Gentle uses a Kaldi speech recogniser and returns word timings; Aeneas synthesises the text with text-to-speech and uses dynamic time warping, returning sentence timings directly. On 210 manually checked clips Aeneas produced 204 correct alignments, 2 mild errors and 4 severe errors, with no error propagation.

How big is the IESTAC corpus?

60,561 aligned segments, 131.23 hours of English speech, 98 speakers, 373 chapters from 9 books, average clip 7.80 seconds, 1,577,118 English tokens and 1,425,072 Italian tokens. Files are named EbookID_Chapter_Segment with .wav, .en and .it extensions, and a SQL database is included.

How do I cite IESTAC?

Della Corte, G., & Stymne, S. (2020). IESTAC: English-Italian Parallel Corpus for End-to-End Speech-to-Text Machine Translation. In Proceedings of the First International Workshop on Natural Language Processing Beyond Text (pp. 41–50). Association for Computational Linguistics. APA and BibTeX below.

08 // Cite & download

CiteDOI

Della Corte & Stymne, 2020. EMNLP, First Workshop on NLP Beyond Text, pages 41–50.

APAand BibTeX

Della Corte, G., & Stymne, S. (2020). IESTAC: English-Italian Parallel Corpus for End-to-End Speech-to-Text Machine Translation. In Proceedings of the First International Workshop on Natural Language Processing Beyond Text (pp. 41–50). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.nlpbt-1.5

.bib file 
@inproceedings{della-corte-stymne-2020-iestac,
    title     = "{IESTAC}: {E}nglish-{I}talian Parallel Corpus for End-to-End Speech-to-Text Machine Translation",
    author    = "Della Corte, Giuseppe and Stymne, Sara",
    booktitle = "Proceedings of the First International Workshop on Natural Language Processing Beyond Text",
    month     = nov,
    year      = "2020",
    address   = "Online",
    publisher = "Association for Computational Linguistics",
    url       = "https://aclanthology.org/2020.nlpbt-1.5/",
    doi       = "10.18653/v1/2020.nlpbt-1.5",
    pages     = "41--50"
}

Paper 01 in Tech & Data Stuff · Presented online at the First International Workshop on NLP Beyond Text, EMNLP 2020, pages 41–50. I wrote this companion as the first author; every table and figure is redrawn from the paper. Tap the tab on the left to keep your place on this device.

Read Data story 01 All publications & stories

Giuseppe Della Corte wrote this paper in 2020 with Professor Sara Stymne, who had supervised his Master's thesis in Language Technology at Uppsala University. He then turned to building products. The rest of the journey →