Detail publikačního výsledku

i-vectors in language modeling: An efficient way of domain adaptation for feed-forward models

BENEŠ, K.; KESIRAJU, S.; BURGET, L.

Originální název

i-vectors in language modeling: An efficient way of domain adaptation for feed-forward models

Anglický název

i-vectors in language modeling: An efficient way of domain adaptation for feed-forward models

Druh

Stať ve sborníku v databázi WoS či Scopus

Originální abstrakt

We show an effective way of adding context information toshallow neural language models. We propose to use SubspaceMultinomial Model (SMM) for context modeling and we addthe extracted i-vectors in a computationally efficient way. Byadding this information, we shrink the gap between shallowfeed-forward network and an LSTM from 65 to 31 points of perplexityon the Wikitext-2 corpus (in the case of neural 5-grammodel). Furthermore, we show that SMM i-vectors are suitablefor domain adaptation and a very small amount of adaptationdata (e.g. endmost 5% of a Wikipedia article) brings asubstantial improvement. Our proposed changes are compatiblewith most optimization techniques used for shallow feedforwardLMs.

Anglický abstrakt

We show an effective way of adding context information toshallow neural language models. We propose to use SubspaceMultinomial Model (SMM) for context modeling and we addthe extracted i-vectors in a computationally efficient way. Byadding this information, we shrink the gap between shallowfeed-forward network and an LSTM from 65 to 31 points of perplexityon the Wikitext-2 corpus (in the case of neural 5-grammodel). Furthermore, we show that SMM i-vectors are suitablefor domain adaptation and a very small amount of adaptationdata (e.g. endmost 5% of a Wikipedia article) brings asubstantial improvement. Our proposed changes are compatiblewith most optimization techniques used for shallow feedforwardLMs.

Klíčová slova

language modeling, feed-forward models, subspacemultinomial model, domain adaptation

Klíčová slova v angličtině

language modeling, feed-forward models, subspacemultinomial model, domain adaptation

Autoři

BENEŠ, K.; KESIRAJU, S.; BURGET, L.

Rok RIV

2019

Vydáno

02.09.2018

Nakladatel

International Speech Communication Association

Místo

Hyderabad

Kniha

Proceedings of Interspeech 2018

ISSN

1990-9772

Periodikum

Proceedings of Interspeech

Svazek

2018

Číslo

9

Stát

Francouzská republika

Strany od

3383

Strany do

3387

Strany počet

5

URL

https://www.isca-speech.org/archive/Interspeech_2018/abstracts/1070.html

BibTex

@inproceedings{BUT155102,
  author="Karel {Beneš} and Santosh {Kesiraju} and Lukáš {Burget}",
  title="i-vectors in language modeling: An efficient way of domain adaptation for feed-forward models",
  booktitle="Proceedings of Interspeech 2018",
  year="2018",
  journal="Proceedings of Interspeech",
  volume="2018",
  number="9",
  pages="3383--3387",
  publisher="International Speech Communication Association",
  address="Hyderabad",
  doi="10.21437/Interspeech.2018-1070",
  issn="1990-9772",
  url="https://www.isca-speech.org/archive/Interspeech_2018/abstracts/1070.html"
}

Dokumenty

benes_interspeech2018_1070