Learning Document Embeddings Along With Their Uncertainties

Learning Document Embeddings Along With Their Uncertainties

WoS Article

Majority of the text modeling techniques yield onlypoint-estimates of document embeddings and lack in capturingthe uncertainty of the estimates. These uncertainties give a notionof how well the embeddings represent a document. We presentBayesian subspace multinomial model (Bayesian SMM), a generativelog-linear model that learns to represent documents in theform of Gaussian distributions, thereby encoding the uncertaintyin its covariance. Additionally, in the proposed Bayesian SMM,we address a commonly encountered problem of intractabilitythat appears during variational inference in mixed-logit models.We also present a generative Gaussian linear classifier for topicidentification that exploits the uncertainty in document embeddings.Our intrinsic evaluation using perplexity measure showsthat the proposed Bayesian SMM fits the unseen test data betteras compared to the state-of-the-art neural variational documentmodel on (Fisher) speech and (20Newsgroups) text corpora. Ourtopic identification experiments showthat the proposed systems arerobust to over-fitting on unseen test data. The topic ID results showthat the proposedmodel outperforms state-of-the-art unsupervisedtopic models and achieve comparable results to the state-of-the-artfully supervised discriminative models.

Majority of the text modeling techniques yield onlypoint-estimates of document embeddings and lack in capturingthe uncertainty of the estimates. These uncertainties give a notionof how well the embeddings represent a document. We presentBayesian subspace multinomial model (Bayesian SMM), a generativelog-linear model that learns to represent documents in theform of Gaussian distributions, thereby encoding the uncertaintyin its covariance. Additionally, in the proposed Bayesian SMM,we address a commonly encountered problem of intractabilitythat appears during variational inference in mixed-logit models.We also present a generative Gaussian linear classifier for topicidentification that exploits the uncertainty in document embeddings.Our intrinsic evaluation using perplexity measure showsthat the proposed Bayesian SMM fits the unseen test data betteras compared to the state-of-the-art neural variational documentmodel on (Fisher) speech and (20Newsgroups) text corpora. Ourtopic identification experiments showthat the proposed systems arerobust to over-fitting on unseen test data. The topic ID results showthat the proposedmodel outperforms state-of-the-art unsupervisedtopic models and achieve comparable results to the state-of-the-artfully supervised discriminative models.

Bayesian methods, embeddings, topic identification.

Bayesian methods, embeddings, topic identification.

KESIRAJU, S.; PLCHOT, O.; BURGET, L.; GANGASHETTY, S.

2021

27.07.2020

2329-9290

IEEE-ACM Transactions on Audio Speech and Language Processing

2020

28

United States of America

2319

2332

14

https://ieeexplore.ieee.org/document/9149686

kesiraju_acm_transactions on ASLP_28_2020_09149686