News

Improving Supervised Massively Multilingual NMT
20 Oct 2020

In our lab meeting tomorrow, Nishant will introduce his work on Multilingual NMT.

A Zoom link will be posted to Twist on the morning of the meeting.

Improving Supervised Massively Multilingual NMT

Abstract: The dominant approach in multilingual neural machine translation (NMT) uses a single model with parameters shared across multiple languages. We’ll look at a novel approach that simultaneously trains two NMT models: a forward model trained on parallel data from multiple languages to a single language and a backward model trained to translate from the same single language into multiple languages. It is purely supervised using no monolingual data but exploits the available parallel training data by learning a shared multilingual representation space.

Tuesday, October 20th, 09:30 a.m.

Effectively pretraining a speech translation decoder with Machine Translation data
29 Sep 2020

In our lab meeting tomorrow, Ashkan will introduce his work on speech translation.

A Zoom link will be posted to Twist on the morning of the meeting.

Effectively pretraining a speech translation decoder with Machine Translation data

Abstract: Directly translating from speech to text using an End-to-End approach is still challenging for many language pairs, due to lack of sufficient data. Although pretraining the encoder parameters using Automatic Speech Recognition (ASR) task improves the results in low resource settings, attempting to use pretrained parameters from Neural Machine Translation (NMT) task was mainly unsuccessful in previous works. In this paper, we will show that by using an adversarial regularizer we can bring closer the encoder representations of the ASR and NMT tasks even though they are in different modalities, and how this helps us to effectively use a pretrained NMT decoder for speech translation.

Tuesday, September 29nd, 09:30 a.m.

NLP Tools for Sumerian
21 Sep 2020

In our lab meeting tomorrow, Logan will discuss his summer internship at the CDLI (Cuneiform Digital Library Initiative).

A Zoom link will be posted to Twist on the morning of the meeting.

NLP Tools for Sumerian

Abstract: He will give an overview of the state of NLP tools for ancient near eastern languages, with a focus on Sumerian. He will introduce a variety of monolingual and parallel corpora with varying degrees of annotation. He will highlight the shortcomings of these datasets, which can be incomplete and sometimes require extensive cleaning in order to be useful. He also plan to survey existing tools for NLP tasks such as translation and POS tagging, and to highlight areas where these tools are missing or inadequate.

He will demonstrate some of these issues using case studies from his own work, which extracted information about counted objects in Sumerian accounting tablets. Lastly, he will survey another project completed at the CDLI this summer, which worked to improve Sumerian-English machine translation.

Tuesday, September 22nd, 09:30 a.m.

Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
04 Aug 2020

In our lab meeting tomorrow, Vincent will introduce a paper: Don’t Stop Pretraining on ACL2020 (Honorable mention for best paper).

A Zoom link will be posted to Twist on the morning of the meeting.

Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks

Abstract: Language models pretrained on text from a wide variety of sources form the foundation of today’s NLP. In light of the success of these broad-coverage models, we investigate whether it is still helpful to tailor a pretrained model to the domain of a target task. We present a study across four domains (biomedical and computer science publications, news, and reviews) and eight classification tasks, showing that a second phase of pretraining indomain (domain-adaptive pretraining) leads to performance gains, under both high- and low-resource settings. Moreover, adapting to the task’s unlabeled data (task-adaptive pretraining) improves performance even after domain-adaptive pretraining. Finally, we show that adapting to a task corpus augmented using simple data selection strategies is an effective alternative, especially when resources for domain-adaptive pretraining might be unavailable. Overall, we consistently find that multiphase adaptive pretraining offers large gains in task performance.

https://www.aclweb.org/anthology/2020.acl-main.740.pdf

Tuesday, Aug 4th, 09:30 a.m.

Neural Module Networks
28 Jul 2020

In our lab meeting tomorrow, Anoop will cover two papers on neural module networks.

A Zoom link will be posted to Twist on the morning of the meeting.

Neural Module Networks

Abstract: Anoop will cover some papers on neural module networks which have previously been applied to VQA and to reading comprehension tasks. However, in my estimation their potential applicability will be quite widespread in NLP even though it was introduced quite some time ago (in 2015) and we haven’t seen it take off yet. We will cover the following papers:

Neural Module Networks. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein. https://arxiv.org/abs/1511.02799.

Neural Module Networks for Reasoning over Text. Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, Matt Gardner. https://arxiv.org/abs/1912.04971

Tuesday, July 28th, 09:30 a.m.

Recent Publications