Controllable Natural Language Generation

In this thesis, various methods are explored to control the textual content of autoregressive neural language generators as used in several Natural Language Generation tasks, including Novel Object Captioning, Commonsense Generation, Story Generation, Machine Translation and Data Augmentation, in entity and sentence levels. The aim is to develop neural language generators that produce fluent natural language output while meeting various task-specific requirements. The significant contributions of this thesis are the following:

• Reinforcement learning is used to encourage novel object captioning systems to mention salient visual objects in the input images. (Published in EACL 2021)

• An approach called Mention Flags is developed, which generalises the Transformer Attention mechanism in a way that permits to control the entities mention. (Published in ACL 2021)

• Mention Flags are generalised to the Neural Rule-Execution Tracking Machine, which can incorporate a wide range of lexical constraints and task-specific prior knowledge. (Published in NeurIPS 2021)

• Soft Prompt Neural Text Generators are applied to generate high-quality and diverse training examples for various NLU tasks under the low-resource setting. (Published in ACL 2022)

From the acquired experimental results, it is demonstrated that the proposed methods outperform a wide range of competitive baselines and a new state-of-the-art on a wide range of tasks is set.