1) I introduce a method for learning compositional structure that incorporates recursive computation and learns representations for each syntactic constituent. This novel approach inspires a prompting method to equip LLMs with compositional skills, leading to state-of-the-art semantic parsing results on a challenging compositional generalization benchmark. In this synthetic setting, strong performance is achieved by retrieving prototypical demonstrations for the semantic parsing task, although real world tasks often require fusing retrieved background knowledge with the model prediction.
2) Retrieving relevant documents from an external datastore is an effective way for language models to automatically ground their predictions externally rather than solely rely on their internal memory. I design an adaptive algorithm that discards distracting or irrelevant documents, and more heavily weights the influence of relevant text. The more precise usage of the datastore leads to state-of-the-art performance on a language modeling benchmark for generating encyclopedic text.
(Future and Ongoing Work) 3) To further improve language model grounding on external knowledge, I develop a parsing task used to train an external controller to incrementally plan and decide when and what to retrieve during text generation. The external controller improves the relevance of retrieved documents and enables the productive incorporation of larger amounts of related information. This is especially helpful when generating complex compositional answers that incorporate multiple documents. As an example question and answer:
Q: If it did, how do you think Brian's opinion on the rebellion changed throughout the story?
A: He's a conformist at the start, a rebel at the end.
I demonstrate that these methods improve the accuracy of LLMs, thus addressing the criticisms and providing a path forward for more effective language model use.
Advisors: Mohit Iyyer and Andrew McCallum