CORDIAL-AI: Innovating with Large Language Models at the UK Data Service for Census Information Discovery and Retrieval using Natural Language

The advancements of AI and the rapid growth of its use across society present both challenges and opportunities. At the UK Data Service (UKDS), we aim to explore the full potential of this technology while committing to its responsible and ethical use. This pilot contributes to the second thematic aspect of the call on data discovery using AI, aiming to employ Large Language Models (LLMs) for data discovery and retrieval.

The pilot will utilise a specific type of set of data from the extensive catalogues of UKDS, which we believe is an ideal example to explore the capabilities and limitations of LLMs while building future data services. Census flow data, which involves the movement of people between an origin and a destination - such as migration and travel to work or study - is particularly intricate, incorporating multiple areas, geographies, and variables.

The complexity and diversity of these datasets make them difficult to locate and use, resulting in underutilisation despite their significant potential for research and policy-making.

Led by UCL as an integral partner of the UK Data Service (UKDS), we aim to develop an intuitive system that allows users to query complex structured data in plain English. It will be incorporated into a new platform that the UCL UKDS team is developing for flow data. The new platform, currently in its final development stages (release expected soon in late Q2 2024), introduces an API for advanced flow data discovery and retrieval as well as an interface designed for seamless interaction with this API that will eventually replace the current legacy software.

We believe the new platform presents an excellent opportunity for advantageous use by this pilot due to its modularity and the relative ease with which it can interact with AI. The pilot will create knowledge bases and fine-tune, test, refine, and integrate LLMs using the latest available methods, ensuring they meet high standards of accuracy and efficiency for the delivery of reliable and affordable solutions to information discovery and retrieval.

The approach and methodologies for developing the model will also work towards enhancing the explainability and interpretability of this technology, which it inherently lacks because of its black-box nature.

This pilot aims to transform the way users interact with complex and specialised datasets through a series of strategic objectives. Firstly, we will train an LLM to adeptly handle census flow data, including intricate code lists and numerical information, pushing beyond the conventional boundaries of LLM applications. To facilitate access, we will introduce an API endpoint and a web interface within our new platform, enabling natural language queries for data discovery and retrieval.

We will work towards enhancing the model's reliability and user trust by implementing data provenance and explainable AI (XAI) methods. The project will include a workshop and a webinar aimed at sharing progress and findings, soliciting feedback, and encouraging discussions on research ideas and collaborations.

Using the experience gained from this pilot, we aim for future interoperability of information retrieval using LLMs across the UKDS. This project could synergise with other future initiatives at UKDS that utilise LLMs across its broad data catalogues to improve data discovery and promote interpretability and explainability as part of a responsible adoption of this technology.