Geographic Data Service

Introduction

The document provides a discussion around the use of our search tool. It details the methods underpinning the technology, outcomes associated with the evaluation of its use, and describes some concerns commonly raised about the work. The creation of this document was recommended and informed through the series of four focus groups with key users of secondary research datasets (i.e., PhD researchers, academic researchers, data service staff, local government and third sector analysts). Focus groups highlighted the importance of transparency in the tool to help build trust and reassurance. This document has been created in the spirit of having an open discussion around the tool that has been created, including its strengths and weaknesses. We have structured the document in relation to the key themes of questions.

How to use the search tool

Federation of data discovery

What is the purpose of the tool?

The aim of the tool is to improve the discoverability of data held across multiple research data catalogues. We achieved this through implementing a semantic search approach, moving beyond traditional search methods which focus solely on the presence on keywords. Semantic search constructs a semantic representation of user queries using a Large Language Model (LLM) and compares this with semantic representations of the metadata of data services. Results returned are more directly linked with the semantic meaning of search queries, retrieving datasets that may have been ignored through traditional search which depends on the user knowing the correct combinations of keywords to match the datasets they need. We further add functionality so that the LLM can optionally explain why results were returned, allowing for increased transparency in the results provided.

When should researchers use this tool?

We recommend that the tool is used when designing a research project to help identify relevant secondary datasets to analyse.

Does the tool make decisions around which datasets researchers should use?

The tool only shows relevant datasets based on the user's search query. Datasets are returned in order of semantic similarity, with those displayed first being the most relevant to their query. The final say is always with researchers for which dataset to use.

Did you consult users in the design of this tool?

In December 2024, we ran four focus groups with a range of groups who use research datasets including: (i) academic researchers, (ii) PhD student researchers, (iii) data service providers, and (iv) local government and third sector analysts. We have written a paper that analyses the insights from these focus groups which a pre-print was published here. Some of the findings from these focus groups informed the design of the tool and other issues raised during focus groups influenced the questions in this document.

Which data services are included in the search tool?

The tool contains four data services: Geographic Data Service (formerly part of the Consumer Data Research Centre, which version 1 of the tool contained), UK Data Service, Administrative Data Research UK and NERC Environmental Data Service. These four were selected since they represented large UKRI investments capturing the main areas for storing research datasets across social and environmental sciences. Federated data discovery means building tools that allow researchers to find relevant data across whole data ecosystems, rather than keeping everything in silos that is often the case currently (i.e., researchers must search across multiple websites). We will continue to update metadata records where possible.

Are there any plans to expand the range of data services included in the tool?

There are no plans to onboard any further data services currently. If you are interested in bringing a new data service into the tool, please get in contact with Professor Mark Green (see contact details at the end of this document).

Methods

How does the search part of the tool work?

We have detailed the exact technical specification of the tool on the GitHub page for the project code. In brief, our search tool combines technology using Large Language Models (LLMs) and traditional retrieval augmented generation (RAG).

We collected the metadata records for a range of UK research data services. We take these metadata and extract out the text information from them. This step strips the structure out of the metadata and converts each record into a single block of text. Next, we extract embeddings out of the text matched to OpenAI's embeddings. This step converts the text into high-dimensional vectors that encapsulate the semantic meaning of the content (i.e., words and terms in each metadata record). We specifically use OpenAI's GPT-4o mini model. The reason that we use this model is that it is lightweight and efficient to call yet still performs well when capturing the semantic meaning of words. All the metadata and embeddings are then stored on a Pinecone vector database.

When a user enters their query into the website hosting our tool, we extract the embeddings of the user's text using the same method as we did for the metadata. The tool then retrieves the top 'k' results from the Pinecone vector database based on the semantic similarity using a nearest neighbour approach. These results are then presented to the user on the website in order of their semantic similarity (i.e., the most 'similar' dataset is returned first). The ranked list is also blended with a keyword-based search so that users can balance both approaches for searching together. We limit the returned results based on a threshold value for similarity so that those which are not relevant are not showed to the user.

What does the 'AskAI' button do and how does it work?

A button appears next to each search result (i.e., returned dataset) which allows a user to query a LLM about why the specific dataset was relevant to their query. This was introduced to improve the transparency of search results and help a researcher see why a dataset would be relevant. Clicking the button sends a query to OpenAI's GPT-4o mini model which sends: (i) the user's query, and (ii) the associated metadata of the selected dataset. It then asks the LLM to write a short response of why the dataset was relevant to their query.

Can the tool 'make up' or hallucinate datasets?

The tool has been designed so that this is not possible. It can only return datasets contained in the metadata documents.

Where hallucinations are possible is in the 'Ask AI' button which creates a generative summary of why a dataset was relevant to the query. Hallucinations are possible since the process uses generative AI to create text summaries. To prevent this from occurring, we have a second 'agent' which checks the generated summary against the metadata and assesses if the information in the summary matches the metadata. Where this process finds divergence, the tool will return to the user "Hallucination found in generation."

How has the AI tool been trained?

We do not undertake any training of the LLM to tailor it to the research datasets in the catalogue. This is because we only use the LLM embeddings to understand the relationships between words/terms. We felt that the LLM has a good grasp of the semantic meanings of words that it did not require further training.

Are the tool and code open source?

The code for the tool are openly published on https://github.com/cjber/semantic-catalogue. The tool is primarily developed using Python which is open source software. The Large Language Model we use (GPT-4o mini) is not open source and the methodology remains unknown. This means that we are unable to explain how the model works or the underlying biases that it may possess. This is the only closed part of the tool.

Who created the tool?

The tool was co-created by teams at the University of Liverpool (Professors Alex Singleton and Mark Green) and University of Manchester (Professors Caroline Jay, Richard Kingston and David Topping, and Dr Maura Halstead). We also had support from Dr Cillian Berragan who was based at the University of Liverpool for the first half of the project. Robert Podmore and Meilin Shi, both from University of Liverpool, have both also helped us with getting the search tool working on the website. The University of Liverpool team led on the software development, with the University of Manchester team leading on the evaluation of the tool.

Some of the URL links do not work?

We present only the URL links that are part of the metadata made available to us by data services. Sometimes the links may be incomplete or erroneous. We will continue to work with data services to ensure complete coverage of metadata. We also note that for some UK Data Service (UKDS) links, they may bring you to a page that says "No study found with id… if this is a Reshare study, please try the link: Reshare study" (e.g., https://datacatalogue.ukdataservice.ac.uk/studies/study/857704#details). This is an issue with how UKDS creates URLs and how they interact with our system. All you need to do is click on the 'reshare study' link on that page and it will bring it through to the correct website.

Evaluation of the tool

How reliable and valid is the tool in finding datasets?

We performed a comparison of how our tool compares to a traditional keyword based search engine when searching for research datasets. We have published the findings from this work alongside this document. Here we provide a short summary of that document but recommend that users read our preprint to understand the evaluation process and how well the tool performs.

Our tool provides users with a larger number of search results than a keyword search tool would for the same query. This appears to be explained by the addition of using a semantic search approach. A keyword-based search tool only matches on exact keyword matches and many datasets have a limited number of keywords. Our approach will find datasets with semantically similar terms that do not exactly match the listed keywords. For example, searching for 'diabetes' on the Consumer Data Research Centre website would give no results returned as there are no datasets with that as a keyword. Using our tool, this gave five datasets for the same data service since it matched to datasets which included a range of health outcomes (including diabetes) that did not list diabetes as a keyword, or matched datasets that might provide contextual explanations for diabetes patterns (e.g., food environment data). Similarly searching for data on a specific place using our tool would show national-level datasets that have geographical data in them where datasets don't list every location as a keyword.

The results returned in our tool were found to be semantically similar to those in a keyword search tool. This would suggest that although more results are returned, they are not thematically different to the user query or what a keyword search tool would show. The order of returned results in our tool was not always the same as in a keyword search tool suggesting some differences in how 'importance' was viewed.

Our tool is resilient to spelling errors and will still give relevant results with minor spelling errors to terms. This is different to a keyword search resource which depends on the exact wording to work. Our tool can also handle more complex queries including full or partial statements or research questions. Traditional keyword searches work best when only one term is used, meaning that our tool might help users write in a natural language or where they do not know the exact terms to search using.

Users should be aware that some research specific acronyms can perform poorly on the search tool. For example, our evaluation work showed that the tool did not recognise 'PDV' which is the name of a Geographic Data Service dataset and would be captured with a keyword search. These were not too common across historical search records, but suggest that if users are concerned they should spell out their search queries fully rather than rely on just the acronym.

We are currently undertaking a pilot analysis where people will be randomly allocated to either our tool or a traditional search website to assess whether it helps people to find datasets quicker or gives them better answers. We will report on these findings openly once this work is complete.

What are the potential biases of using a LLM for processing user queries?

The underlying LLM is trained on a large body of information across the Internet, books and other text sources. These tend to reflect the dominate perspectives of the authors of these sources, meaning that it was trained on information over-represented by mainstream Western, English-speaking and often male voices. It is plausible that these inequities may feed through into the inter-relationships between words (e.g., connections between words being favoured in one context over another). We have not noticed any obvious biases due to this during our testing, but this does not necessarily mean they don't exist.

Are there opportunities for the tool to learn based on feedback?

We log all search records and what was returned. This information is anonymous (i.e., we do not know who searches for information, only what is typed in the search box). We will use these records to refine the tool based on identifying areas where it is less successful. We will revise the tool considering any feedback and tailor the code so that it continues to offer the best service possible.

What is the environmental impact of using this tool?

This will depend on how the user engages with the tool. Below we give our best estimate for all possible parts of the tool usage, although accept that we do not know the exact figure as it is difficult to calculate.

When a user searches for a dataset on the tool, querying the Pinecone vector database and returning the result contributes to ∼0.0005 gram of CO²-equivalent. This impact is negligible (e.g., equivalent to 1/100,000^th of a cup of coffee or 1/23,000^th of the CO² exhaled in a single breath).

If the user clicks on the 'AskAI' button, this has a larger impact. We estimate that each query here contributes between 0.1 and 0.2 grams CO²-equivalent. This is equivalent to driving a petrol car 1-2 metres or 1/250^th of a cup of coffee.

In comparison, it has been estimated that a typical Google search query uses between 0.1 and 0.2 grams of CO²-equivalent. This would suggest that our tool has a significantly lower environmental footprint than Google if one just uses the search functionality alone. Clicking on the 'AskAI' button once would bring the impact in line with the impact of a single Google search.

The estimates for our tool above relate to ongoing searches. They do not include the cost of the development of the technology that underpins our tool.

There is significant environmental footprint of training, running and maintaining large language models both in electricity generation and water consumption. For example, it was estimated that OpenAI's ChatGPT-3 model consumed enough energy during training to power 1000 households for one year. Our tool benefits from these emissions, even if it we do not re-train the model itself.

We store the metadata embeddings on a Pinecone vector database. There is no public documentation of the environment footprint of this service. We estimate based on data for similar cloud-based servers that hosting our data on Pinecone equates to ∼1.7 grams of CO²-equivalent per year. This is equivalent to driving a petrol car 15 metres once a year.

Contact

If you have any questions about this report, please get in touch with:

Professor Mark Green
Department of Geography and Planning
University of Liverpool
Liverpool
L69 7ZT
Email: mark.green@liverpool.ac.uk

Acknowledgements

This work was supported by the Economic and Social Research Council [grant numbers ES/Z502947/1, ES/L011840/1 and ES/Z504464/1].

Mark Green¹, Alex Singleton¹, Maura Halstead², Caroline Jay², Richard Kingston³, David Topping³.

1. Department of Geography and Planning, University of Liverpool, UK
2. Department of Computer Science, University of Manchester, UK
3. Department of Geography, University of Manchester, UK
4. Department of Earth and Environmental Sciences, University of Manchester, UK