Knowledge Graphs as a source of trust for LLM-powered enterprise question answering - Leitura de Artigo

J. Sequeda, D. Allemang and B. Jacob, Knowledge Graphs as a source of
trust for LLM-powered enterprise question answering, Web Semantics: Science, Services and
Agents on the World Wide Web (2025), doi: https://doi.org/10.1016/j.websem.2024.100858.

1. Introduction

These question answering systems that enable to chat with your structured data hold tremendous potential for transforming the way self service and data-driven decision making is executed within enterprises. Self service and data-driven decision making in organizations today is largly made through Business Intelligence (BI) and analytics reporting. Data teams gather the original data, integrate the data, build a SQL data warehouse (i.e. star schemas), and create BI dashboards and reports that are
then used by business users and analysts to answer specific questions (i.e. metrics, KPIs) and make decisions. The bottleneck of this approach is that business users are only able to answer questions given the views of existing dashboards.

If a new question is required and it is not answerable through a report, then a new report needs to be created or an existing report needs to be extended. Furthermore, the data may exist in the warehouse, but if a dashboard does not exist to access that data, then business users can not use the data. Thus, self
service is only possible within the current set of dashboards, even if the data is currently available.

DW são schema on write, as perguntas precisam ser conhecidas em tempo de modelagem

In the second scenario, the LLM transforms the user’s question into a structured query, which is then executed against a governed database or knowledge graph. This setup introduces several improvements:

• • Readable and Verifiable Queries: Unlike an opaque response, a generated query can be reviewed, assessed for correctness, and matched against a schema/ontology
•  • Potential for Explanation: The generated query allows for a transparent examination of the relationships it attempts to address, providing a way to explain how the LLM interpreted the question and why the given answer is correct.
• • Governed Data Source: The response comes from curated data, which means the accuracy can be evaluated, and the data source can be referenced directly.

Natural Language Interfaces for Databases

2. Knowledge Graphs to Increase Accuracy of LLM-powered Question Answering on SQL databases

Our first contribution is a benchmark, with data and knowledge in the insurance domain, and experimental results providing evidence that an GPT-4 powered question answering system using a zero-shot prompt that answers enterprise natural language questions over a Knowledge Graph representation of an enterprise SQL database returns 3x more accurate results compared to using only the SQL database. 

Converter banco relacional em KG me lembra as abordagens de integração de dados

These research results have been productized and are part of the data.world AI Context Engine2 (AICE) product, an agent-based and API-driven system that connects our customer’s data and metadata with Generative AI to power trusted conversations with structured data and metadata. .... The GraphRAG Manifesto by Neo4j argues that one of the benefits of GraphRAG relative to vector-only RAG is due to higher accurate responses, citing our benchmark and results.

É um produto

3. Lessons Learned
3.1. Knowledge Engineering is key for Accuracy

This brings us to an important realization: this work of knowledge engineering - figuring out what the data truly means - is critical and should ideally be a standard practice in organizations.

Another lesson learned, and arguably a controversial one: the ontology should be defined in a way that is conducive to the success of the LLM. Often, the “correct” ontology might confuse the LLM resulting in incorrect SPARQL queries being generated.

Processo de Engenharia do Conhecimento

3.2. Explainability

This transparency allows users to understand how the answer is derived.

We are providing further explainability by presenting back to users the generated SPARQL query, the parts of the ontology and mappings referenced, the specific business terms applied, and the resulting generated SQL query. All these elements contribute to building trust. This level of explainability allows technical users to verify each part of the process and increases confidence in the final answer.

A confiança neste produto/arquitetura depende da capacidade de explicar as respostas geradas

3.3. Governance

Governance involves ensuring that the terms used in the system align with the organization’s business glossary, which is maintained by data stewards. These terms and definitions are passed as additional context during processing. Governance is also crucial in defining specific metrics, providing consistency, and ensuring that the terminology used in querying aligns with approved business definitions.

Terminologia nos esquemas e ontologias que ajudem no mapeamento do LLM

3.4. Avoiding “Boiling the Ocean”

Instead, we follow the “pay-as-yougo” methodology, starting small and building up gradually Sequeda, Briggs, Miranker and Heideman (2019). This approach begins with identifying a small set of business questions, and building the ontology and mapping to model those questions. The lesson learned is that this is new technology and stakeholders are eager to see results, thus we must show results quickly (on a weekly basis) and iterate. These questions must be business-oriented, not technical, and require clear context: Who is asking the question? Why is it being asked?

Abordagem para lidar com a complexidade de modo gradual gerando perguntas de competência contextualizas (5W1H)

4. Industry Needs and Future Research Contributions

4.1. Simplifying Knowledge Engineering

Creating Knowledge Graphs by defining target ontologies and mappings from source relational databases to a target ontology is still a complex social process. Knowledge engineering needs to be simplified which underscores the requirement for more effective tools and methodologies. From a tooling perspective, we believe there is an opportunity to draw lessons from established practices in areas such as Extract, Transform, Load (ETL) tools, which have effectively navigated similar challenges with tabular data

Our position is that ontology engineering methods needs to extend their approaches to consider mapping to existing data sources

Usar o conhecimento sobre ETL para produzir estes mapeamentos de consulta em tempo real.

4.2. User-Centric Explainability

Our position is that we must also focus on: "explainable to whom?" A research opportunity is to better understand the specific needs of users regarding explainability. While there are numerous ways to provide explainability, it is crucial to tailor these methods to the requirements and expectations
of different user groups. A user-centric approach to explainability means understanding what level of detail or type of explanation various users need, ranging from business analysts to technical experts.

A ROT poderia ser usada para explicar as evidências e sinais da Confiança do Usuário

4.3. New Approaches to Testing Non Deterministic Systems

We are entering an era of working with non-deterministic systems (i.e LLMs). Traditional testing approaches assume systems are deterministic, providing the same output for a given input every time.We need to explore new frameworks and approaches for testing these non-deterministic systems.

Ainda é necessário classificar os resultados como certos ou errados.

4.4. Small Semantics vs. Larger Semantics

4.5. Multi Agent-Based Question Answering Systems and Problem Decomposition

As we build question-answering agent systems, a major consideration is how to break down the problem into smaller, more manageable components. Should an agent deal with the core task of answering questions and managing ambiguity, or should these should be split into separate agents.

Busca Exploratórias e necessidades de informação decompostas em diferentes perguntas

5. Final Remarks

Trust in a question answering system comes from a combination of sources. The agent that provides the final answer must be an accountable one; an LLM alone does not qualify. The database or a knowledge base can, as they are curated and accountable resources. When a knowledge graph is used as the data source, the ontology can be formally used to verify queries, help vet incorrect queries, identify necessary fixes, which leads to increased accuracy, explainability and governance.

Q&A não é o suficiente para cenários de tomada de decisões.