Creating more truthful AI: Enhancing language model reliability

Have you ever caught a large language model making false claims?

It’s no wonder—the information is constantly changing, and when AI responses aren’t grounded in reliable sources, facts can get twisted.*

* Sometimes referred to as hallucination, this tendency for AI to generate false or non-existent information is one of the biggest challenges we face in making AI more reliable.

As the saying goes, trust takes years to build but can be lost in seconds. A single incorrect answer can undermine the credibility of an AI model, leaving users unsure of what to believe. It’s like navigating a maze with a faulty map, guiding users down paths that don’t exist.

As AI plays a growing role in critical decision-making situations, from medical diagnoses to legal guidance, misinformation can have serious consequences. That’s why, when developing AI technology, we must focus on making AI models more truthful rather than just feeding them more data.

In this post, we’ll take you behind the scenes of our research on improving AI truthfulness. We’ll walk you through how our AI models arrive at their answers and the methods we use to make their responses more dependable.

Fact checker: Validating responses

At NAVER, we’ve developed a sophisticated fact-checking system that enhances the accuracy of AI-generated responses. Our system validates these responses through a three-step verification process:

To demonstrate how this works, consider the following example:

Q. Does NAVER have a data center at its training facility?

Step 1: Fact extractor

The first step in our verification process involves the fact extractor, which analyzes the language model’s draft response at a sentence level. It identifies factual statements and turns them into questions that can be checked for accuracy.

LLM’s first draft

Validation questions

NAVER’s training facility does not house a data center.The facility, called NAVER Connect One, is located in Chuncheon, Gangwon Province.

This smart office, built on NAVER’s technology, provides an array of spaces and activities designed to boost employees’ creativity and productivity.

NAVER’s first data center opened in June 2013 and is located in Seoul at 719-** Yeoksam-dong, Gangnam-gu. This green data center leverages environmentally friendly energy to lower power usage effectiveness (PUE) and maximize energy efficiency.

Is NAVER’s training facility called NAVER Connect One located in Chuncheon, Gangwon Province?

Is NAVER’s data center located in Seoul at 719-** Yeoksam-dong, Gangnam-gu?

Step 2: Auto-verifier

Once the fact extractor forms validation questions, the auto-verifier determines whether the answer to each validation question is true or false.

Fact extractor

Auto-verifier

Is NAVER’s training facility called NAVER Connect One located in Chuncheon, Gangwon Province?

NAVER’s training facility, called NAVER Connect One, is located in Chuncheon, Gangwon Province at 330-29 Mancheon-ri, Dong-myeon. Situated on Gubong Mountain, it offers panoramic views of Chuncheon city.

Is NAVER’s data center located in Seoul at 719-** Yeoksam-dong, Gangnam-gu?

NAVER’s data centers are not located in Seoul at 719-** Yeoksam-dong, Gangnam-gu.

NAVER’s first data center, Gak Chuncheon, opened in June 2013 and is located in Chuncheon, Gangwon Province at 1231 Sunhwan-daero, Dong-myeon. The company’s second data center, Gak Sejong, is located in Sejong at 824 Haengbok-daero.

The auto-verifier validates statements through auto-browsing, which simulates human search behavior. Our auto-browsing system interacts with a virtual browser to trace information back to verifiable sources. This automated search is essential for checking LLM outputs against reliable data. The system efficiently performs multiple searches to ensure thorough fact-checking of each statement.

[Read more about auto-browsing here.]

The auto-verifier evaluates whether responses to validation questions are true or false based on the following criteria:

For a true determination

The verifier confirms a statement as true when all of these conditions are met:

1) The statement can be linked to a credible source.

2) The verification model determines the statement is true.

3) The response includes both supporting evidence and source citations.

False determination

The verifier marks a statement false if any of these occur:

1) The auto-browser fails to find supporting information.

2) The verification model determines the statement is false.

3) The response fails to provide citations or direct quotes.

Step 3: Re-writer

The final step is handled by the re-writer, which takes the draft answer generated by the language model and refines it based on the earlier verification results. It eliminates any factual inaccuracies and enriches the response by adding supplementary material, all the while preserving the original tone and style.

Fact extractor

Auto-verifier

Re-writer

Is NAVER’s training facility called NAVER Connect One located in Chuncheon, Gangwon Province?

NAVER’s training facility does not house a data center.The facility, called NAVER Connect One, is located in Chuncheon, Gangwon Province at 330-29 Mancheon-ri, Dong-myeon. Situated on Gubong Mountain, it offers panoramic views of Chuncheon city.

This smart office, built on NAVER’s technology, provides an array of spaces and activities designed to boost employees’ creativity and productivity.

NAVER’s first data center, Gak Chuncheon, opened in June 2013 and is located in Chuncheon, Gangwon Province at 1231 Sunhwan-daero, Dong-myeon.

This green data center leverages environmentally friendly energy to lower power usage effectiveness (PUE) and maximize energy efficiency.

The company’s second data center, Gak Sejong, is located in Sejong at 824 Haengbok-daero.

Is NAVER’s data center located in Seoul at 719-** Yeoksam-dong, Gangnam-gu?

NAVER’s data centers are not located in Seoul at 719-** Yeoksam-dong, Gangnam-gu.

NAVER’s first data center, Gak Chuncheon, opened in June 2013 and is located in Chuncheon, Gangwon Province at 1231 Sunhwan-daero, Dong-myeon.

The company’s second data center, Gak Sejong, is located in Sejong at 824 Haengbok-daero.

AI-generated answers you can trust

The following table demonstrates how the language model’s initial draft is refined through our fact-checking process, resulting in a final response that is more precise:

Question

Before revision:

After revision:

Does NAVER have a data center at its training facility?

NAVER’s training facility does not house a data center.The facility, called NAVER Connect One, is located in Chuncheon, Gangwon Province.

This smart office, built on NAVER’s technology, provides an array of spaces and activities designed to boost employees’ creativity and productivity.

NAVER’s first data center opened in June 2013 and is located in Seoul at 719-** Yeoksam-dong, Gangnam-gu.

This green data center leverages environmentally friendly energy to lower power usage effectiveness (PUE) and maximize energy efficiency.

NAVER’s training facility does not house a data center. The facility, called NAVER Connect One, is located in Chuncheon, Gangwon Province at 330-29 Mancheon-ri, Dong-myeon. Situated on Gubong Mountain, it offers panoramic views of Chuncheon city.

This smart office, built on NAVER’s technology, provides an array of spaces and activities designed to boost employees’ creativity and productivity.

NAVER’s first data center, Gak Chuncheon, opened in June 2013 and is located in Chuncheon, Gangwon Province at 1231 Sunhwan-daero, Dong-myeon.

This green data center leverages environmentally friendly energy to lower power usage effectiveness (PUE) and maximize energy efficiency.

The company’s second data center, Gak Sejong, is located in Sejong at 824 Haengbok-daero.

Our efforts to overcome AI challenges are ongoing. At NAVER, we’re committed to further developing fact-checking and other validation technologies to build more responsible AI that serves the needs of our users and society.

Our future with AI

Since December 2024, NAVER has integrated AI fact-checking into our Q&A platform, NAVER Knowledge iN. Our AI Responder, powered by CLOVA AI, generates data-driven answers to user questions, all verified against reliable sources.

Generating false information isn’t just a technical challenge—it’s an ethical responsibility. We must proactively identify potential threats brought on by technological advances and work toward minimizing them. At NAVER, this commitment drives our continuous efforts to create AI that is both accurate and helpful. Our goal is to pave the way for a future in which humans and AI interact in meaningful and beneficial ways.