Introduction

In short, LLM response validation is the use of an LLM to evaluate AI-generated texts based on user-specified criteria stated in the evaluation prompt.
Since AI-generated texts may not always align with facts or the user-provided context, issues of trustworthiness, traceability, quality, and even safety can arise. There is a need to detect false or hallucinated responses before delivering them to users.

Validation is not limited to factual alignment; it can also assess other aspects of responses, such as:

• Bias: prejudice against certain group
• Tone: formal or casual
• Format: JSON or custom format
• Sentiment: positive, negative, or neutral
• Faithfulness to source: only include information appearing in the source
• Correctness: May need tools to access external information source
• Helpfulness: if the response fully answers the query?   

To validate, the AI-generated text together with the evaluation prompt (with criteria stated) is sent to the LLM. The LLM then provides comments or a score to indicate the evaluation results.

Prompt Example

Given the user query/question and context information: {{QUERY +CONTEXT}} and the response: {{RESPONSE}}, examine whether the responseinformation is solely based on the context. Ensure that all facts described inthe response can be found in the context. Return your result as a label: ‘Yes’or ‘No,’ followed by a concise reason to support your decision.

‍

You may wonder why an LLM is used to assess the responses of another LLM — or even itself. The point is that generating a response to a user query is more difficult than evaluating the quality of that response, since the latter focuses only on quality factors. Content critique is easier than content generation. Moreover, we can even use the same LLM for evaluation because this leverages different capabilities of the model.