Basic Metrics

The metrics on this page require basic information from the Step 1.

Faithfulness

This checks how well the answer matches the provided information. It looks at the answer and the context together.

An answer is considered accurate if everything stated in it can be supported by the context. To figure this out, we first identify the main points in the answer and then see if each one can be found in the context.

Answer Relevance

The evaluation metric measures how relevant the answer is to the question. Answers that are incomplete or repeat information get a lower score, while more relevant answers receive higher scores.

Answer Relevancy is determined by comparing the original question to several artificial questions created based on the answer, looking at their similarity.

Harmfulness

Harmfulness indicates whether the generated answer includes elements that could cause emotional or psychological harm, such as offensive language, insults, misinformation, or any content that negatively impacts the participant's well-being. It may also include behaviors that promote conflict, discrimination, or distress.

Maliciousness

Maliciousness indicates whether the generated answer mentions the intention to harm, deceive, or provoke a negative reaction. It includes comments intended to hurt others, spread misinformation, or create conflict.

Coherence

Coherence shows the clarity and logical flow of the answer. It means that the messages are connected and make sense together, allowing participants to easily understand and follow the discussion.

Correctness

Correctness refers to the accuracy and truthfulness of the information being shared. It means that the statements made are factual and reliable, ensuring that participants can trust the content of the discussion.

Conciseness

Being concise in a conversation means expressing ideas clearly and directly without unnecessary words. This includes getting your point across effectively, focusing on the essential information and avoiding any filler or redundancy. This metric expresses that the message is understood quickly and effectively.

Context Length

Context length shows if the length of the context used by the LLM to answer the question is within a specified range.

Response Length

Response Length shows whether the answer from the LLM is within a specified length range.