Metadata
- Author: Eugene Yan
- Full Title:: A Builder’s Guide to Evals for LLM-based Applications
- Category:: 🗞️Articles
- Document Tags:: Foundational models,
- URL:: https://eugeneyan.com//writing/evals/
- Finished date:: 2024-04-01
Highlights
A related task is opinion summarization. This is where we generate a summary that captures the key aspects and associated sentiments from a set of opinions, such as customer feedback, social media, or product reviews. We adapt the metrics of consistency and relevancy for: • Sentiment consistency: For each key aspect, does the summary accurately reflect the overall sentiment expressed? For example, if most reviews praise the battery life but criticize the camera quality, the summary should capture this. • Aspect relevance: Does the summary cover the main topics discussed? If many reviews raise concerns about battery life and camera quality, these points should be included in the summary. The OpinSummEval paper explored several evals and found two to be most effective: BARTScore and Question-Answering (QA) based evals. (View Highlight)