An Evaluation Metric for Assessing Summary-Level Semantic Similarity in Abstractive Text Summarization

Abstractive text summarization focuses on producing summaries that retain the most critical information from the original document. ROUGE is a widely used metric for evaluating summarization quality. However, it relies solely on comparing n-grams between the generated and reference summaries. To overcome this limitation, this paper introduces a semantic similarity-based evaluation metric. Unlike ROUGE, the proposed approach evaluates summaries by assessing the similarity between system-generated and reference summaries using cosine similarity between their sentence embeddings. When evaluating the abstractive summaries on the ROUGE and the proposed evaluation metric, it is observed that the latter has a stronger correlation with human judgment.