@inproceedings{yoo-etal-2025-patentscore,
title = “{P}atent{S}core: Multi-dimensional Evaluation of {LLM}-Generated Patent Claims”,
author = “Yoo, Yongmin and
Xu, Qiongkai and
Cao, Longbing”,
editor = “Christodoulopoulos, Christos and
Chakraborty, Tanmoy and
Rose, Carolyn and
Peng, Violet”,
booktitle = “Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing”,
month = nov,
year = “2025”,
address = “Suzhou, China”,
publisher = “Association for Computational Linguistics”,
url = “https://aclanthology.org/2025.emnlp-main.1564/”,
doi = “10.18653/v1/2025.emnlp-main.1564”,
pages = “30727–30746”,
ISBN = “979-8-89176-332-6”,
abstract = “High-stakes texts such as patent claims, medical records, and technical reports are structurally complex and demand a high degree of reliability and precision. While large language models (LLMs) have recently been applied to automate their generation in high-stakes domains, reliably evaluating such outputs remains a major challenge. Conventional natural language generation (NLG) metrics are effective for generic documents but fail to capture the structural and legal characteristics essential to evaluating complex high-stakes documents. To address this gap, we propose PatentScore, a multi-dimensional evaluation framework specifically designed for one of the most intricate and rigorous domains, patent claims. PatentScore integrates hierarchical decomposition of claim elements, validation patterns grounded in legal and technical standards, and scoring across structural, semantic, and legal dimensions. In experiments on our dataset which consists of 400 Claim1, PatentScore achieved the highest correlation with expert annotations ($r = 0.819$), significantly outperforming widely used NLG metrics. This work establishes a new standard for evaluating LLM-generated patent claims, providing a solid foundation for research on patent generation and validation.”
}
PatentScore: Multi-dimensional Evaluation of LLM-Generated Patent Claims
Yongmin
Yoo
author
Qiongkai
Xu
author
Longbing
Cao
author
2025-11
text
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Christos
Christodoulopoulos
editor
Tanmoy
Chakraborty
editor
Carolyn
Rose
editor
Violet
Peng
editor
Association for Computational Linguistics
Suzhou, China
conference publication
979-8-89176-332-6
High-stakes texts such as patent claims, medical records, and technical reports are structurally complex and demand a high degree of reliability and precision. While large language models (LLMs) have recently been applied to automate their generation in high-stakes domains, reliably evaluating such outputs remains a major challenge. Conventional natural language generation (NLG) metrics are effective for generic documents but fail to capture the structural and legal characteristics essential to evaluating complex high-stakes documents. To address this gap, we propose PatentScore, a multi-dimensional evaluation framework specifically designed for one of the most intricate and rigorous domains, patent claims. PatentScore integrates hierarchical decomposition of claim elements, validation patterns grounded in legal and technical standards, and scoring across structural, semantic, and legal dimensions. In experiments on our dataset which consists of 400 Claim1, PatentScore achieved the highest correlation with expert annotations (r = 0.819), significantly outperforming widely used NLG metrics. This work establishes a new standard for evaluating LLM-generated patent claims, providing a solid foundation for research on patent generation and validation.
yoo-etal-2025-patentscore
10.18653/v1/2025.emnlp-main.1564
https://aclanthology.org/2025.emnlp-main.1564/
2025-11
30727
30746
%0 Conference Proceedings
%T PatentScore: Multi-dimensional Evaluation of LLM-Generated Patent Claims
%A Yoo, Yongmin
%A Xu, Qiongkai
%A Cao, Longbing
%Y Christodoulopoulos, Christos
%Y Chakraborty, Tanmoy
%Y Rose, Carolyn
%Y Peng, Violet
%S Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
%D 2025
%8 November
%I Association for Computational Linguistics
%C Suzhou, China
%@ 979-8-89176-332-6
%F yoo-etal-2025-patentscore
%X High-stakes texts such as patent claims, medical records, and technical reports are structurally complex and demand a high degree of reliability and precision. While large language models (LLMs) have recently been applied to automate their generation in high-stakes domains, reliably evaluating such outputs remains a major challenge. Conventional natural language generation (NLG) metrics are effective for generic documents but fail to capture the structural and legal characteristics essential to evaluating complex high-stakes documents. To address this gap, we propose PatentScore, a multi-dimensional evaluation framework specifically designed for one of the most intricate and rigorous domains, patent claims. PatentScore integrates hierarchical decomposition of claim elements, validation patterns grounded in legal and technical standards, and scoring across structural, semantic, and legal dimensions. In experiments on our dataset which consists of 400 Claim1, PatentScore achieved the highest correlation with expert annotations (r = 0.819), significantly outperforming widely used NLG metrics. This work establishes a new standard for evaluating LLM-generated patent claims, providing a solid foundation for research on patent generation and validation.
%R 10.18653/v1/2025.emnlp-main.1564
%U https://aclanthology.org/2025.emnlp-main.1564/
%U https://doi.org/10.18653/v1/2025.emnlp-main.1564
%P 30727-30746Markdown (Informal)(PatentScore: Multi-dimensional Evaluation of LLM-Generated Patent Claims)(https://aclanthology.org/2025.emnlp-main.1564/) (Yoo et al., EMNLP 2025)ACL
Source link
Leave a Reply