OMX Helsinki — S&P 500 — DAX — NASDAQ 100 — STOXX 600 — EUR/USD — EUR/SEK — BTC/USD — ETH/USD — Euribor 3 mån — Euribor 12 mån —
AI Sweden

AI Sweden

Teknik och AI 3,000 prenumeranter

NLP Seminar Series – Text Summarization and Evaluation in the Era of GPT-3 – Tanya Goyal

16.12.2022 223 visningar
Publicerad: 16.12.2022 Kategori: Teknik och AI Visningar: 223

Avsnittsinformation

Avsnittsbeskrivning

Titel: Text Summarization and Evaluation in the Era of GPT-3 Speaker: Tanya Goyal, PhD student at University of Texas at Austin Abstract: The recent success of zero- and few-shot prompting with models like GPT-3 has led to a paradigm shift in NLP research. We study its impact on text summarization, focusing on the classic benchmark domain of news summarization. First, we investigate how zero-shot GPT-3 compares against fine-tuned models trained on large summarization datasets. We show that not only do humans overwhelmingly prefer GPT-3 summaries, but these also do not suffer from common dataset-specific issues such as poor factuality. Next, we study what this means for evaluation, particularly the role of gold standard test sets. Our experiments show that both reference-based and reference-free automatic metrics, e.g. recently proposed QA- or entailment-based factuality approaches, cannot reliably evaluate zero-shot summaries. Finally, we discuss future research challenges beyond generic summarization, specifically, keyword- and aspect-based summarization, showing how dominant fine-tuning approaches compare to zero-shot prompting. AI Sweden is the Swedish national center for applied artificial intelligence. Our mission is to accelerate the use of AI for the benefit of our society, our competitiveness, and for everyone living in Sweden. Website: https://ai.se Contact: info@ai.se Follow us Newsletter: https://www.ai.se/en/AI-Sweden-newsletter LinkedIn: https://www.linkedin.com/company/aisweden Medium: https://medium.com/ai-sweden

Öppna chatten med AI-assistenten. Chatten laddas först när du öppnar den.