OMX Helsinki — S&P 500 — DAX — NASDAQ 100 — STOXX 600 — EUR/USD — EUR/SEK — BTC/USD — ETH/USD — Euribor 3 mån — Euribor 12 mån —
AI Sweden

AI Sweden

Teknik och AI 3,000 prenumeranter

NLP Seminar Series – Ibrahim Alabdulmohsin & Xiaohua Zhai

15.11.2022 193 visningar
Publicerad: 15.11.2022 Kategori: Teknik och AI Visningar: 193

Avsnittsinformation

Avsnittsbeskrivning

Titel: Revisiting Neural Scaling Laws in Language and Vision Speaker: Ibrahim Alabdulmohsin, Research Scientist at Google Brain, and Xiaohua Zhai, Staff Researcher at Google Brain Abstract: The remarkable progress in deep learning in recent years is largely driven by improvements in scale, where bigger models are trained on larger datasets for longer schedules. To predict the benefit of scale empirically, we argue for a more rigorous methodology based on the extrapolation loss, instead of reporting the best-fitting (interpolating) parameters. We then present a recipe for estimating scaling law parameters reliably from learning curves. We demonstrate that it extrapolates more accurately than previous methods in a wide range of architecture families across several domains, including image classification, neural machine translation (NMT) and language modeling, in addition to tasks from the BIG-Bench evaluation benchmark. To accelerate research in this domain, we also release a benchmark dataset comprising of 92 evaluation tasks. Finally, we dive in depth into our efforts to study the scaling properties of Vision Transformers by characterizing the relationships between their error rate, data, and compute, which has resulted in state-of-the-art top-1 accuracy on ImageNet. The NLP Seminar Series at AI Sweden is a bi-weekly forum for people who work with or are interested in Natural Language Processing (NLP). It is organized by AI Sweden and RISE NLP Group. AI Sweden is the Swedish national center for applied artificial intelligence. Our mission is to accelerate the use of AI for the benefit of our society, our competitiveness, and for everyone living in Sweden. Website: https://ai.se Contact: info@ai.se Follow us Newsletter: https://www.ai.se/en/AI-Sweden-newsletter LinkedIn: https://www.linkedin.com/company/aisweden Medium: https://medium.com/ai-sweden

Öppna chatten med AI-assistenten. Chatten laddas först när du öppnar den.