The Hardware Lottery (Paper Explained)

#ai #research #hardware We like to think that ideas in research succeed because of their merit, but this story is likely incomplete. The term "hardware lottery" describes the fact that certain algorithmic ideas are successful because they happen to be suited well to the prevalent hardware, whereas other ideas, which would be equally viable, are left behind because no accelerators for them exists. This paper is part history, part opinion and gives lots of inputs to think about. OUTLINE: 0:00 - Intro & Overview 1:15 - The Hardware Lottery 8:30 - Sections Overview 11:30 - Why ML researchers are disconnected from hardware 16:50 - Historic Examples of Hardware Lotteries 29:05 - Are we in a Hardware Lottery right now? 39:55 - GPT-3 as an Example 43:40 - Comparing Scaling Neural Networks to Human Brains 46:00 - The Way Forward 49:25 - Conclusion & Comments Paper: https://arxiv.org/abs/2009.06489 Website: https://hardwarelottery.github.io/ Abstract: Hardware, systems and algorithms research communities have historically had different incentive structures and fluctuating motivation to engage with each other explicitly. This historical treatment is odd given that hardware and software have frequently determined which research ideas succeed (and fail). This essay introduces the term hardware lottery to describe when a research idea wins because it is suited to the available software and hardware and not because the idea is superior to alternative research directions. Examples from early computer science history illustrate how hardware lotteries can delay research progress by casting successful ideas as failures. These lessons are particularly salient given the advent of domain specialized hardware which makes it increasingly costly to stray off of the beaten path of research ideas. Authors: Sara Hooker Links: YouTube: https://www.youtube.com/c/yannickilcher Twitter: https://twitter.com/ykilcher Discord: https://discord.gg/4H8xxDF BitChute: https://www.bitchute.com/channel/yannic-kilcher Minds: https://www.minds.com/ykilcher Parler: https://parler.com/profile/YannicKilcher LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/ If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this): SubscribeStar: https://www.subscribestar.com/yannickilcher Patreon:…