(The difference is in the relative number of fraction bits, which give you precision, and exponent bits, which give you range. Google even came up with its own version called bfloat16. (The energy needed for multiplication is proportional to the square of the number of bits, Dally explained.) So, with the P100, Nvidia cut that number in half, using FP16. The clear advantage of doing this is that the logic that does machine learning’s key computation-multiply and accumulate-can be made faster, smaller, and more efficient if they need to process fewer bits. Defined by the IEEE 754 standard, these are 32 bits long, with 23 bits representing a fraction, 8 bits acting essentially as an exponent applied to the fraction, and one bit for the number’s sign.īut machine-learning researchers were quickly learning that in many calculations, they could use less precise numbers and their neural network would still come up with answers that were just as accurate. Before the P100, Nvidia GPUs represented those weights using single precision floating-point numerals. One such parameter is weights-the strength of neuron-to-neuron connections in a model-and another is activations-what you multiply the sum of the weighted input at the neuron to determine if it activates, propagating information to the next layer. These numbers represent the key parameters of a neural network. “By and large, the biggest gain we got was from better number representation,” Dally told engineers. Nvidia chief scientist Bill Dally summed up how Nvidia has boosted the performance of its GPUs on AI tasks a thousandfold over 10 years. Put it all together and you get what Dally called Huang’s Law (for Nvidia CEO Jensen Huang). Moore’s Law was a surprisingly small part of Nvidia’s magic and new number formats a very large part. How did Nvidia get here? The company’s chief scientist, Bill Dally, managed to sum it all up in a single slide during his keynote address to the IEEE’s Hot Chips 2023 symposium in Silicon Valley on high-performance microprocessors last week. The company has managed to increase the performance of its chips on AI tasks a thousandfold over the past 10 years, it’s raking in money, and it’s reportedly very hard to get your hands on its newest AI-accelerating GPU, the H100.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |