In an age when your company mandates you to raise your productivity right now with hundreds of percentage points using LLMs, how do you find an excuse to sit down and read a book?
Increasingly (for instance ADSP podcast [1]) those in nvidia's inner circle are advocating against writing your own CUDA kernels. (Unless that's your full time job at nvidia, that is).
It’s not about whether you work at Nvidia. Avoid writing CUDA kernels if there are higher level libraries that do what you need. Do write CUDA kernels if you want to learn how, or if you need the low level control, or to micro-optimize. Being able to fuse kernels to avoid memory traffic or get better specialization is also a reason to reach for raw CUDA. Just consider what’s the right tool for the job…
That would be cool but nvidia released blackwell and still have not released unbroken kernels for sm120. Sm120 is not the data center gpu, so it doesn't get its love. So we can't depend on nvidia to do the right thing is my point unfortunately
That advice seems like nonsense. It's like saying avoid C because you can use Python, or avoid writing a graphics engine because you can license Unreal.
can very much agree about not writing stuff like reductions yourself, unless you have good reason to.
but this sort of feels like another "implement everything with <nvidia stuff> and you'll have a great time!! (but also coincidentally get locked in even more to Nvidia hardware)"
I started learning about GPU and CUDA from this book recently, and I agree the writing is confusing, and code examples have errors. However, it is still a nice reference about many types of algorithms for heterogeneous memory devices, it helped me understand better some patterns for CPUs.
Regarding the section on Python and high-level CUDA, anyone interested should maybe first take a peek at Warp, which I’m guessing is too new to have a book yet. Warp lets you write CUDA kernels directly in Python, and it’s a breeze to get started. https://github.com/nvidia/warp
Having read or at least skimmed most of those books, I think the best intro is 'CUDA Programming: A Developer's Guide to Parallel Computing with GPUs'
Massively Parallel Processors: A Hands-on Approach is not really good in my opinion, many small mistakes and confusing sentences (even when you know cuda).
CUDA by Example: An Introduction to General-Purpose GPU Programming is too simple and abstract too much the architecture.
Next year I'm planning to start writing a cuda book that starts by engineering the hardware, and goes up to the optimization part on that harware (which is basically a nvidia card) including all the main algorithms (except for graphs).
I'm already teaching the course in this way at uni, and it is quite successful among students.
Not really, Hardware didn't really change that much, of course you'll not find Tensor or raytracing cores, but you will have a very solid grasp of gpu programming and the cuda language (that didn't change that much either), and then you can easily learn those more modern things with blog posts or even, at worst, chatgpt.
I really wish there were better options to PMPP... It's by far the most up-to-date book, but I totally agree the writing is sort of bad and some of the code examples are straight up incorrect.
So tl;dr, you have at least one person who would pay for a better book :-)
Probably worth noting that writing performant kernels for modern Nvidia hardware looks almost nothing like what the books from 2012 are going to teach you. You can read them for fun if you'd like but they're basically irrelevant.
phoronixrly | 12 hours ago
q8zd3 | 12 hours ago
phoronixrly | 10 hours ago
canyp | 3 hours ago
fileeditview | 11 hours ago
hartator | 11 hours ago
mohamedkoubaa | 5 hours ago
chrsw | 10 hours ago
zparky | 10 hours ago
lacedeconstruct | 7 hours ago
juvoly | 10 hours ago
[1] https://adspthepodcast.com/2024/08/30/Episode-197.html
dahart | 9 hours ago
saagarjha | 43 minutes ago
halJordan | 8 hours ago
drnick1 | 6 hours ago
bobmarleybiceps | 5 hours ago
pwython | 9 hours ago
synergy20 | 8 hours ago
cdavid | an hour ago
I started learning about GPU and CUDA from this book recently, and I agree the writing is confusing, and code examples have errors. However, it is still a nice reference about many types of algorithms for heterogeneous memory devices, it helped me understand better some patterns for CPUs.
dahart | 9 hours ago
somethingsome | 8 hours ago
Massively Parallel Processors: A Hands-on Approach is not really good in my opinion, many small mistakes and confusing sentences (even when you know cuda).
CUDA by Example: An Introduction to General-Purpose GPU Programming is too simple and abstract too much the architecture.
Next year I'm planning to start writing a cuda book that starts by engineering the hardware, and goes up to the optimization part on that harware (which is basically a nvidia card) including all the main algorithms (except for graphs).
I'm already teaching the course in this way at uni, and it is quite successful among students.
synergy20 | 8 hours ago
somethingsome | 6 hours ago
bobmarleybiceps | 6 hours ago
So tl;dr, you have at least one person who would pay for a better book :-)
Aurornis | an hour ago
I always appreciate book lists like this one, but having a small targeted list is more practical for those of us with limited reading time.
iamcreasy | 36 minutes ago
What makes CUDA Programming: A Developer's Guide to Parallel Computing with GPUs better among its peers?
boomzilla | 12 minutes ago
https://docs.nvidia.com/cuda/cuda-programming-guide/pdf/cuda...
brcmthrowaway | 8 hours ago
fwx | 4 hours ago
fwx | 5 hours ago
SkiFreeWin3 | 3 hours ago
In this day and age when programming is so accessible, why not have a more tempting pitch than just book titles categorized by difficulty.
saagarjha | 42 minutes ago
wces | 32 minutes ago
Understand everything he talks about and you understand CUDA.