Sunday, May 8, 2011

Reflections on Teaching GPU Programming and Architecture

I just finished teaching CIS 565: GPU Programming and Architecture, a graduate-level course on programming GPUs using mostly GLSL and CUDA, at the University of Pennsylvania. Being my first time teaching, I gained a lot of valuable insights, which I will share here. These ideas are on teaching computer science in general, not necessarily about the GPU.

For eye candy, I sprinkled screenshots of our student projects throughout this post.

1. Exams have a hidden agenda.

At least my exams. On the surface, exams are used to evaluate students, and ultimately assign grades. Historical, this course never had exams. Given its pragmatic, hands-on material, grades were primarily based on coding assignments and a final project.

Screen Space Fluid Rendering - Leftheris Kaleas
This year I added a final - not because I wanted to evaluate students on their ability to think about the GPU under pressure, but because I wanted them to study for the final. It is a valuable exercise to go over material a second time, especially after we've had hands-on experience, to gain new insights. For example, the performance implications of register pressure will seem trivial once the student has some coding experience, but may have been foreign when first introduced. Likewise, going back and looking at GLSL after learning CUDA, will lead to a deeper understanding of GLSL.

In general, one reason people go to grad school is to gain a deeper understanding of topics they already have some exposure to. Studying for a final has the same effect on a more local scale.

So if I am only interested in students studying for a final, but not so much in how they do, what did I do? Two things. First, I made the final only worth 10% of the semester grade. That is not enough to ruin someone's grade if they perform poorly, but it can be the difference between an A and a B. The second technique I use is a bit more interesting...

2. Crowdsourcing the final is fun.

To help students study, I requested they make a ten question exam (and answer it!) with questions of different types, e.g., short answer, coding, etc., and of different difficulties, e.g., easy, medium, hard, and challenge. I made this "take-home" portion of their final worth 10%. It is a good studying technique to get in the instructor's head and think about what they are going to ask. I also joked that I might take all of their exams in the time they take my final.

After I handed out the final, I told the class that their exams were so good and hard that I wouldn't be able to take them while they took my final. In fact, I didn't even make their final - they did. Every single question on their final was from one of their exams. The class got a big kick out of it.

When looking through the student-made exams, I was simply amazed at the collective breadth and depth of knowledge of the class. It was also interesting to see different students with different specialties - each exam covered diverse topics, but some where biased towards GPU architecture, while others were heavy on GPU compute, while others favored rendering. Overall, the quality of their questions were quite good. Some of the exams were really hard too - harder than I made the final.

Am I nervous that a student taking this course in the future will read this post or hear from previous students how the final is made? Maybe the entire class will self-organize and send their exams to each other before the final. If a student reviewed every exam, they would know every question that will be on the final. Is that cheating? I don't know, but students rigorously studying for a final sounds like a great problem to have. After all, the main point of my final is to have them revisit the material.

Mobile Depth of Field as Post-Processing using
OpenGL ES 2.0 - Han Li and Qing Sun

3. Student confidence is important.

Perhaps this is my most controversial idea...

Fast Human Detection on GPU - Fan Deng
Some courses have notoriously difficult exams and homeworks. The mean score may be in the 50s, 40s, or even lower, and the standard deviation is large indicating that some students get it, or at least can handle difficult problems, and some students don't. This is common in algorithms courses across universities.

I believe the rational is challenging students to push their limits and think rigorously ultimately leads to a deeper understanding of the material. At the least, it leads to a certain respect for the subject.

I think this approach has some merit, but I am on the other end of the extreme. I want to see a mean around 90% with a tight standard deviation, indicating that everyone is getting it for the most part. In practice, I saw means from 75 to 94 and standard deviations from 4 to 33.

A higher mean and tighter standard deviations means I didn't make every question incredibly difficult. I like to build the student's confidence by allowing them to answer most questions if they have a solid understanding of the material. I want them to finish the course and feel that they can program CUDA in a production environment. I don't want them to think "I couldn't do the CUDA assignments in class; there is no way I can program GPUs in the real world."

With that said, this doesn't imply that I make homework problems trivial. Have a look for yourself. In future semesters, I will continue to add harder questions, but perhaps weight them lower.

Regardless of difficulty, it is actually hard to answer most of my questions by looking at the slides alone, which leads to our next topic...

4. Build an interactive classroom.

Tile-based Techniques in Deferred Shading - Sean Thomas
With only 12 registered students, plus some lurkers as they are called, we were very successful in building an interactive culture in the classroom. In an average 80 minute lecture, there might be 20-60 student comment/questions. The best situation is when students start talking to each other and I largely remove myself from the discussion and just facilitate.

To create an interactive culture, my slides rarely have answers, but often have questions. I start each topic with a question. For example, when introducing memory coalescing, I ask if given a matrix in row-major order, is it better for a thread to access elements column by column or row by row? Some questions are quite fun because the answer for a CPU and GPU are different.

I think I got lucky with an interactive classroom. I didn't realize it until Milo pointed it out, but students in my class self-selected; they were there because they wanted to learn about the GPU, not because it was a required course. They could have taken any number of courses instead. The other thing that helped with interaction was the class size. With 12 students, I knew all of them reasonably well, and many of them knew each other. I imagine it is a significant challenge to maintain interaction as class size increases.

5. Teaching at the graduate level requires depth of knowledge.

There is, of course, a downside to an interactive classroom: hard questions. Students are looking for deep discussion beyond the surface of the slides.

Efficient Real-Time Shadow Rendering - Stuart Summerfield
I'm a graphics guy, so I could handle most graphics questions with reasonably deep answers, usually from battle-tested experience.

I'm also pretty familiar with the compute side of GPU programming, but I don't have the same day-to-day experience with it, so fielding some compute questions was hard. I'm embarrassed to admit that one lecture early in the semester I was asked if the warp size is 32 threads and an SM (streaming multiprocessor) only has 8 SPs (stream processor) on the G80, how is the warp executed? Good question. Actually, great question. It even appeared on our final. At the time I didn't know.

 I suggest two ways to handle questions you don't know.  First, say you don't know and turn the question back to the class. Sometimes someone will know. Second, write it down and cover it at the start of the next lecture. That is what I did for the above question and a handful of other questions throughout the semester.

The wrong way to answer these questions is to make something up.

6. Reusing other instructor's material is hard.

GPU-Accelerated Fluid Simulation - Kalin Gochev
I have a minor case of Not-Invented-Here Syndrome when it comes to code, but I have a major case when it comes to course material, particularly slides. I find it very difficult to teach off other people's slides. I want to present topics in a certain order, in a certain manner (usually with lots of questions), and I want certain examples (usually from my real-world experience). I want slides with lots of images and not a lot of text.

Even though this course was taught for five years prior, the vast majority of my lectures were new or changed significantly - not because the old course material was bad, actually, it was quite good, but because I wanted a certain style. This came at significant effort. I found, on average, preparing an 80 minute lecture requires 12 to 15 hours, assuming some background in the material. For example, even though I am familiar with GLSL, preparing a lecture is time consuming because I need to carefully craft the discussion and double check everything. Preparing a lecture on topics I know less about, e.g., Fermi, is even more time consuming.

I did reuse slides in one lecture though. Kayvon Fatahalian's presentation, From Shader Code to a Teraflop: How GPU Shader Cores Work is the best introduction to GPU architecture that I have ever seen - and it made teaching CUDA afterward much easier. Reusing the slides was still quite a bit of work; I watched Kayvon's talk on SIGGRAPH Encore a few times and went over his slides and notes several times before I felt comfortable giving the talk.  By the way, Kayvon's slides are from the Beyond Programmable Shading SIGGRAPH course, which is a great resource in general (2008, 2009, 2010).

7. Teaching takes a tremendous amount of time.

On the surface, teaching one class doesn't sound like much work. Teach two 1.5 hour lectures a week, hold another three hours of office hours, prepare lectures, and read and write some email. What is that? Maybe 10-20 hours a week? For me, it was more like 30-40 hours: 3 hours lecture + 3 office hours + 24-30 hours preparing lectures + 3-8 hours email.

Email, really? For this past semester, I have over 500 gmail conversations, each averaging between two and ten emails. I feel staying on top of email is a huge part of teaching well.

I'm under the impression that teaching a course for the second time is much less work. I suspect the 12-15 hour prep time drops to 1-4 hours depending on how much material you change. Also, as obvious as it sounds, make sure to create the homework/exam answers at the same time you make the questions!

There were a few weeks were I didn't have to prepare lectures because the student's gave presentations...

8. Student presentations are for building experience.

A tradition Joe Kider started a few years ago, and I continued, is for each student to use half a lecture to present a GPU-related topic. The motivation was to help them research their project, give them experience presenting, and provide the class with exposure to diverse GPU topics.
CBench: Analyzing Compute Performance for Modern
NVIDIA and AMD GPUs - Varun Sampath

As instructor, I helped provide the presentation's outline and gave the presenter immediate, detailed feedback. I also tried to ask a lot of questions during their talk. Depending on the topic, this can be challenging. If a student presents on a topic I know a lot about, e.g., WebGL, I will have insightful comments and perhaps even thought-provoking questions, but if they present on something I know very little about, e.g., computer vision on the GPU, I have much less interesting things to say, but I still try to absorb as much as I can by asking questions that are probably obvious to them.

Based on student feedback, their presentations were great for experience, but not necessarily for learning new material. There are, of course, exceptions like Varun's Larrabee presentation; he probably covered the material better than I would have.

9. Recruit and empower a killer teaching assistant.

Rendering Massive Game Worlds on a WebGL
 Based Browser - Krishnan Ramachandran
I had the pleasure of working with Jon McCaffrey, a bright masters student who is (by the time you read this) now at NVIDIA writing OpenGL drivers. As an avid OpenGL developer, it makes me feel good to know that people like Jon are writing the drivers I rely on.

I would say something like "let's do an assignment on SSAO" or "let's have the class code a globe with a gloss and bump map" to Jon, and suddenly his answer key would be coded over night, shortly followed by the assignment writeup. I don't think I would have gotten through the semester without his hard work. It's obvious to me that the TA plays a key role in a course; having a good one makes a world of difference. I owe a big thanks to Joe for recruiting Jon, and encouraging me to teach this course in the first place.

10. Teaching students with diverse backgrounds is hard.

Over the years, GPUs have evolved from fixed-function rasterization pipelines for graphics to fully-programmable massively-parallel processors. As such, this course covers both the graphics and compute side of GPU programming. Therefore, the course attracts computer engineering students who don't know the difference between a pixel and a fragment (well, they do now!) and computer graphics students who ask "Cycle? What's a cycle?" (I think he was joking though).

It is very hard to serve everyone well. This is an area I need to improve. More flexible "choose your own graphics or compute adventure" assignments will help, but ultimately, there needs to be two courses: a rendering course and a programming massively-parallel processors course.

11. Embrace openness.

I like to do things out in the open, so I encouraged my students to do the same. The result is many students open sourced their projects and all students had public project blogs. To be fair, the blogs were required for their projects, another excellent tradition Joe started a few years ago.  Many students told me they plan to continue their project and/or blog beyond the class. Great! The more open source software - the better. The more bright people sharing ideas via blogs - the better.

Radio Propagation Modeling via Fourier Optics - Cem Karan

12. Know how to measure your success.

I don't have the official university-given survey results yet, but when asked to rate "knowing what I know now, I still would have taken this course" from 1 to 10 on my in-class survey, the average answer was 9.25. Does this mean I was successful in teaching this course? On one level, perhaps.

But I really judge my success by the success of my students. One student came running into the lab so excited that I thought he won the lottery. Actually, he just received a job offer. He went on to tell me that a big part of his interview success was explaining how he could improve the performance of their cloth simulation by running it on the GPU with CUDA. I'll take that ten to one over an answer on a survey.

When we ended the semester I reminded students of one way to be successful - by immersing yourself with the best and brightest people you can find.

Full disclaimer: These are my own opinions and do not necessarily reflect those of the University of Pennsylvania or the Department of Computer and Information Science. My entire teaching experience is teaching this course once, so you shouldn't consider me an authority or even take me very seriously.


  1. Congratulations, great post!

    Personally, I teach civil engineering at BSc and MSc level. GLSL shadering is just my hobby, but I can use some of your ideas in my teaching experience.

    Thanks: Gergely Marko from Hungary

  2. Thanks Gergely. Some of these ideas were new this semester, like the student-made final and open source projects, but others, like the student presentations and project blogs, were inherited from other instructors. In fact, we still have the previous course websites up: 2010, 2009, 2008, 2007, and 2005. I was a student in the course in 2007.

  3. Well said and well done. From the students I talked with in your course, they certainly enjoyed it.

  4. Thanks for your support and encouragement this semester Milo. After seeing your guest lecture, I know I have a lot I can improve.


Note: Only a member of this blog may post a comment.