Graduate students tend to TA or teach one or two classes at a time--often multiple sections of the same class. But you’re right, the actual ‘in class’ time is not 40 hours. Nonetheless, class prep takes a good amount of time, as well as grading, which, especially in a writing intensive discipline such as philosophy, takes a lot of time. Often, grad students can have up to 100 students at a time.
Faculty can teach anywhere from one course to seven, and this comes with it’s own prep time and grading. Similarly, faculty can have, in theory, hundreds of students. Sometimes--but not often, and only at large research programs--they will have TAs to assist them.
Setting aside this big chunk of time related to teaching, both graduate students and faculty spend much of the remainder of their time researching, writing, and publishing (see above). This means a lot of reading, a lot of writing, a lot of thinking, some talking to other researchers, taking notes, going to conferences where people present their work, and so on. Writing a paper is not just a matter of thinking up an idea and then writing it down, as when you may have taken a class in high school or college and had to write a book report or some such. We must have a good understanding of what other people have said, situate our project within that context, and this involves many many revisions and sending drafts off to others for feedback.
There are also various other administrative duties that both graduate students and faculty must undertake, such as serving on committees dedicated to organizing curricula, student clubs, hiring new faculty members, reporting to higher up administrative people, and a lot of other various and not very exciting but required work.