3D modeling is the process of developing a mathematical representation of any surface of an object (either inanimate or living) in three dimensions via specialized software. 3D modeling takes a lot of math to do if you want to animate the model.
Linear Algebra - Just plain linear algebra can take you a long way. You can do basic modeling, rasterization, lighting, collision detection etc. without knowing much more than linear algebra
Splines - These are used heavily in 3D modeling to create more natural looking surfaces. Spline interpolation is also used in animation, procedural generation and rasterization
Signal Processing - This is used in more places than you would think. Anti-aliasing is the most well known, but it's also used in various other applications like wavelet transforms, which are used to do fast global illumination
Radiometry - This isn't strictly a branch of mathematics, but understanding how light interacts with surfaces and media is extremely useful in doing realistic rendering. Much of graphics research is based on approximating light's behavior with limited computing power
Differential Equations - If you want to simulate any natural phenomenon, such as fluids, crowds, fire, participating media or even sound, you will end up having to solve differential equations that govern these phenomena