I'm a novice graphic designer and a tabletop game amateur writer/reviewer. As such, I come across a lot of samples of work, and I get to see a lot of what people do, and I've got some pretty big pet peeves when it comes to typesetting. Here's my advice on how to avoid making blunders when releasing a game.
I love when people include images in their game, it's wonderful. However, sometimes it just detracts from the feel of the piece. There's a few things I see that really draw away from their intentions.
Modern digital printing and PDF files allow a great versatility in images, including images with transparency. If you have a background that's colored, and an image with a white background box, cut off the white background box with a program like Photoshop or the GIMP, or get someone to do it for you. If this is too difficult, put it in a bordered frame so that your image doesn't directly drop off into the background. If this is also too difficult, at least convert the file to PNG and blur around the edges so that it smoothly fades off, rather than just being a sharp line. Ideally, if you have to include an image you should make it either feel like part of the text (see Eclipse Phase or its free Quick Start Guide for great examples of this) as an exposition, or you should make it be an obvious inclusion (i.e. paper edges like on Polaroid projects, preferably written on) in such a way that it fits into the text (Outbreak: Undead does this chillingly well, and The Last Band Advanced Player's Guide should contain examples of this concept with text).
It helps if your font is legible. I'm guilty of occasionally overlooking this, since I have a pretty good time with most fonts, but you definitely want to get a couple other people to take a look at the fonts before you publish. I ran into some trouble with this with The Last Band for Game Chef 2012, so you can see an example of what not to do.
Use a creative font. I'm not talking about being super duper crazy, but if you're using a setting appropriate font your game will be that much more immersive. For instance, when writing stuff for Aduelle I use Timeless for the body text since its difference from normal Times New Roman which I would otherwise have used creates a slight sense of mystery, and the very eye-catching Optimus Princeps for the title. I use Aquilline Two for anything "handwritten", both to give notes more credibility, though its legibility is a slight hit. All of these fonts are by Manfred Klein, which are free to use for any purpose with the clause that you're supposed to donate some profits you make through commercial use, meaning that I'm in no way in danger of violating the licensing even if I forget to credit him. Optimus Princeps is also what I've been using for my new logo (see the bottom of the sidebar).
Backgrounds are a great way to engage your reader. I remember Eclipse Phase the first time I read it, and its background is basically the graphic design equivalent of Paradise. It works well, and flows great. Plus, it's really cool and engaging without detracting from the text. I can read Eclipse Phase on my old e-ink Kindle without zooming in, meaning that I can read a full document sized page on the significantly smaller paperback-style display. That's how well it's formatted. This is because it does something smart, and puts the beautiful elements of the background in places that don't interfere with text. It also is setting appropriate, going for a information-overload style to go with the sci-fi transhumanism of the game, without becoming so busy as to detract from the text.
It's okay to include additional side-information, but delineate it or your reader is going to get confused pretty quickly. I can't remember what I was reading, but there was something where the sidebars were just offset by a small border and I missed that border and got caught up on a sidebar before reading the appropriate part of the page. Make sidebars have their own background, or a pretty noticeable border with a dedicated heading (Eclipse Phase, again, does this pretty well).
Tables of contents and indexing are nice. While you can use PDF bookmarks if you're going 100% digital, less adept users may not know how to use them, and even I, despite using them left and right on all sorts of things, still go for an end-book index. I've made a tutorial on how to do this in Scribus, and the way it works is that you can basically link to the page and even the specific part of the page if your PDF editor supports it (Scribus is how I make my PDF's, it's free and open source and will do this pretty simply). Index a ton of stuff, if you haven't indexed something on each page you'd better have full page art on those pages. The only exception I'd say that you should make for this is when you've got either a small team and don't have the time and resources before publishing, or you only have a very short book (10 or fewer pages).
If you mention something that's in another book, e.g. a supplement you wrote, be sure to include a reference to the book and page if you think there's any chance whatsoever of someone wanting to look it up. I've had to look through a couple books to try to find stats for a specific mentioned critter in one supplement, and even with PDF searching I had to resort to a character creator for the same game which has a great and accurate index of books and page numbers to find it, since it was in the last place I would have looked. Don't do that. If you really, really, don't want to do this, include necessary information in a sidebar.