In 1999 I wrote the first draft of How to Think Like a Computer Scientist, an introduction to programming using Java. I published it under a free license (the GNU Free Documentation License). Jeff Elkner, a high school teacher in Virginia, picked up the book and translated it from Java to Python. He sent me his modified version, and I learned Python by reading my own book. That very strange experience was the first hint I got that a free book is not just a conventional book with a different license; it is a fundamentally different product.
Since then I have written several other books. All of them are available under free licenses from Green Tea Press. My process for writing books goes something like this:
I usually write a first draft before teaching a class. I publish the draft immediately, and distribute it to my students. During the semester I make notes about things I want to change, and my students tell me about mistakes, rough spots, etc.
I try to revise the book as soon as possible after the semester, while the ideas are fresh in my head.
By the time I publish the revised edition, I have heard from other readers (not just my students) and I have a sense of whether there is a broader market for the book. If so, then I might publish the book myself or approach another publisher.
The advantage of publishing it myself is that it is faster, and I don't have to argue with the publisher about the license.
The advantage of mainstream publishers is that they can sell more copies. I have worked with Cambridge University Press (Python for Software Design) and O'Reilly Media (Think Stats, Think Complexity, Think Python).
I have heard from a number of people interested in publishing their own free books. I wrote some general advice about it several years ago. But here are answers to some more concrete questions.
If you already know the difference between a copyright and a license, you can skip this section. But some people (including one of my editors) are confused, so here is a quick overview. I am not a lawyer; this is not legal advice.
If you write a book, you own the copyright, which means that you can decide who can do what with your book. It is conventional to put a copyright notice in the book that indicates the date of first publication and the copyright holder.
A license is a description of who can do what with the book. The most restrictive license, which is unfortunately the most common, is "All rights reserved," which means that only the copyright holder is allowed to distribute the book or create modified version.
The GNU Free Documentation License is a "free license" because it allows anyone to copy, sell and distribute the book, and create and distribute modified versions (under the same license).
Creative Commons provides a number of different licenses that grant and reserve different rights. Most of my books are now under the Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0), which allows users to redistribute the book and modified versions, which attribution, but not for commercial purposes.
But here's one important point: as the copyright holder, you can grant different licenses to different people. So by default, I grant the CC license to everyone, but I also have the option to grant a less restrictive license to someone who might be interested in developing a commercial derivative version.
Before you start thinking about selling hard copies, I recommend that you start with electronic distribution.
While you are developing the book, get it under version control. Most of my books are hosted on Google Code (for example, here's the repository for Think Python). Anyone who wants to make a modified version of the book can check out the repository, make a branch, and build their own version. I keep the supporting code for the book in the same repo.
As soon as you are comfortable releasing a draft, create a web page for it. I post my books at Green Tea Press, which is hosted by DreamHost. That way I can build a book on my computer and then copy over the files using rsync.
For most of my books, I also have a Facebook page (for example, here's the page for Think Python). You can post news about the book there, and engage with readers.
Facebook is one cheap kind of publicity, but you should start thinking about other ways to get the word out. Find online communities that will be interested and let them know about the book. Think about writing a blog with articles related to the book. For example, I started writing Probably Overthinking It to publicize Think Stats.
For hard copies, there are lots of short-run printers you can use. I use Lulu.com quite a lot, and I have generally been very happy. You can upload a book and get good quality printed copies in a few days for a reasonable price. If you want other people to buy your book, they can order it from Lulu and you get a share of the cover price.
For a fee, Lulu can make your book available through Amazon and other booksellers. Depending on how well you publicize the book, that might or might not be worth it.