Many advance Linux  users prefer to use it in runlevel 3 and use the raw power of linux and do not want to go into the X server mode (runlevel 5).

The normal way out was to make changes in the /etc/inittab file to run the system in full multi-user mode. Things have changed now as we are moving to the Upstart's event based daemon from the Sys V init daemon system.

The way out which works for me is to remove the symlink of the gdm (Gnome Desktop Manager) or kdm (KDE Desktop Manager) programs from /etc/init.d folder. This will prevent starting of X-server.

Whenever you want to work in graphics mode, issue the command 

avinesh@avinesh-desktop:~$ sudo /etc/init.d/gdm start

                         or 

avinesh@avinesh-desktop:~$ sudo /etc/init.d/kdm-kde4 start

Hope you find this helpful!