The debugger we will use for this course is called gdb. For the most part, debugging your kernel will be no different from debugging any other program with gdb. There is a small caveat, however. Consider the following: the program that is actually running on your virtual machine will be System/161 (the executable will be called sys161). You don't want to debug System/161. You want to debug what is being run on System/161. Thus, a little cleverness is required.
You will have gdb communicate with System/161 through a Unix socket in order to send and receive debugging information. Thus, debugging your kernel generally takes three steps (after you've built your kernel, give this a try):
In your installation root directory (~/os161/root/), launch System/161 but have it wait for a debugger connection:
sys161 -w kernel
In a different terminal window, again in your installation root directory, launch os161-gdb, giving it the binary that you want to debug:
os161-gdb kernel
Next, tell gdb where to find the sources for your kernel, so you know where you are in the source as you step through the code.
(gdb) directory ~/os161/src/kern/compile/DUMBVM
Hint: you can put this line in the .gdbinit file in your ~/os161/root directory, so you don't have to perform this step every time.
Once in gdb, instruct gdb to connect to System/161:
(gdb) target remote unix:.sockets/gdb
Take a look at this sample .gdbinit file. You can place a file like this in ~/os161/root/.gdbinit (or wherever your os161 root directory is) and simply use the commands defined in that file. For instance, once that file is in place, instead of typing the "target remote..." line above, you would simply type "db". That file also contains many useful commands for printing resizable arrays.
Now that you are connected, carefully study this document to understand what gdb can do for you.