Debugging Tools
***************

Many tools lie at your disposal for debugging Pintos.  This appendix introduces you to a few of them.

printf()
========

Don't underestimate the value of printf().  The way printf() is implemented in Pintos, you can call it from practically anywhere in the kernel, whether it's in a kernel thread or an interrupt handler, almost regardless of what locks are held.

printf() isn't useful just because it can print data members. It can also help figure out when and where something goes wrong, even when the kernel crashes or panics without a useful error message.  The strategy is to sprinkle calls to print() with different strings (e.g. "1\n", "2\n", ...) throughout the pieces of code you suspect are failing.  If you don't even see 1 printed, then something bad happened before that point, if you see 1 but not 2, then something bad happened between those two points, and so on.  Based on what you learn, you can then insert more printf() calls in the new, smaller region of code you suspect. Eventually you can narrow the problem down to a single statement.

ASSERT
======

Assertions are useful because they can catch problems early, before they'd otherwise be notices.  Pintos provides a macro for assertions named ASSERT, defined in <debug.h>, that you can use for this purpose.  Ideally, each function should begin with a set of assertions that check its arguments for validity.  (Initializers for functions' local variables are evaluated before assertions are checked, so be careful not to assume that an argument is valid in an initializer.)  You can also sprinkle assertions throughout the body of functions in places where you suspect things are likely to go wrong.

When an assertion proves untrue, the kernel panics.  The panic message should help you to find the problem.  See the description of backtraces below for more information.

DEBUG
=====

The DEBUG macro, also defined in <debug.h>, is a sort of conditional printf().  It takes as its arguments the name of a ``message class'' and a printf()-like format string and arguments.  The message class is used to filter the messages that are actually displayed.  You select the messages to display on the Pintos command line using the -d option.  This allows you to easily turn different types of messages on and off while you debug, without the need to recompile.

For example, suppose you want to output thread debugging messages.  To use a class named thread, you could invoke DEBUG like this:

     DEBUG(thread, "thread id: %d\n", id);

and then to start Pintos with thread messages enable you'd use a command line like this:

     pintos run -d thread

Backtraces
==========

When the kernel panics, it prints a ``backtrace,'' that is, a summary of how your program got where it is, as a list of addresses inside the functions that were running at the time of the panic.  You can also insert a call to debug_backtrace(), prototyped in <debug.h>, at any point in your code.

The addresses in a backtrace are listed as raw hexadecimal numbers, which are meaningless in themselves.  You can translate them into function names and source file line numbers using a tool called i386-elf-addr2line. (If you're using an 80X86 system for development, it's probably just called addr2line.)

The output format of i386-elf-addr2line is not ideal, so we've supplied a wrapper for it simply called backtrace. Give it the name of your kernel.o as the first argument and the hexadecimal numbers composing the backtrace (including the 0x prefixes) as the remaining arguments.  It outputs the function name and source file line numbers that correspond to each address.

If the translated form of a backtrace is garbled, or doesn't make sense (e.g. function A is listed above function B, but B doesn't call A), then it's a good sign that you're corrupting a kernel thread's stack, because the backtrace is extracted from the stack. Alternatively, it could be that the kernel.o you passed to backtrace does not correspond to the kernel that produced the backtrace.

i386-elf-gdb
============

You can run the Pintos kernel under the supervision of the i386-elf-gdb debugger. (If you're using an 80X86 system for development, it's probably just called addr2line.)  There are two steps in the process.  First, start Pintos with the --gdb option, e.g. pintos --gdb run.  Second, in a second terminal, invoke gdb on kernel.o:

     i386-elf-gdb kernel.o

and issue the following gdb command:

     target remote localhost:1234

At this point, gdb is connected to Bochs over a local network connection.  You can now issue any normal gdb commands.  If you issue the c command, the Bochs BIOS will take control, load Pintos, and then Pintos will run in the usual way.  You can pause the process at any point with Ctrl+C.  If you want gdb to stop when Pintos starts running, set a breakpoint on main() with the command break main before c.

You can read the gdb manual by typing info gdb at a terminal command prompt, or you can view it in Emacs with the command C-h i.  Here's a few commonly useful gdb commands:

c
     Continue execution until the next breakpoint or until Ctrl+C is typed.

break FUNCTION
break FILENAME:LINENUM
break *ADDRESS
     Sets a breakpoint at the given function, line number, or address. (Use a 0x prefix to specify an address in hex.)

p EXPRESSION
     Evaluates the given C expression and prints its value. If the expression contains a function call, the function will actually be executed, so be careful.

l *ADDRESS
     Lists a few lines of code around the given address. (Use a 0x prefix to specify an address in hex.)

bt
     Prints a stack backtrace similar to that output by the backtrace program described above.

p/a ADDRESS
     Prints the name of the function or variable that occupies the given address. (Use a 0x prefix to specify an address in hex.)

If you notice unexplainable behavior while using gdb, there are three possibilities.  The first is that there is a bug in your modified Pintos.  The second is that there is a bug in Bochs's interface to gdb or in gdb itself.  The third is that there is a bug in the original Pintos code.  The first and second are quite likely, and you should seriously consider both.  We hope that the third is less likely, but it is also possible.

Modifying Bochs
===============

An advanced debugging technique is to modify and recompile the simulator.  This proves useful when the simulated hardware has more information than it makes available to the OS.  For example, page faults have a long list of potential causes, but the hardware does not report to the OS exactly which one is the particular cause. Furthermore, a bug in the kernel's handling of page faults can easily lead to recursive faults, but a ``triple fault'' will cause the CPU to reset itself, which is hardly conducive to debugging.

In a case like this, you might appreciate being able to make Bochs print out more debug information, such as the exact type of fault that occurred.  It's not very hard.  You start by retrieving the source code for Bochs 2.1.1 from http://bochs.sourceforge.net and extracting it into a directory.  Then read pintos/src/misc/bochs-2.1.1.patch and apply the patches needed. Then run ./configure, supplying the options you want (some suggestions are in the patch file).  Finally, run make. This will compile Bochs and eventually produce a new binary bochs.  To use your bochs binary with pintos, put it in your PATH, and make sure that it is earlier than /usr/class/cs140/i386/bochs.

Of course, to get any good out of this you'll have to actually modify Bochs.  Instructions for doing this are firmly out of the scope of this document.  However, if you want to debug page faults as suggested above, a good place to start adding printf()s is BX_CPU_C::dtranslate_linear() in cpu/paging.cc.