1 @node Debugging Tools, Development Tools, Project Documentation, Top
2 @appendix Debugging Tools
4 Many tools lie at your disposal for debugging Pintos. This appendix
5 introduces you to a few of them.
10 * Function and Parameter Attributes::
13 * Debugging by Infinite Loop::
19 @section @code{printf()}
21 Don't underestimate the value of @func{printf}. The way
22 @func{printf} is implemented in Pintos, you can call it from
23 practically anywhere in the kernel, whether it's in a kernel thread or
24 an interrupt handler, almost regardless of what locks are held (but see
25 @ref{printf Reboots} for a counterexample).
27 @func{printf} is useful for more than just examining data.
28 It can also help figure out when and where something goes wrong, even
29 when the kernel crashes or panics without a useful error message. The
30 strategy is to sprinkle calls to @func{print} with different strings
31 (e.g.@: @code{"<1>"}, @code{"<2>"}, @dots{}) throughout the pieces of
32 code you suspect are failing. If you don't even see @code{<1>} printed,
33 then something bad happened before that point, if you see @code{<1>}
34 but not @code{<2>}, then something bad happened between those two
35 points, and so on. Based on what you learn, you can then insert more
36 @func{printf} calls in the new, smaller region of code you suspect.
37 Eventually you can narrow the problem down to a single statement.
38 @xref{Debugging by Infinite Loop}, for a related technique.
41 @section @code{ASSERT}
43 Assertions are useful because they can catch problems early, before
44 they'd otherwise be noticed. Pintos provides the
45 @code{ASSERT}, defined in @file{<debug.h>}, for assertions.
46 Ideally, each function should begin with a set of
47 assertions that check its arguments for validity. (Initializers for
48 functions' local variables are evaluated before assertions are
49 checked, so be careful not to assume that an argument is valid in an
50 initializer.) You can also sprinkle assertions throughout the body of
51 functions in places where you suspect things are likely to go wrong.
52 They are especially useful for checking loop invariants.
54 When an assertion proves untrue, the kernel panics. The panic message
55 should help you to find the problem. See the description of
56 backtraces below for more information.
58 @node Function and Parameter Attributes
59 @section Function and Parameter Attributes
61 These macros defined in @file{<debug.h>} tell the compiler special
62 attributes of a function or function parameter. Their expansions are
66 Appended to a function parameter to tell the compiler that the
67 parameter might not be used within the function. It suppresses the
68 warning that would otherwise appear.
72 Appended to a function prototype to tell the compiler that the
73 function never returns. It allows the compiler to fine-tune its
74 warnings and its code generation.
78 Appended to a function prototype to tell the compiler to never emit
79 the function in-line. Occasionally useful to improve the quality of
80 backtraces (see below).
83 @defmac PRINTF_FORMAT (@var{format}, @var{first})
84 Appended to a function prototype to tell the compiler that the function
85 takes a @func{printf}-like format string as the argument numbered
86 @var{format} (starting from 1) and that the corresponding value
87 arguments start at the argument numbered @var{first}. This lets the
88 compiler tell you if you pass the wrong argument types.
94 When the kernel panics, it prints a ``backtrace,'' that is, a summary
95 of how your program got where it is, as a list of addresses inside the
96 functions that were running at the time of the panic. You can also
97 insert a call to @func{debug_backtrace}, prototyped in
98 @file{<debug.h>}, to print a backtrace at any point in your code.
100 The addresses in a backtrace are listed as raw hexadecimal numbers,
101 which are meaningless by themselves. You can translate them into
102 function names and source file line numbers using a tool called
103 @command{addr2line} (80@var{x}86) or @command{i386-elf-addr2line}
106 The output format of @command{addr2line} is not ideal, so
107 we've supplied a wrapper for it simply called @command{backtrace}.
108 Give it the name of your @file{kernel.o} as the first argument and the
109 hexadecimal numbers composing the backtrace (including the @samp{0x}
110 prefixes) as the remaining arguments. It outputs the function name
111 and source file line numbers that correspond to each address.
113 If the translated form of a backtrace is garbled, or doesn't make
114 sense (e.g.@: function A is listed above function B, but B doesn't
115 call A), then it's a good sign that you're corrupting a kernel
116 thread's stack, because the backtrace is extracted from the stack.
117 Alternatively, it could be that the @file{kernel.o} you passed to
118 @command{backtrace} does not correspond to the kernel that produced
121 Sometimes backtraces can be confusing without implying corruption.
122 Compiler optimizations can cause surprising behavior. For example, when
123 a function has called another function as its final action (a @dfn{tail
124 call}), the calling function may not appear in a backtrace at all.
127 * Backtrace Example::
130 @node Backtrace Example
133 Here's an example. Suppose that Pintos printed out this following call
134 stack, which is taken from an actual Pintos submission for the file
138 Call stack: 0xc0106eff 0xc01102fb 0xc010dc22 0xc010cf67 0xc0102319
139 0xc010325a 0x804812c 0x8048a96 0x8048ac8.
142 You would then invoke the @command{backtrace} utility like shown below,
143 cutting and pasting the backtrace information into the command line.
144 This assumes that @file{kernel.o} is in the current directory. You
145 would of course enter all of the following on a single shell command
146 line, even though that would overflow our margins here:
149 backtrace kernel.o 0xc0106eff 0xc01102fb 0xc010dc22 0xc010cf67
150 0xc0102319 0xc010325a 0x804812c 0x8048a96 0x8048ac8
153 The backtrace output would then look something like this:
156 0xc0106eff: debug_panic (../../lib/debug.c:86)
157 0xc01102fb: file_seek (../../filesys/file.c:405)
158 0xc010dc22: seek (../../userprog/syscall.c:744)
159 0xc010cf67: syscall_handler (../../userprog/syscall.c:444)
160 0xc0102319: intr_handler (../../threads/interrupt.c:334)
161 0xc010325a: ?? (threads/intr-stubs.S:1554)
167 (You will probably not get the same results if you run the command above
168 on your own kernel binary, because the source code you compiled from is
169 different from the source code that panicked.)
171 The first line in the backtrace refers to @func{debug_panic}, the
172 function that implements kernel panics. Because backtraces commonly
173 result from kernel panics, @func{debug_panic} will often be the first
174 function shown in a backtrace.
176 The second line shows @func{file_seek} as the function that panicked,
177 in this case as the result of an assertion failure. In the source code
178 tree used for this example, line 405 of @file{filesys/file.c} is the
182 ASSERT (file_ofs >= 0);
186 (This line was also cited in the assertion failure message.)
187 Thus, @func{file_seek} panicked because it passed a negative file offset
190 The third line indicates that @func{seek} called @func{file_seek},
191 presumably without validating the offset argument. In this submission,
192 @func{seek} implements the @code{seek} system call.
194 The fourth line shows that @func{syscall_handler}, the system call
195 handler, invoked @func{seek}.
197 The fifth and sixth lines are the interrupt handler entry path.
199 The remaining lines are for addresses below @code{PHYS_BASE}. This
200 means that they refer to addresses in the user program, not in the
201 kernel. If you know what user program was running when the kernel
202 panicked, you can re-run @command{backtrace} on the user program, like
203 so: (typing the command on a single line, of course):
206 backtrace grow-too-big 0xc0106eff 0xc01102fb 0xc010dc22 0xc010cf67
207 0xc0102319 0xc010325a 0x804812c 0x8048a96 0x8048ac8
210 The results look like this:
213 0xc0106eff: ?? (??:0)
214 0xc01102fb: ?? (??:0)
215 0xc010dc22: ?? (??:0)
216 0xc010cf67: ?? (??:0)
217 0xc0102319: ?? (??:0)
218 0xc010325a: ?? (??:0)
219 0x804812c: test_main (../../tests/filesys/extended/grow-too-big.c:20)
220 0x8048a96: main (../../tests/main.c:10)
221 0x8048ac8: _start (../../lib/user/entry.c:9)
224 Here's an extra tip for anyone who read this far: @command{backtrace}
225 is smart enough to strip the @code{Call stack:} header and @samp{.}
226 trailer from the command line if you include them. This can save you
227 a little bit of trouble in cutting and pasting. Thus, the following
228 command prints the same output as the first one we used:
231 backtrace kernel.o Call stack: 0xc0106eff 0xc01102fb 0xc010dc22
232 0xc010cf67 0xc0102319 0xc010325a 0x804812c 0x8048a96 0x8048ac8.
236 @section @command{gdb}
238 You can run the Pintos kernel under the supervision of the
239 @command{gdb} (80@var{x}86) or @command{i386-elf-gdb} (SPARC)
241 start Pintos with the @option{--gdb} option, e.g.@: @command{pintos
242 --gdb -- run mytest}. Second, in a separate terminal, invoke @command{gdb} (or
243 @command{i386-elf-gdb}) on
248 @noindent and issue the following @command{gdb} command:
250 target remote localhost:1234
253 (If the @command{target remote} command fails, then make sure that both
254 @command{gdb} and @command{pintos} are running on the same machine by
255 running @command{hostname} in each terminal. If the names printed
256 differ, then you need to open a new terminal for @command{gdb} on the
257 machine running @command{pintos}.)
259 Now @command{gdb} is connected to the simulator over a local
260 network connection. You can now issue any normal @command{gdb}
261 commands. If you issue the @samp{c} command, the simulated BIOS will take
262 control, load Pintos, and then Pintos will run in the usual way. You
263 can pause the process at any point with @key{Ctrl+C}. If you want
264 @command{gdb} to stop when Pintos starts running, set a breakpoint on
265 @func{main} with the command @code{break main} before @samp{c}.
267 You can read the @command{gdb} manual by typing @code{info gdb} at a
268 terminal command prompt, or you can view it in Emacs with the command
269 @kbd{C-h i}. Here's a few commonly useful @command{gdb} commands:
273 Continues execution until @key{Ctrl+C} or the next breakpoint.
275 @item break @var{function}
276 @itemx break @var{filename}:@var{linenum}
277 @itemx break *@var{address}
278 Sets a breakpoint at the given function, line number, or address.
279 (Use a @samp{0x} prefix to specify an address in hex.)
281 @item p @var{expression}
282 Evaluates the given C expression and prints its value.
283 If the expression contains a function call, that function will actually
286 @item l *@var{address}
287 Lists a few lines of code around the given address.
288 (Use a @samp{0x} prefix to specify an address in hex.)
291 Prints a stack backtrace similar to that output by the
292 @command{backtrace} program described above.
294 @item p/a @var{address}
295 Prints the name of the function or variable that occupies the given
297 (Use a @samp{0x} prefix to specify an address in hex.)
299 @item diassemble @var{function}
300 Disassembles the specified @var{function}.
303 If you notice other strange behavior while using @command{gdb}, there
304 are three possibilities: a bug in your
305 modified Pintos, a bug in Bochs's
306 interface to @command{gdb} or in @command{gdb} itself, or
307 a bug in the original Pintos code. The first and second
308 are quite likely, and you should seriously consider both. We hope
309 that the third is less likely, but it is also possible.
311 You can also use @command{gdb} to debug a user program running under
312 Pintos. Start by issuing this @command{gdb} command to load the
313 program's symbol table:
315 add-symbol-file @var{program}
318 where @var{program} is the name of the program's executable (in the host
319 file system, not in the Pintos file system). After this, you should be
320 able to debug the user program the same way you would the kernel, by
321 placing breakpoints, inspecting data, etc. Your actions apply to every
322 user program running in Pintos, not just to the one you want to debug,
323 so be careful in interpreting the results. Also, a name that appears in
324 both the kernel and the user program will actually refer to the kernel
325 name. (The latter problem can be avoided by giving the user executable
326 name on the @command{gdb} command line, instead of @file{kernel.o}.)
328 @node Debugging by Infinite Loop
329 @section Debugging by Infinite Loop
331 If you get yourself into a situation where the machine reboots in a
332 loop, that's probably a ``triple fault.'' In such a situation you
333 might not be able to use @func{printf} for debugging, because the
334 reboots might be happening even before everything needed for
335 @func{printf} is initialized. In such a situation, you might want to
336 try what I call ``debugging by infinite loop.''
338 What you do is pick a place in the Pintos code, insert the statement
339 @code{for (;;);} there, and recompile and run. There are two likely
344 The machine hangs without rebooting. If this happens, you know that
345 the infinite loop is running. That means that whatever caused the
346 reboot must be @emph{after} the place you inserted the infinite loop.
347 Now move the infinite loop later in the code sequence.
350 The machine reboots in a loop. If this happens, you know that the
351 machine didn't make it to the infinite loop. Thus, whatever caused the
352 reboot must be @emph{before} the place you inserted the infinite loop.
353 Now move the infinite loop earlier in the code sequence.
356 If you move around the infinite loop in a ``binary search'' fashion, you
357 can use this technique to pin down the exact spot that everything goes
358 wrong. It should only take a few minutes at most.
360 @node Modifying Bochs
361 @section Modifying Bochs
363 An advanced debugging technique is to modify and recompile the
364 simulator. This proves useful when the simulated hardware has more
365 information than it makes available to the OS. For example, page
366 faults have a long list of potential causes, but the hardware does not
367 report to the OS exactly which one is the particular cause.
368 Furthermore, a bug in the kernel's handling of page faults can easily
369 lead to recursive faults, but a ``triple fault'' will cause the CPU to
370 reset itself, which is hardly conducive to debugging.
372 In a case like this, you might appreciate being able to make Bochs
373 print out more debug information, such as the exact type of fault that
374 occurred. It's not very hard. You start by retrieving the source
375 code for Bochs 2.1.1 from @uref{http://bochs.sourceforge.net} and
376 extracting it into a directory. Then read
377 @file{pintos/src/misc/bochs-2.1.1.patch} and apply the patches needed.
378 Then run @file{./configure}, supplying the options you want (some
379 suggestions are in the patch file). Finally, run @command{make}.
380 This will compile Bochs and eventually produce a new binary
381 @file{bochs}. To use your @file{bochs} binary with @command{pintos},
382 put it in your @env{PATH}, and make sure that it is earlier than
383 @file{/usr/class/cs140/`uname -m`/bochs}.
385 Of course, to get any good out of this you'll have to actually modify
386 Bochs. Instructions for doing this are firmly out of the scope of
387 this document. However, if you want to debug page faults as suggested
388 above, a good place to start adding @func{printf}s is
389 @func{BX_CPU_C::dtranslate_linear} in @file{cpu/paging.cc}.
394 The page allocator in @file{threads/palloc.c} and the block allocator in
395 @file{threads/malloc.c} both clear all the bytes in pages and blocks to
396 @t{0xcc} when they are freed. Thus, if you see an attempt to
397 dereference a pointer like @t{0xcccccccc}, or some other reference to
398 @t{0xcc}, there's a good chance you're trying to reuse a page that's
399 already been freed. Also, byte @t{0xcc} is the CPU opcode for ``invoke
400 interrupt 3,'' so if you see an error like @code{Interrupt 0x03 (#BP
401 Breakpoint Exception)}, Pintos tried to execute code in a freed page or
404 An assertion failure on the expression @code{sec_no < d->capacity}
405 indicates that Pintos tried to access a file through an inode that has
406 been closed and freed. Freeing an inode clears its starting sector
407 number to @t{0xcccccccc}, which is not a valid sector number for disks
408 smaller than about 1.6 TB.