1 @node Debugging Tools, Development Tools, Project Documentation, Top
2 @appendix Debugging Tools
4 Many tools lie at your disposal for debugging Pintos. This appendix
5 introduces you to a few of them.
11 * UNUSED NO_RETURN NO_INLINE PRINTF_FORMAT::
14 * Debugging by Infinite Loop::
20 @section @code{@code{printf()}}
22 Don't underestimate the value of @func{printf}. The way
23 @func{printf} is implemented in Pintos, you can call it from
24 practically anywhere in the kernel, whether it's in a kernel thread or
25 an interrupt handler, almost regardless of what locks are held.
27 @func{printf} isn't useful just because it can print data members.
28 It can also help figure out when and where something goes wrong, even
29 when the kernel crashes or panics without a useful error message. The
30 strategy is to sprinkle calls to @func{print} with different strings
31 (e.g.@: @code{"1\n"}, @code{"2\n"}, @dots{}) throughout the pieces of
32 code you suspect are failing. If you don't even see @code{1} printed,
33 then something bad happened before that point, if you see @code{1}
34 but not @code{2}, then something bad happened between those two
35 points, and so on. Based on what you learn, you can then insert more
36 @func{printf} calls in the new, smaller region of code you suspect.
37 Eventually you can narrow the problem down to a single statement.
40 @section @code{ASSERT}
42 Assertions are useful because they can catch problems early, before
43 they'd otherwise be notices. Pintos provides a macro for assertions
44 named @code{ASSERT}, defined in @file{<debug.h>}, that you can use for
45 this purpose. Ideally, each function should begin with a set of
46 assertions that check its arguments for validity. (Initializers for
47 functions' local variables are evaluated before assertions are
48 checked, so be careful not to assume that an argument is valid in an
49 initializer.) You can also sprinkle assertions throughout the body of
50 functions in places where you suspect things are likely to go wrong.
52 When an assertion proves untrue, the kernel panics. The panic message
53 should help you to find the problem. See the description of
54 backtraces below for more information.
59 The @code{DEBUG} macro, also defined in @file{<debug.h>}, is a sort of
60 conditional @func{printf}. It takes as its arguments the name of a
61 ``message class'' and a @func{printf}-like format string and
62 arguments. The message class is used to filter the messages that are
63 actually displayed. You select the messages to display on the Pintos
64 command line using the @option{-d} option. This allows you to easily
65 turn different types of messages on and off while you debug, without
66 the need to recompile.
68 For example, suppose you want to output thread debugging messages. To
69 use a class named @code{thread}, you could invoke @code{DEBUG} like
72 DEBUG(thread, "thread id: %d\n", id);
75 and then to start Pintos with @code{thread} messages enable you'd use
76 a command line like this:
81 @node UNUSED NO_RETURN NO_INLINE PRINTF_FORMAT
82 @section UNUSED, NO_RETURN, NO_INLINE, and PRINTF_FORMAT
84 These macros defined in @file{<debug.h>} tell the compiler special
85 attributes of a function or function parameter. Their expansions are
89 Appended to a function parameter to tell the compiler that the
90 parameter might not be used within the function. It suppresses the
91 warning that would otherwise appear.
95 Appended to a function prototype to tell the compiler that the
96 function never returns. It allows the compiler to fine-tune its
97 warnings and its code generation.
101 Appended to a function prototype to tell the compiler to never emit
102 the function in-line. Occasionally useful to improve the quality of
103 backtraces (see below).
106 @defmac PRINTF_FORMAT (@var{format}, @var{first})
107 Appended to a function prototype to tell the compiler that the
108 function takes a @func{printf}-like format string as its
109 @var{format}th argument and that the corresponding value arguments
110 start at the @var{first}th argument. This lets the compiler tell you
111 if you pass the wrong argument types.
117 When the kernel panics, it prints a ``backtrace,'' that is, a summary
118 of how your program got where it is, as a list of addresses inside the
119 functions that were running at the time of the panic. You can also
120 insert a call to @func{debug_backtrace}, prototyped in
121 @file{<debug.h>}, at any point in your code.
123 The addresses in a backtrace are listed as raw hexadecimal numbers,
124 which are meaningless in themselves. You can translate them into
125 function names and source file line numbers using a tool called
126 @command{i386-elf-addr2line}.@footnote{If you're using an 80@var{x}86
127 system for development, it's probably just called
128 @command{addr2line}.}
130 The output format of @command{i386-elf-addr2line} is not ideal, so
131 we've supplied a wrapper for it simply called @command{backtrace}.
132 Give it the name of your @file{kernel.o} as the first argument and the
133 hexadecimal numbers composing the backtrace (including the @samp{0x}
134 prefixes) as the remaining arguments. It outputs the function name
135 and source file line numbers that correspond to each address.
137 If the translated form of a backtrace is garbled, or doesn't make
138 sense (e.g.@: function A is listed above function B, but B doesn't
139 call A), then it's a good sign that you're corrupting a kernel
140 thread's stack, because the backtrace is extracted from the stack.
141 Alternatively, it could be that the @file{kernel.o} you passed to
142 @command{backtrace} does not correspond to the kernel that produced
146 * Backtrace Example::
149 @node Backtrace Example
152 Here's an example. Suppose that Pintos printed out this following call
153 stack, which is taken from an actual Pintos submission for the file
157 Call stack: 0xc0106eff 0xc01102fb 0xc010dc22 0xc010cf67 0xc0102319
158 0xc010325a 0x804812c 0x8048a96 0x8048ac8.
161 You would then invoke the @command{backtrace} utility like shown below,
162 cutting and pasting the backtrace information into the command line.
163 This assumes that @file{kernel.o} is in the current directory. You
164 would of course enter all of the following on a single shell command
168 backtrace kernel.o 0xc0106eff 0xc01102fb 0xc010dc22 0xc010cf67 0xc0102319
169 0xc010325a 0x804812c 0x8048a96 0x8048ac8
172 The backtrace output would then look something like this:
175 0xc0106eff: debug_panic (../../lib/debug.c:86)
176 0xc01102fb: file_seek (../../filesys/file.c:405)
177 0xc010dc22: seek (../../userprog/syscall.c:744)
178 0xc010cf67: syscall_handler (../../userprog/syscall.c:444)
179 0xc0102319: intr_handler (../../threads/interrupt.c:334)
180 0xc010325a: ?? (threads/intr-stubs.S:1554)
186 (You will probably not get the same results if you run the command above
187 on your own kernel binary, because the source code you compiled from is
188 different from the source code that emitted the panic message.)
190 The first line in the backtrace refers to @func{debug_panic}, the
191 function that implements kernel panics. Because backtraces commonly
192 result from kernel panics, @func{debug_panic} will often be the first
193 function shown in a backtrace.
195 The second line shows @func{file_seek} to be the function that panicked,
196 in this case as the result of an assertion failure. In the source code
197 tree used for this example, line 405 of @file{filesys/file.c} is the
201 ASSERT (file_ofs >= 0);
205 Thus, @func{file_seek} panicked because it passed a negative file offset
208 The third line indicates that @func{seek} called @func{file_seek},
209 presumably without validating the offset argument. In this submission,
210 @func{seek} implements the @code{seek} system call.
212 The fourth line shows that @func{syscall_handler}, the system call
213 handler, invoked @func{seek}.
215 The fifth and sixth lines are the interrupt handler entry path.
217 The remaining lines are for addresses below @code{PHYS_BASE}. This
218 means that they refer to addresses in the user program, not in the
219 kernel. If you know what user program was running when the kernel
220 panicked, you can re-run @command{backtrace} on the user program, like
221 so: (typing the command on a single line, of course):
224 backtrace grow-too-big 0xc0106eff 0xc01102fb 0xc010dc22 0xc010cf67
225 0xc0102319 0xc010325a 0x804812c 0x8048a96 0x8048ac8
228 The results look like this:
231 0xc0106eff: ?? (??:0)
232 0xc01102fb: ?? (??:0)
233 0xc010dc22: ?? (??:0)
234 0xc010cf67: ?? (??:0)
235 0xc0102319: ?? (??:0)
236 0xc010325a: ?? (??:0)
237 0x804812c: test_main (/home/blp/cs140/pintos/grading/filesys/grow-too-big.c:20)
238 0x8048a96: main (/home/blp/cs140/pintos/grading/filesys/fsmain.c:10)
239 0x8048ac8: _start (../../src/lib/user/entry.c:9)
242 Here's an extra tip for anyone who read this far: @command{backtrace}
243 is smart enough to strip the @code{Call stack:} header and @samp{.}
244 trailer from the command line if you include them. This can save you
245 a little bit of trouble in cutting and pasting. Thus, the following
246 command prints the same output as the first one we used:
249 backtrace kernel.o Call stack: 0xc0106eff 0xc01102fb 0xc010dc22
250 0xc010cf67 0xc0102319 0xc010325a 0x804812c 0x8048a96 0x8048ac8.
254 @section @command{i386-elf-gdb}
256 You can run the Pintos kernel under the supervision of the
257 @command{i386-elf-gdb} debugger.@footnote{If you're using an
258 80@var{x}86 system for development, it's probably just called
259 @command{gdb}.} There are two steps in the process. First,
260 start Pintos with the @option{--gdb} option, e.g.@: @command{pintos
261 --gdb run}. Second, in a second terminal, invoke @command{gdb} on
264 i386-elf-gdb kernel.o
266 @noindent and issue the following @command{gdb} command:
268 target remote localhost:1234
271 At this point, @command{gdb} is connected to Bochs over a local
272 network connection. You can now issue any normal @command{gdb}
273 commands. If you issue the @samp{c} command, the Bochs BIOS will take
274 control, load Pintos, and then Pintos will run in the usual way. You
275 can pause the process at any point with @key{Ctrl+C}. If you want
276 @command{gdb} to stop when Pintos starts running, set a breakpoint on
277 @func{main} with the command @code{break main} before @samp{c}.
279 You can read the @command{gdb} manual by typing @code{info gdb} at a
280 terminal command prompt, or you can view it in Emacs with the command
281 @kbd{C-h i}. Here's a few commonly useful @command{gdb} commands:
285 Continue execution until the next breakpoint or until @key{Ctrl+C} is
288 @item break @var{function}
289 @itemx break @var{filename}:@var{linenum}
290 @itemx break *@var{address}
291 Sets a breakpoint at the given function, line number, or address.
292 (Use a @samp{0x} prefix to specify an address in hex.)
294 @item p @var{expression}
295 Evaluates the given C expression and prints its value.
296 If the expression contains a function call, the function will actually
297 be executed, so be careful.
299 @item l *@var{address}
300 Lists a few lines of code around the given address.
301 (Use a @samp{0x} prefix to specify an address in hex.)
304 Prints a stack backtrace similar to that output by the
305 @command{backtrace} program described above.
307 @item p/a @var{address}
308 Prints the name of the function or variable that occupies the given
310 (Use a @samp{0x} prefix to specify an address in hex.)
313 If you notice other strange behavior while using @command{gdb}, there
314 are three possibilities. The first is that there is a bug in your
315 modified Pintos. The second is that there is a bug in Bochs's
316 interface to @command{gdb} or in @command{gdb} itself. The third is
317 that there is a bug in the original Pintos code. The first and second
318 are quite likely, and you should seriously consider both. We hope
319 that the third is less likely, but it is also possible.
321 @node Debugging by Infinite Loop
322 @section Debugging by Infinite Loop
324 If you get yourself into a situation where the machine reboots in a
325 loop, you've probably hit a ``triple fault.'' In such a situation you
326 might not be able to use @func{printf} for debugging, because the
327 reboots might be happening even before everything needed for
328 @func{printf} is initialized. In such a situation, you might want to
329 try what I call ``debugging by infinite loop.''
331 What you do is pick a place in the Pintos code, insert the statement
332 @code{for (;;);} there, and recompile and run. There are two likely
337 The machine hangs without rebooting. If this happens, you know that
338 the infinite loop is running. That means that whatever caused the
339 problem must be @emph{after} the place you inserted the infinite loop.
340 Now move the infinite loop later in the code sequence.
343 The machine reboots in a loop. If this happens, you know that the
344 machine didn't make it to the infinite loop. Thus, whatever caused the
345 reboot must be @emph{before} the place you inserted the infinite loop.
346 Now move the infinite loop earlier in the code sequence.
349 If you move around the infinite loop in a ``binary search'' fashion, you
350 can use this technique to pin down the exact spot that everything goes
351 wrong. It should only take a few minutes at most.
353 @node Modifying Bochs
354 @section Modifying Bochs
356 An advanced debugging technique is to modify and recompile the
357 simulator. This proves useful when the simulated hardware has more
358 information than it makes available to the OS. For example, page
359 faults have a long list of potential causes, but the hardware does not
360 report to the OS exactly which one is the particular cause.
361 Furthermore, a bug in the kernel's handling of page faults can easily
362 lead to recursive faults, but a ``triple fault'' will cause the CPU to
363 reset itself, which is hardly conducive to debugging.
365 In a case like this, you might appreciate being able to make Bochs
366 print out more debug information, such as the exact type of fault that
367 occurred. It's not very hard. You start by retrieving the source
368 code for Bochs 2.1.1 from @uref{http://bochs.sourceforge.net} and
369 extracting it into a directory. Then read
370 @file{pintos/src/misc/bochs-2.1.1.patch} and apply the patches needed.
371 Then run @file{./configure}, supplying the options you want (some
372 suggestions are in the patch file). Finally, run @command{make}.
373 This will compile Bochs and eventually produce a new binary
374 @file{bochs}. To use your @file{bochs} binary with @command{pintos},
375 put it in your @env{PATH}, and make sure that it is earlier than
376 @file{/usr/class/cs140/i386/bochs}.
378 Of course, to get any good out of this you'll have to actually modify
379 Bochs. Instructions for doing this are firmly out of the scope of
380 this document. However, if you want to debug page faults as suggested
381 above, a good place to start adding @func{printf}s is
382 @func{BX_CPU_C::dtranslate_linear} in @file{cpu/paging.cc}.
387 The page allocator in @file{threads/palloc.c} and the block allocator in
388 @file{threads/malloc.c} both clear all the bytes in pages and blocks to
389 @t{0xcc} when they are freed. Thus, if you see an attempt to
390 dereference a pointer like @t{0xcccccccc}, or some other reference to
391 @t{0xcc}, there's a good chance you're trying to reuse a page that's
392 already been freed. Also, byte @t{0xcc} is the CPU opcode for ``invoke
393 interrupt 3,'' so if you see an error like @code{Interrupt 0x03 (#BP
394 Breakpoint Exception)}, Pintos tried to execute code in a freed page or
397 An assertion failure on the expression @code{sec_no < d->capacity}
398 indicates that Pintos tried to access a file through an inode that has
399 been closed and freed. Freeing an inode clears its starting sector
400 number to @t{0xcccccccc}, which is not a valid sector number for disks
401 smaller than about 1.6 TB.