Commit 7a9ac36

@@ -557,3 +557,65 @@ Android runs on a variety of mobile platforms and is open-source.
 |                  Linux kernel                    |
 ----------------------------------------------------
 ```
+
+## Operating System Debugging
+
+`debugging` is the activity of finding and fixing errors in a system,
+both in hardware and in software.
+`debugging` can also include `performance tuning` which seeks to
+improve the performance by removing `bottlenecks`.
+
+### Failure Analysis
+
+If a process fails, most operating ssytems write the error information
+to a log file to alert system operators of users that the problem occurred.
+The OS can also take a `core dump`. A `core dump` is a capture of the
+process memory and stored in a file. Memory used to be referred to as
+"core" memory.
+
+A failure in the kernel is called a `crash`. When a crash occurs, error
+information is saved to a log file, and the memory state is saved to a `crash dump`.
+
+A common technique is to save the kernel's memory state to a section of disk
+set aside for this purpose that contains no file system. If the kernel detects
+an unrecoverable error, it writes the entire contents of memory, or at least the
+kernel-owned parts of the system memory, to the disk area.
+When the system reboots, a process runs to gather the data from that area and write
+it to a crash dump file within a file system for analysis.
+
+### Performance Tuning
+
+The operating system must have some means of computing and displaying
+measures of system behaviour.
+
+In many systems, the OS does this by providing `trace listings` of system behaviour.
+E.g. `top`.
+
+### DTrace
+
+DTrace is a facility that dynamically adds probes to a running system, both
+in user processes and in the kernel. These probes can be queried via the
+D programming language to determine an astonishing amount about the kernel,
+the system state, and process activities.
+
+Lines ending with "U" are executed in user mode, and lines ending in "K" in kernel mode.
+
+Debugging interactions between user-level and kernel code is difficult without tools
+that understand both sets of code and can instrument the interaction.
+
+`profiling`, which periodically samples the instruction pointer to determine which code
+is being executed, can show statistical trends but not individual activities.
+
+Code can be included in the kernel to emit specific data under specific circumstances, but
+that code slows down the kernel and test not to be included in the part of the kernel
+where the specific problem being debugged is occurring.
+
+`DTrace` runs on production systems and causes no harm to the system.
+It slows activies while enabled, but after execution it resets the system to its
+pre-debugging state.
+
+It can broadly debug everything happening in the system (both the user and kernel levels
+and between the user and kernel layers).
+
+DTrace is composed of a compiler, a framework, `providers` of `probes` written within
+that framework, and `consumers` of those probes.