|FROM ||Ruben Safir
|SUBJECT ||Subject: [LIU Comp Sci] OS Class Ctp 2 Dtrace
|From owner-learn-outgoing-at-mrbrklyn.com Tue Feb 17 02:46:38 2015
Received: by mrbrklyn.com (Postfix)
id EC938161162; Tue, 17 Feb 2015 02:46:37 -0500 (EST)
Received: by mrbrklyn.com (Postfix, from userid 28)
id D85391612E7; Tue, 17 Feb 2015 02:46:37 -0500 (EST)
Received: from mailbackend.panix.com (mailbackend.panix.com [188.8.131.52])
by mrbrklyn.com (Postfix) with ESMTP id 66C71161162
for ; Tue, 17 Feb 2015 02:46:37 -0500 (EST)
Received: from [10.0.0.19] (unknown [184.108.40.206])
by mailbackend.panix.com (Postfix) with ESMTPSA id EAD8419F13
for ; Tue, 17 Feb 2015 02:46:36 -0500 (EST)
Date: Tue, 17 Feb 2015 02:46:35 -0500
From: Ruben Safir
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0
Subject: [LIU Comp Sci] OS Class Ctp 2 Dtrace
Content-Type: text/plain; charset=utf-8
Does anyone understand this? It is from Chapter to of the OS class
text. I'm very confused by the text here because it seems to be saying
that when probes are turned on they are rewritten and I don't know what
is rewritten. Can you rewrite a binary while its in memory? That would
be news to me.
DTrace is a facility that dynamically adds probes to a running system, both
in user processes and in the kernel. These probes can be queried via the D
programming language to determine an astonishing amount about the kernel,
the system state, and process activities. For example, Figure 2.20
application as it executes a system call (ioctl()) and shows the functional
calls within the kernel as they execute to perform the system call.
with “U” are executed in user mode, and lines ending in “K” in kernel mode.
Chapter 2 Operating-System Structures
Figure 2.19 The Windows task manager.
Debugging the interactions between user-level and kernel code is nearly
impossible without a toolset that understands both sets of code and can
instrument the interactions. For that toolset to be truly useful, it
must be able
to debug any area of a system, including areas that were not written with
debugging in mind, and do so without affecting system reliability. This tool
must also have a minimum performance impact—ideally it should have no
impact when not in use and a proportional impact during use. The DTrace tool
meets these requirements and provides a dynamic, safe, low-impact debugging
Until the DTrace framework and tools became available with Solaris 10,
kernel debugging was usually shrouded in mystery and accomplished via
happenstance and archaic code and tools. For example, CPUs have a breakpoint
feature that will halt execution and allow a debugger to examine the
state of the
system. Then execution can continue until the next breakpoint or
This method cannot be used in a multiuser operating-system kernel without
negatively affecting all of the users on the system. Profiling, which
samples the instruction pointer to determine which code is being
show statistical trends but not individual activities. Code can be
the kernel to emit specific data under specific circumstances, but that code
slows down the kernel and tends not to be included in the part of the kernel
where the specific problem being debugged is occurring.
2.8 Operating-System Debugging
# ./all.d ‘pgrep xclock‘ XEventsQueued
dtrace: script ’./all.d’ matched 52377 probes
0 –> XEventsQueued
–> _X11TransSocketBytesReadable U
<– _X11TransSocketBytesreadable U
0 <– XEventsQueued
Figure 2.20 Solaris 10 dtrace follows a system call within the kernel.
In contrast, DTrace runs on production systems—systems that are running
important or critical applications—and causes no harm to the system. It
slows activities while enabled, but after execution it resets the system
pre-debugging state. It is also a broad and deep tool. It can broadly debug
everything happening in the system (both at the user and kernel levels and
between the user and kernel layers). It can also delve deep into code,
individual CPU instructions or kernel subroutine activities.
DTrace is composed of a compiler, a framework, providers of probes
written within that framework, and consumers of those probes. DTrace
providers create probes. Kernel structures exist to keep track of all
the providers have created. The probes are stored in a hash-table data
that is hashed by name and indexed according to unique probe identifiers.
When a probe is enabled, a bit of code in the area to be probed is rewritten
to call dtrace probe(probe identifier) and then continue with the code’s
original operation. Different providers create different kinds of
example, a kernel system-call probe works differently from a user-process
probe, and that is different from an I/O probe.
DTrace features a compiler that generates a byte code that is run in the
kernel. This code is assured to be “safe” by the compiler. For example,
are allowed, and only specific kernel state modifications are allowed when
specifically requested. Only users with DTrace “privileges” (or “root”
Chapter 2 Operating-System Structures
are allowed to use DTrace, as it can retrieve private kernel data (and
data if requested). The generated code runs in the kernel and enables
It also enables consumers in user mode and enables communications between
A DTrace consumer is code that is interested in a probe and its results.
A consumer requests that the provider create one or more probes. When a
probe fires, it emits data that are managed by the kernel. Within the
actions called enabling control blocks, or ECBs, are performed when probes
fire. One probe can cause multiple ECBs to execute if more than one consumer
is interested in that probe. Each ECB contains a predicate (“if
can filter out that ECB. Otherwise, the list of actions in the ECB is
most common action is to capture some bit of data, such as a variable’s
that point of the probe execution. By gathering such data, a complete
a user or kernel action can be built. Further, probes firing from both
and the kernel can show how a user-level action caused kernel-level
Such data are invaluable for performance monitoring and code optimization.
Once the probe consumer terminates, its ECBs are removed. If there are no
ECBs consuming a probe, the probe is removed. That involves rewriting the
code to remove the dtrace probe() call and put back the original code. Thus,
before a probe is created and after it is destroyed, the system is
same, as if no probing occurred.
DTrace takes care to assure that probes do not use too much memory or
CPU capacity, which could harm the running system. The buffers used to hold
the probe results are monitored for exceeding default and maximum limits.
CPU time for probe execution is monitored as well. If limits are
consumer is terminated, along with the offending probes. Buffers are
per CPU to avoid contention and data loss.
An example of D code and its output shows some of its utility. The following
program shows the DTrace code to enable scheduler probes and record the
amount of CPU time of each process running with user ID 101 while those
probes are enabled (that is, while the program runs):
uid == 101
self->ts = timestamp;
-at-time[execname] = sum(timestamp - self->ts);
self->ts = 0;
The output of the program, showing the processes and how much time (in
nanoseconds) they spend running on the CPUs, is shown in Figure 2.21.
Because DTrace is part of the open-source OpenSolaris version of the Solaris
10 operating system, it has been added to other operating systems when those
2.9 Operating-System Generation
# dtrace -s sched.d
dtrace: script ’sched.d’ matched 6 probes
Figure 2.21 Output of the D code.
systems do not have conflicting license agreements. For example, DTrace has
been added to Mac OS X and FreeBSD and will likely spread further due to its
unique capabilities. Other operating systems, especially the Linux
are adding kernel-tracing functionality as well. Still other operating
are beginning to include performance and tracing tools fostered by
various institutions, including the Paradyn project.