# The utrace User Debugging Infrastructure

## utrace concepts

### Introduction

utrace is infrastructure code for tracing and controlling user threads. This is the foundation for writing tracing engines, which can be loadable kernel modules.

The basic actors in utrace are the thread and the tracing engine. A tracing engine is some body of code that calls into the `<linux/utrace.h>` interfaces, represented by a struct utrace_engine_ops. (Usually it's a kernel module, though the legacy `ptrace` support is a tracing engine that is not in a kernel module.) The interface operates on individual threads (struct task_struct). If an engine wants to treat several threads as a group, that is up to its higher-level code.

Tracing begins by attaching an engine to a thread, using `utrace_attach_task` or `utrace_attach_pid`. If successful, it returns a pointer that is the handle used in all other calls.

### Events and Callbacks

An attached engine does nothing by default. An engine makes something happen by requesting callbacks via `utrace_set_events` and poking the thread with `utrace_control`. The synchronization issues related to these two calls are discussed further below in .

Events are specified using the macro `UTRACE_EVENT(type)`. Each event type is associated with a callback in struct utrace_engine_ops. A tracing engine can leave unused callbacks `NULL`. The only callbacks required are those used by the event flags it sets.

Many engines can be attached to each thread. When a thread has an event, each engine gets a callback if it has set the event flag for that event type. Engines are called in the order they attached. Engines that attach after the event has occurred do not get callbacks for that event. This includes any new engines just attached by an existing engine's callback function. Once the sequence of callbacks for that one event has completed, such new engines are then eligible in the next sequence that starts when there is another event.

Event reporting callbacks have details particular to the event type, but are all called in similar environments and have the same constraints. Callbacks are made from safe points, where no locks are held, no special resources are pinned (usually), and the user-mode state of the thread is accessible. So, callback code has a pretty free hand. But to be a good citizen, callback code should never block for long periods. It is fine to block in `kmalloc` and the like, but never wait for i/o or for user mode to do something. If you need the thread to wait, use `UTRACE_STOP` and return from the callback quickly. When your i/o finishes or whatever, you can use `utrace_control` to resume the thread.

### Stopping Safely

#### Writing well-behaved callbacks

Well-behaved callbacks are important to maintain two essential properties of the interface. The first of these is that unrelated tracing engines should not interfere with each other. If your engine's event callback does not return quickly, then another engine won't get the event notification in a timely manner. The second important property is that tracing should be as noninvasive as possible to the normal operation of the system overall and of the traced thread in particular. That is, attached tracing engines should not perturb a thread's behavior, except to the extent that changing its user-visible state is explicitly what you want to do. (Obviously some perturbation is unavoidable, primarily timing changes, ranging from small delays due to the overhead of tracing, to arbitrary pauses in user code execution when a user stops a thread with a debugger for examination.) Even when you explicitly want the perturbation of making the traced thread block, just blocking directly in your callback has more unwanted effects. For example, the `CLONE` event callbacks are called when the new child thread has been created but not yet started running; the child can never be scheduled until the `CLONE` tracing callbacks return. (This allows engines tracing the parent to attach to the child.) If a `CLONE` event callback blocks the parent thread, it also prevents the child thread from running (even to process a `SIGKILL`). If what you want is to make both the parent and child block, then use `utrace_attach_task` on the child and then use `UTRACE_STOP` on both threads. A more crucial problem with blocking in callbacks is that it can prevent `SIGKILL` from working. A thread that is blocking due to `UTRACE_STOP` will still wake up and die immediately when sent a `SIGKILL`, as all threads should. Relying on the utrace infrastructure rather than on private synchronization calls in event callbacks is an important way to help keep tracing robustly noninvasive.

#### Using `UTRACE_STOP`

To control another thread and access its state, it must be stopped with `UTRACE_STOP`. This means that it is stopped and won't start running again while we access it. When a thread is not already stopped, `utrace_control` returns `-EINPROGRESS` and an engine must wait for an event callback when the thread is ready to stop. The thread may be running on another CPU or may be blocked. When it is ready to be examined, it will make callbacks to engines that set the `UTRACE_EVENT(QUIESCE)` event bit. To wake up an interruptible wait, use `UTRACE_INTERRUPT`.

As long as some engine has used `UTRACE_STOP` and not called `utrace_control` to resume the thread, then the thread will remain stopped. `SIGKILL` will wake it up, but it will not run user code. When the stop is cleared with `utrace_control` or a callback return value, the thread starts running again. (See also .)

### Tear-down Races

#### Primacy of `SIGKILL`

Ordinarily synchronization issues for tracing engines are kept fairly straightforward by using `UTRACE_STOP`. You ask a thread to stop, and then once it makes the `report_quiesce` callback it cannot do anything else that would result in another callback, until you let it with a `utrace_control` call. This simple arrangement avoids complex and error-prone code in each one of a tracing engine's event callbacks to keep them serialized with the engine's other operations done on that thread from another thread of control. However, giving tracing engines complete power to keep a traced thread stuck in place runs afoul of a more important kind of simplicity that the kernel overall guarantees: nothing can prevent or delay `SIGKILL` from making a thread die and release its resources. To preserve this important property of `SIGKILL`, it as a special case can break `UTRACE_STOP` like nothing else normally can. This includes both explicit `SIGKILL` signals and the implicit `SIGKILL` sent to each other thread in the same thread group by a thread doing an exec, or processing a fatal signal, or making an `exit_group` system call. A tracing engine can prevent a thread from beginning the exit or exec or dying by signal (other than `SIGKILL`) if it is attached to that thread, but once the operation begins, no tracing engine can prevent or delay all other threads in the same thread group dying.

#### Final callbacks

The `report_reap` callback is always the final event in the life cycle of a traced thread. Tracing engines can use this as the trigger to clean up their own data structures. The `report_death` callback is always the penultimate event a tracing engine might see; it's seen unless the thread was already in the midst of dying when the engine attached. Many tracing engines will have no interest in when a parent reaps a dead process, and nothing they want to do with a zombie thread once it dies; for them, the `report_death` callback is the natural place to clean up data structures and detach. To facilitate writing such engines robustly, given the asynchrony of `SIGKILL`, and without error-prone manual implementation of synchronization schemes, the utrace infrastructure provides some special guarantees about the `report_death` and `report_reap` callbacks. It still takes some care to be sure your tracing engine is robust to tear-down races, but these rules make it reasonably straightforward and concise to handle a lot of corner cases correctly.

#### Engine and task pointers

The first sort of guarantee concerns the core data structures themselves. struct utrace_engine is a reference-counted data structure. While you hold a reference, an engine pointer will always stay valid so that you can safely pass it to any utrace call. Each call to `utrace_attach_task` or `utrace_attach_pid` returns an engine pointer with a reference belonging to the caller. You own that reference until you drop it using `utrace_engine_put`. There is an implicit reference on the engine while it is attached. So if you drop your only reference, and then use `utrace_attach_task` without `UTRACE_ATTACH_CREATE` to look up that same engine, you will get the same pointer with a new reference to replace the one you dropped, just like calling `utrace_engine_get`. When an engine has been detached, either explicitly with `UTRACE_DETACH` or implicitly after `report_reap`, then any references you hold are all that keep the old engine pointer alive.

There is nothing a kernel module can do to keep a struct task_struct alive outside of `rcu_read_lock`. When the task dies and is reaped by its parent (or itself), that structure can be freed so that any dangling pointers you have stored become invalid. utrace will not prevent this, but it can help you detect it safely. By definition, a task that has been reaped has had all its engines detached. All utrace calls can be safely called on a detached engine if the caller holds a reference on that engine pointer, even if the task pointer passed in the call is invalid. All calls return `-ESRCH` for a detached engine, which tells you that the task pointer you passed could be invalid now. Since `utrace_control` and `utrace_set_events` do not block, you can call those inside a `rcu_read_lock` section and be sure after they don't return `-ESRCH` that the task pointer is still valid until `rcu_read_unlock`. The infrastructure never holds task references of its own. Though neither `rcu_read_lock` nor any other lock is held while making a callback, it's always guaranteed that the struct task_struct and the struct utrace_engine passed as arguments remain valid until the callback function returns.

The common means for safely holding task pointers that is available to kernel modules is to use struct pid, which permits `put_pid` from kernel modules. When using that, the calls `utrace_attach_pid`, `utrace_control_pid`, `utrace_set_events_pid`, and `utrace_barrier_pid` are available.

#### Serialization of `DEATH` and `REAP`

The second guarantee is the serialization of `DEATH` and `REAP` event callbacks for a given thread. The actual reaping by the parent (`release_task` call) can occur simultaneously while the thread is still doing the final steps of dying, including the `report_death` callback. If a tracing engine has requested both `DEATH` and `REAP` event reports, it's guaranteed that the `report_reap` callback will not be made until after the `report_death` callback has returned. If the `report_death` callback itself detaches from the thread, then the `report_reap` callback will never be made. Thus it is safe for a `report_death` callback to clean up data structures and detach.

#### Interlock with final callbacks

The final sort of guarantee is that a tracing engine will know for sure whether or not the `report_death` and/or `report_reap` callbacks will be made for a certain thread. These tear-down races are disambiguated by the error return values of `utrace_set_events` and `utrace_control`. Normally `utrace_control` called with `UTRACE_DETACH` returns zero, and this means that no more callbacks will be made. If the thread is in the midst of dying, it returns `-EALREADY` to indicate that the `report_death` callback may already be in progress; when you get this error, you know that any cleanup your `report_death` callback does is about to happen or has just happened--note that if the `report_death` callback does not detach, the engine remains attached until the thread gets reaped. If the thread is in the midst of being reaped, `utrace_control` returns `-ESRCH` to indicate that the `report_reap` callback may already be in progress; this means the engine is implicitly detached when the callback completes. This makes it possible for a tracing engine that has decided asynchronously to detach from a thread to safely clean up its data structures, knowing that no `report_death` or `report_reap` callback will try to do the same. `utrace_detach` returns `-ESRCH` when the struct utrace_engine has already been detached, but is still a valid pointer because of its reference count. A tracing engine can use this to safely synchronize its own independent multiple threads of control with each other and with its event callbacks that detach.

In the same vein, `utrace_set_events` normally returns zero; if the target thread was stopped before the call, then after a successful call, no event callbacks not requested in the new flags will be made. It fails with `-EALREADY` if you try to clear `UTRACE_EVENT(DEATH)` when the `report_death` callback may already have begun, if you try to clear `UTRACE_EVENT(REAP)` when the `report_reap` callback may already have begun, or if you try to newly set `UTRACE_EVENT(DEATH)` or `UTRACE_EVENT(QUIESCE)` when the target is already dead or dying. Like `utrace_control`, it returns `-ESRCH` when the thread has already been detached (including forcible detach on reaping). This lets the tracing engine know for sure which event callbacks it will or won't see after `utrace_set_events` has returned. By checking for errors, it can know whether to clean up its data structures immediately or to let its callbacks do the work.

#### Using `utrace_barrier`

When a thread is safely stopped, calling `utrace_control` with `UTRACE_DETACH` or calling `utrace_set_events` to disable some events ensures synchronously that your engine won't get any more of the callbacks that have been disabled (none at all when detaching). But these can also be used while the thread is not stopped, when it might be simultaneously making a callback to your engine. For this situation, these calls return `-EINPROGRESS` when it's possible a callback is in progress. If you are not prepared to have your old callbacks still run, then you can synchronize to be sure all the old callbacks are finished, using `utrace_barrier`. This is necessary if the kernel module containing your callback code is going to be unloaded.

After using `UTRACE_DETACH` once, further calls to `utrace_control` with the same engine pointer will return `-ESRCH`. In contrast, after getting `-EINPROGRESS` from `utrace_set_events`, you can call `utrace_set_events` again later and if it returns zero then know the old callbacks have finished.

Unlike all other calls, `utrace_barrier` (and `utrace_barrier_pid`) will accept any engine pointer you hold a reference on, even if `UTRACE_DETACH` has already been used. After any `utrace_control` or `utrace_set_events` call (these do not block), you can call `utrace_barrier` to block until callbacks have finished. This returns `-ESRCH` only if the engine is completely detached (finished all callbacks). Otherwise it waits until the thread is definitely not in the midst of a callback to this engine and then returns zero, but can return `-ERESTARTSYS` if its wait is interrupted.

## utrace core API

The utrace API is declared in `<linux/utrace.h>`.

!Iinclude/linux/utrace.h !Ekernel/utrace.c

## Machine State

The `task_current_syscall` function can be used on any valid struct task_struct at any time, and does not even require that `utrace_attach_task` was used at all.

The other ways to access the registers and other machine-dependent state of a task can only be used on a task that is at a known safe point. The safe points are all the places where `utrace_set_events` can request callbacks (except for the `DEATH` and `REAP` events). So at any event callback, it is safe to examine `current`.

One task can examine another only after a callback in the target task that returns `UTRACE_STOP` so that task will not return to user mode after the safe point. This guarantees that the task will not resume until the same engine uses `utrace_control`, unless the task dies suddenly. To examine safely, one must use a pair of calls to `utrace_prepare_examine` and `utrace_finish_examine` surrounding the calls to struct user_regset functions or direct examination of task data structures. `utrace_prepare_examine` returns an error if the task is not properly stopped and not dead. After a successful examination, the paired `utrace_finish_examine` call returns an error if the task ever woke up during the examination. If so, any data gathered may be scrambled and should be discarded. This means there was a spurious wake-up (which should not happen), or a sudden death.

### struct user_regset

The struct user_regset API is declared in `<linux/regset.h>`.

!Finclude/linux/regset.h

### `System Call Information`

This function is declared in `<linux/ptrace.h>`.

!Elib/syscall.c

### `System Call Tracing`

The arch API for system call information is declared in `<asm/syscall.h>`. Each of these calls can be used only at system call entry tracing, or can be used only at system call exit and the subsequent safe points before returning to user mode. At system call entry tracing means either during a report_syscall_entry callback, or any time after that callback has returned `UTRACE_STOP`.

!Finclude/asm-generic/syscall.h

## Kernel Internals

This chapter covers the interface to the tracing infrastructure from the core of the kernel and the architecture-specific code. This is for maintainers of the kernel and arch code, and not relevant to using the tracing facilities described in preceding chapters.

### Core Calls In

These calls are declared in `<linux/tracehook.h>`. The core kernel calls these functions at various important places.

!Finclude/linux/tracehook.h

### Architecture Calls Out

An arch that has done all these things sets `CONFIG_HAVE_ARCH_TRACEHOOK`. This is required to enable the utrace code.

#### `<asm/ptrace.h>`

An arch defines these in `<asm/ptrace.h>` if it supports hardware single-step or block-step features.

!Finclude/linux/ptrace.h arch_has_single_step arch_has_block_step !Finclude/linux/ptrace.h user_enable_single_step user_enable_block_step !Finclude/linux/ptrace.h user_disable_single_step

#### `<asm/syscall.h>`

An arch provides `<asm/syscall.h>` that defines these as inlines, or declares them as exported functions. These interfaces are described in .

#### `<linux/tracehook.h>`

An arch must define `TIF_NOTIFY_RESUME` and `TIF_SYSCALL_TRACE` in its `<asm/thread_info.h>`. The arch code must call the following functions, all declared in `<linux/tracehook.h>` and described in :

- `tracehook_notify_resume`

- `tracehook_report_syscall_entry`

- `tracehook_report_syscall_exit`

- `tracehook_signal_handler`