DRAFT: Take more than your usual care.

NtTraceControl

This function is the central switching point for managing Event Tracing For Windows (ETW).

Declaration

NTSTATUS 
NtTraceControl (
    ULONG FunctionCode, 
    PVOID InBuffer, 
    ULONG InBufferLen, 
    PVOID OutBuffer, 
    ULONG OutBufferLen, 
    ULONG *ReturnSize);

Parameters

The FunctionCode argument selects from the function’s many operations.

The optional InBuffer and InBufferLen arguments are respectively the address and size (in bytes) of a buffer that provides the function with input. What the function interprets of this input depends on the FunctionCode.

The optional OutBuffer and OutBufferLen arguments are respectively the address and size (in bytes) of a buffer that receives the function’s output. What the function puts into this buffer depends on the FunctionCode.

The required ReturnSize argument is the address of a variable whose value on output tells how many bytes the successful function has put into the output buffer or may tell how much the failed function might have put into the output buffer (had it been large enough).

Return Value

The function returns STATUS_SUCCESS if successful, else a negative error code.

For some function codes, the function can return some other (positive) indication of success, such as STATUS_MORE_ENTRIES.

The variable at ReturnSize is set in the following circumstances:

Availability

Both the NtTraceControl and ZwTraceControl functions are exported by name from NTDLL in version 6.0 and higher. There, in user mode, the functions are aliases for a stub that transfers execution to the NtTraceControl implementation in kernel mode such that the execution is recognised as originating in user mode.

This NtTraceControl implementation is exported by name from the kernel in version 6.0 and higher. Only in version 10.0 and higher does the kernel also export a ZwTraceControl. The kernel-mode version of ZwTraceControl is also a stub that transfers execution to the NtTraceControl implementation but such that the execution is recognised as originating in kernel mode.

Though the kernel has no NtTraceControl in versions 5.1 and 5.2, it does export functions that are recognisable as precursors in the sense that each does the work that is later done through one NtTraceControl function code. These earlier functions are WmiStartTrace, WmiStopTrace, WmiQueryTrace, WmiUpdateTrace and WmiFlushTrace. This and other functionality of NtTraceControl is also supported in versions 5.0 to 5.2 as Device I/O Control through the WMI support device. Except for this paragraph to note that NtTraceControl did not arrive out of the blue, none of this earlier support in any form is any concern here.

Documentation Status

Though the NtTraceControl and ZwTraceControl functions are not documented under either name, C-language declarations have been published by Microsoft in headers from the Enterprise edition of the Windows Driver Kit (WDK) for Windows 10 version 1511: NtTraceControl in NTWMI.H and ZwTraceControl in ZWAPI.H.

Behaviour

The following implementation notes are from inspection of the kernel from the original release of Windows 10 only. They may some day get revised to account for
other versions, whether to update or to follow through with the history. Meanwhile, where anything is added about earlier versions, take it not as an attempt at comprehensiveness but as a bonus from my being unable to resist a trip down memory lane.

User-Mode Defences

If executing for a user-mode request, the function has some general defensiveness about addresses passed as arguments. Failure at any of these defences is failure for the function, which typically returns STATUS_DATATYPE_MISALIGNMENT or STATUS_ACCESS_VIOLATION (showing in kernel mode as raised but handled exceptions).

Buffers

The InBuffer argument can be NULL to provide no input, in which case InBufferLen is ignored (literally, treated as zero). If an input buffer is given, meaning here that InBuffer is not NULL and InBufferLen is not zero, then the whole of the buffer must be in user-mode address space.

The OutBuffer argument can be NULL so that no output is requested, in which case OutBufferLen is ignored (literally, treated as zero). If an output buffer is given, meaning here that OutBuffer is not NULL and OutBufferLen is not zero, then the whole buffer must be in user-mode address space and be writable (at its first byte and also for a byte at each page boundary that is inside the buffer).

Return Size

A variable for learning how much output is or could be produced is required. The variable must be in user-mode address space and be writable. If instead ReturnSize is NULL, the function returns STATUS_INVALID_PARAMETER.

Kernel-Mode Acceptance

If executing for a kernel-mode request, all arguments are trusted as given. This means in particular that:

Buffered I/O

Except for function codes 0x0C, 0x0E and 0x1B or if given no buffer for either input or output, the function double-buffers. Specifically, it obtains from the paged pool an allocation whose size is the larger of the input and output buffers. If it cannot get this memory, it returns STATUS_NO_MEMORY. If an input buffer is given, the function copies the whole of it to the double buffer so that all further work with the input is from the double buffer, not from the input buffer. If the function prepares output, it does so in the double buffer and copies to the caller-supplied output buffer only when about to return STATUS_SUCCESS.

If executing for a kernel-mode request, the need for and size of the double buffer is determined from the InBufferLen and OutBufferLen arguments as given, even if either or both of InBuffer and OutBuffer is NULL.

Exception Handling

The function never accesses InBuffer, OutBuffer or ReturnSize without preparing for exceptions. The occurrence of an exception during such access is fatal to the function, which returns the exception code as its own result.

Valid Function Codes

Microsoft’s names for eight of the valid function codes are known from type information in symbol files for Windows 8 and higher—though even then, not the symbol files for the kernel, which interprets the codes, nor for the obvious low-level user-mode DLLs that use the codes for their calls to NtTraceControl. The function codes apparently take their values from an enumeration named ETWTRACECONTROLCODE. A formal C-language definition is published in the NTETW.H from the Enterprise WDK for Windows 10 version 1511, but just repeats these eight.

The table below lists the function codes that the function does not dismiss as invalid (after the preceding defences). For all others, the function returns STATUS_INVALID_DEVICE_REQUEST. The versions that are presently shown for each function code are just from a cursory look for the upper limit that the function applies in different builds.

Numeric Value Symbolic Name Versions
0x01 EtwStartLoggerCode 6.0 and higher
0x02 EtwStopLoggerCode 6.0 and higher
0x03 EtwQueryLoggerCode 6.0 and higher
0x04 EtwUpdateLoggerCode 6.0 and higher
0x05 EtwFlushLoggerCode 6.0 and higher
0x0B real-time connect 6.0 and higher
0x0C EtwActivityIdCreate 6.0 and higher
0x0D EtwWdiScenarioCode 6.0 and higher
0x0E real-time disconnect consumer by handle 6.0 and higher
0x0F register user-mode GUID 6.0 and higher
0x10 receive notification 6.0 and higher
0x11 enable GUID 6.0 and higher
0x12 send reply data block 6.0 and higher
0x13 receive reply data block 6.0 and higher
0x14 EtwWdiSemUpdate 6.0 and higher
0x15 get trace GUID list 6.0 and higher
0x16 get trace GUID information 6.0 and higher
0x17 enumerate trace GUIDs 6.0 and higher
0x18 register security provider 6.0 and higher
0x19 query reference time 6.2 and higher
0x1A track provider binary 6.2 and higher
0x1B add notification event 6.3 and higher
0x1C update disallow list 10.0 and higher
0x1E set provider traits 10.0 and higher
0x1F use descriptor type 10.0 and higher
0x20 get trace group list 10.0 and higher
0x21 get trace group information 10.0 and higher
0x22 get disallow list 10.0 and higher

All remaining behaviour varies with the function code.

EtwStartLoggerCode (0x01) to EtwFlushLoggerCode (0x05) Inclusive

The first five function codes start or control a logger, known more formally as a tracing session. For all, a WMI_LOGGER_INFORMATION is expected to begin the input and is produced as the function’s successful output.

Some validation is common to all five of these function codes. If either the input or output buffer is too small for this structure, including because a user-mode request gives NULL for either buffer’s address, the function returns STATUS_INVALID_BUFFER_SIZE. If InBuffer is NULL (which is posssible only for a badly formed kernel-mode request), the function returns STATUS_INVALID_PARAMETER. The expected structure begins with a WNODE_HEADER. Within this, the BufferSize is expected to tell how many bytes of data this header is the first part of. If this is too small even for the fixed-size WMI_LOGGER_INFORMATION, the function returns STATUS_INVALID_BUFFER_SIZE. If the Flags in the WNODE_HEADER do not have the WNODE_FLAG_TRACED_GUID (0x00020000) bit set, the function returns STATUS_INVALID_PARAMETER. If BufferSize is larger than InBufferLen, the function returns STATUS_INVALID_BUFFER_SIZE.

All remaining behaviour varies with the function code.

Function Code 0x0B

This function code connects a real-time event consumer to a logger. A fixed-size context is expected as both input and output.

Microsoft’s name for this real-time connection context is not known. (Among the insufficiently substantial clues are that another structure’s pointer to this structure is named RealtimeConnectContext.) In version 6.1 and higher, most of the structure provides input for creation of an EtwConsumer object. Internally this is an ETW_REALTIME_CONSUMER structure. Type information for this is available in public symbol files and it seems highly likely that Microsoft uses the same names and types in both structures for members that correspond closely. Where pointers are padded to 64 bits so that the kernel deals with one format for both 32-bit and 64-bit callers, type information for other structures that are involved with NtTraceControl, e.g., WMI_LOGGER_INFORMATION, suggests a convention that Microsoft perhaps uses for this structure too.

Offset Definition Versions Remarks
0x00
ULONG LoggerId;
6.0 and higher input
0x04
ULONG ReservedBufferSpaceSize;
6.1 and higher input
0x08
union {
    UCHAR *ReservedBufferSpace;
    ULONG64 ReservedBufferSpace64;
};
6.1 and higher input
0x10 a 64-bit allowance for the address of a buffer to use for the ReservedBufferSpaceBitMap 6.1 and higher input
0x18
union {
    HANDLE DisconnectEvent;
    ULONG64 DisconnectEvent64;
};
6.1 and higher input
0x20
union {
    HANDLE DataAvailableEvent;
    ULONG64 DataAvailableEvent64;
};
6.1 and higher input
0x28
union {
    SINGLE_LIST_ENTRY *UserBufferListHead;
    ULONG64 UserBufferListHead64;
};
6.1 and higher input
0x30
union {
    ULONG *UserBufferCount;
    ULONG64 UserBufferCount64;
};
6.1 and higher input
0x38
union {
    ULONG *EventsLostCount;
    ULONG64 EventsLostCount64;
};
6.3 and higher input
0x40
union {
    ULONG *BuffersLostCount;
    ULONG64 BuffersLostCount;
};
6.3 and higher input
0x04 (6.0) a 32-bit consumer ID 6.0 only output
0x08 (6.0);
0x38 (6.1);
0x48
a 64-bit allowance for a HANDLE to a pipe 6.0 only output
a 64-bit allowance for a HANDLE to the EtwConsumer object 6.1 and higher output
0x10 (6.0);
0x40 (6.1);
0x50
an ETW_REF_CLOCK 6.0 and higher output

Version 6.0 defines no object type for real-time event consumers and therefore has a very different implementation. Each connection of a real-time consumer to a logger is represented not by a handle but by a sequence number. When tracing events, the kernel does not have direct access to the user-mode memory supplied by the consumer but instead writes through a pipe.

If the input and output buffers are not both exactly the expected size, i.e., 0x20 bytes in version 6.0, 0x50 in versions 6.1 to 6.2, and 0x60 bytes in version 6.3 and higher, the function returns STATUS_INVALID_PARAMETER.

EtwActivityIdCreate (0x0C)

This function code creates an activity ID such as may be associated with some occurrence of an event to help track relationships between occurrences.

The output buffer is to receive a GUID. If the output buffer does not allow for exactly 0x10 bytes, the function returns STATUS_INVALID_PARAMETER. (In versions 6.0 to 6.1, for no known reason, the input buffer also must be exactly 0x10 bytes.)

EtwWdiScenarioCode (0x0D)

This function code tells the kernel’s Scenario Event Mapper (SEM) to start or stop every Windows Diagnostic Infrastructure (WDI) scenario for which the given event from the given event provider is configured as a start or end event.

Microsoft presumably defines this function code’s expected input as a structure but Microsoft’s name for it is not known:

Offset Definition Versions
0x00 a 64-bit allowance for a 32-bit registration handle to provider 6.0 and higher
0x08 an EVENT_DESCRIPTOR 6.0 and higher
0x18 a GUID as the Activity ID 6.0 and higher
0x28 32-bit control code:
10 to start scenario;
else (but apparently intended as 11) to end scenario
6.0 and higher

If the input buffer does not provide exactly 0x30 bytes or if any output buffer is given, the function returns STATUS_INVALID_PARAMETER.

Function Code 0x0E

This function code disconnects a real-time consumer.

If the input buffer does not provide exactly 8 bytes or if any output buffer is given, the function returns STATUS_INVALID_PARAMETER. In version 6.0, all eight bytes of input are meaningful: two dwords identify respectively the logger and the consumer. Version 6.1 introduced the EtwConsumer as an object type such that a handle to this object represents the consumer and its connection to a logger, and is all that’s needed for disconnection. The input buffer provides this handle as its first 32 bits even on 64-bit Windows.

Function Code 0x0F

This function code registers a user-mode event provider. A provider is represented by a GUID and can have any number of registrations. For a user-mode registration, the effect really is like opening a provider in that it produces an Object Manager handle to the underlying registration object. As well as opening this handle for the user-mode caller, the function also produces information. In all versions, this includes to describe a tracing session, if any, that has started in anticipation of the provider’s registration. Starting with version 6.1, the output can continue with a description of a schematized event filter that is already known for the GUID.

The function returns STATUS_INVALID_PARAMETER if any of the following are true:

Microsoft’s name for the 0xA0 bytes of fixed-size input and output is not known. Indeed, it is not known that Microsoft defines them as one structure. They certainly are interpreted in parts, not that Microsoft’s names are known for these either. A 0x28-byte header describes the event provider, both to tell the kernel about the provider and to return the handle that the kernel creates for the registration.  

Offset Definition Versions Remarks
0x00 the provider’s GUID 6.0 and higher input
0x10 an ETW_NOTIFICATION_TYPE 6.0 and higher input
0x14 the process’s 16-bit index for this registration 6.0 and higher input
0x18 a 64-bit allowance for a HANDLE to the provider’s ETW_REG_ENTRY 6.0 and higher output
0x20 apparently reserved for future definition 6.0 to 6.2  
a 64-bit allowance for the address of the provider’s callback function 6.3 and higher input

Incidentally, that the kernel is nowadays told of the provider’s user-mode callback function is not because the kernel has any role in the calling back. It is instead so that the kernel can identify the calling module. This matters in version 6.3 and higher because tracing sessions can get system-defined events written to them to record which providers were enabled for the session.

Though the remaining 0x78 bytes are expected on input, they are used only for output. Moreover, though the header looks to be particular to this function code, the remaining 0x78 bytes (and any variable-size data that follows) are shared with function code 0x11 to describe how a provider is enabled for a tracing session. In this case, the tracing session that’s described as output for function code 0x0F is the logger for which the provider was most recently enabled. Microsoft’s name for this structure is not known. Relevant routines refer to it as an enable notification: the name ETW_ENABLE_NOTIFICATION_PACKET would follow a pattern that might be induced from the known name for another structure that begins with an ETW_NOTIFICATION_HEADER, but is too much of a guess to commit to here.

Offset Definition Versions Remarks
0x00 an ETW_NOTIFICATION_HEADER 6.0 and higher NotificationSize for output
(but see note after table)
0x48 a TRACE_ENABLE_INFO 6.0 and higher output
0x68 a TRACE_ENABLE_CONTEXT 6.0 and higher output
0x70 0 to disable events;
1 to enable events
6.0 and higher output
0x74 apparently reserved for future definition 6.0 only  
indicator of what data follows 6.1 and higher output

The ETW_NOTIFICATION_HEADER appears to be just a formality—literally from sharing the form of data that does matter for function code 0x11. For function code 0x0F, the version 6.0 kernel ignores this header completely and in no version does the kernel set anything in this header other than the NotificationSize. Even then, versions 6.1 and 6.2 set it only if following with variable-size data. It’s set always on success in versions 6.3 and higher, but to the size of the whole output when surely what would be expected is the size just from the notification header onwards. (NTDLL, however, expects nothing.)

What can follow this fixed-size data is a description of the schematized event filters that yet apply to the provider. The format for this description is version-dependent. In versions 6.1 and 6.2, it is a sequence of EVENT_FILTER_HEADER structures, each introducing variable-size data. The header’s Size member is the total size, in bytes, of the header and data. Its NextOffset member is the possibly larger number of bytes from the header to the next header, or is zero in the last header. In version 6.3 and higher, there is first an EVENT_FILTER_DESCRIPTOR. Its Ptr member is the number of bytes from the notification header to the EVENT_FILTER_HEADER sequence. Its Size is that of the sequence. This change of format also affects the indicator at offset 0x74. In versions 6.1 and 6.2, this is 0x80000000, which may be intended as the filter type, specifically EVENT_FILTER_TYPE_SCHEMATIZED. In later versions, the indicator is boolean and the filter type is explicitly in the Type member of the EVENT_FILTER_DESCRIPTOR.

This function code is expressly not for registering the security provider. If the provider’s GUID is {54849625-5478-4994-A5BA-3E3B0328C30D}, which Microsoft represents symbolically as SecurityProviderGuid, the function fails, returning STATUS_ACCESS_DENIED.

Function Code 0x10

The caller asks to retrieve a notification. No input is expected. The output buffer is to receive a fixed-size ETW_NOTIFICATION_HEADER and some variable amount of other data. The function returns STATUS_INVALID_PARAMETER if any of the following is true:

When a process first registers as an event provider, it gets a thread in which NTDLL polls for notifications. The precise mechanics of this is highly version-dependent: in versions 6.0 to 6.2 inclusive, the thread routine is an NTDLL export! In version 6.3 and higher, NTDLL has a thread from the thread pool share the polling with arbitrary other work. It polls for notifications whenever a corresponding event is signalled (see function code 0x1B). The EPROCESS for the calling process must have an EtwDataSource, else the function returns STATUS_INVALID_PARAMETER. Moreover, the data source must have at least one pending notification, else the function returns STATUS_NO_MORE_ENTRIES. Such notifications are held in a queue. Each is represented by an ETW_QUEUE_ENTRY whose DataBlock points to the notification data. This data is what the caller seeks. It begins with an ETW_NOTIFICATION_HEADER whose NotificationSize is the total size, in bytes, of the data. If this is too big for the caller’s buffer, the function returns STATUS_BUFFER_TOO_SMALL having set the return size (and the caller who expects to do anything useful would better repeat the call but with a suitably bigger buffer). Otherwise, the function copies the notification data to the caller’s buffer and returns success. If the notification queue is not empty, the return is STATUS_MORE_ENTRIES (so that the caller can know to poll again rather than wait on the event).

Function Code 0x11

Thie caller is sending an event notification. The input buffer must provide a fixed-size ETW_NOTIFICATION_HEADER and some variable amount of other data. The output buffer is to receive exactly an ETWP_NOTIFICATION_HEADER. In version 10.0, the function returns STATUS_INVALID_PARAMETER if any of the following are true:

The output on success is necessarily 0x48 bytes.

Function code 0x12

The input buffer must provide a fixed-size ETW_NOTIFICATION_HEADER and some variable amount of other data. The function returns STATUS_INVALID_PARAMETER if either of the following is true:

Function Code 0x13

If the input buffer does not provide exactly 8 bytes, the function returns STATUS_INVALID_PARAMETER.

EtwWdiSemUpdate (0x14)

This function code restarts the Scenario Event Mapper (SEM). It takes no input and produces no output. If given any input or output buffer, the function returns STATUS_INVALID_PARAMETER.

Managing the SEM is not widely permitted. The caller must be executing as a member of the Administrators group or as the local system account or as the Diagnostic Policy Service (DPS). Failure for the access check is failure for the function. Except that one-time initialisation is not repeated, restarting the SEM means shutting it down (if it is enabled) and starting as if for the first time. Regrettably, though the SEM is surely a very important (yet almost entirely undocumented) add-on to ETW, it lies far outside the scope of this note.

Function Code 0x15

This function code has no immediate buffer validation.

Function Code 0x16

If the input buffer does not provide exactly 0x10 bytes, the function returns STATUS_INVALID_PARAMETER.

Function Code 0x17

This function code has no immediate buffer validation.

Function Code 0x18

This function expects no input and produces no output: if given either, the function returns STATUS_INVALID_PARAMETER.

Function Code 0x19

This function code queries a given logger for its reference time. The meaningful input is a 16-bit logger ID, but it must be presented as 32 bits. The expected output is the ReferenceTime from the logger’s WMI_LOGGER_CONTEXT.

If the input buffer is not exactly 4 bytes or the output buffer does not allow for exactly the 0x10 bytes of an ETW_REF_CLOCK, the function returns STATUS_INVALID_PARAMETER. If the input does not select a running logger, the function returns STATUS_WMI_INSTANCE_NOT_FOUND.

Function Code 0x1A

If the input buffer does not provide exactly 8 bytes, the function returns STATUS_INVALID_PARAMETER. The output on success is necessarily zero bytes.

Function Code 0x1B

The calling process is telling the kernel of an event to signal whenever the process has a notification to retrieve (see function code 0x10). The input buffer must provide a handle to the event. Only 32 bits are provided for this handle even on 64-bit Windows. The function returns STATUS_INVALID_PARAMETER if either:

Each process can have only one such event, which is anyway just a refinement of an older mechanism for managing data for notifications. If this mechanism is already initialised, the function returns STATUS_UNSUCCESSFUL. Only now that it looks like the event can be added does the function check that the handle actually does represent an event object and grant EVENT_MODIFY_STATE access. Failure is fatal to the function. The event is suitable for use, but if the notification mechanism has meanwhile got initialised by other means, the function forgets this event and returns STATUS_ALREADY_REGISTERED.

Function Code 0x1C

If the input buffer does not provide at least 8 bytes, the function returns STATUS_INVALID_PARAMETER. The second dword of input is to be a number of 0x10-byte entries that follow. If the total size does not equal InBufferLen, the function returns STATUS_INVALID_PARAMETER.

Function Code 0x1E

If the input buffer does not provide exactly 0x18 bytes or the output buffer does not allow for between 0x78 and 0x00010000 bytes inclusive, the function returns STATUS_INVALID_PARAMETER.

Function Code 0x1F

The caller is indicating whether the given event provider is sufficiently modern that the Type member in its EVENT_DATA_DESCRIPTOR structures is to be treated as meaningful. This Type member is defined in space that used to be Reserved but which will not certainly have been initialised by all event providers. Its use therefore needs to be explicitly enabled.

The input buffer provides (at offset 0x00) a HANDLE to the provider and (at offset 0x08) a BOOLEAN setting for whether to use types in descriptors. Though Microsoft surely has a structure for this, its name is not known. No output is expected. The function returns STATUS_INVALID_PARAMETER if either of the following is true:

The given HANDLE is to be an Object Manager handle for an event registration object, i.e., an ETW_REG_ENTRY, that grants TRACELOG_REGISTER_GUIDS access to user-mode callers. Failure to reference an event registration object from the given HANDLE is failure for the function. The whole of the function’s essential work is to set the given BOOLEAN into the UseDescriptorType member of the referenced ETW_REG_ENTRY. If the given BOOLEAN is anything but TRUE or FALSE, the function fails, returning STATUS_INVALID_PARAMETER.

Function Code 0x20

This function code has no immediate buffer validation.

Function Code 0x21

If the input buffer does not provide exactly 0x10 bytes, the function returns STATUS_INVALID_PARAMETER.

Function Code 0x22

If the input buffer does not provide exactly 8 bytes, the function returns STATUS_INVALID_PARAMETER.