Trace Headers

An event provider delivers an event to Event Tracing for Windows (ETW) by specifying some general description of the event in combination with arbitrary data that is specific to that occurrence of the event. ETW marshals the description and the data, along with system properties such as times and the current process and thread, into one data block in a trace buffer. Each such buffer is a fixed-size WMI_BUFFER_HEADER and a sequence of these variable-size data blocks, each with 8-byte alignment. If the event tracing session is configured to flush the trace buffers to an Event Trace Log (ETL) file, the buffers and thus also the events in this raw form become more than a transitory implementation detail and instead persist for easy inspection.

Broadly speaking, each data block that describes an event is in two parts:

ETW has accreted through the decades by fitting together several schemes that appear to have been developed not quite independently but also not in continuous progression. This can be seen, for instance, in the documentation’s sepearate talk of classic and manifest-based event providers, each with their own sets of API functions. An outcome for the implementation is that several types of header are defined for events as they appear in the trace buffers. They fall naturally into two sets according to whether they begin with a size. This difference also aligns roughly with whether their use is external or internal, and with some sense of their being old or new. For now, however, it suffices just to list the known headers alphabetically, which has the advantage of starting with the most prosaically named:

This article is concerned with differentiating the headers according to the few bytes that all have in common near their start. For details of how each header continues, follow the links.

Documentation Status

The EVENT_HEADER and EVENT_TRACE_HEADER are documented. C-language definitions are in EVNTCONS.H and EVNTRACE.H, respectively. None of the other trace headers are documented, but C-language definitions are published in the NTWMI.H header from some editions of the Windows Driver Kit (WDK) for Windows 10.

That two of the trace headers are documented is plainly not so that programmers or other interested computer users can read the events directly in ETL files. Of all the trace headers, these two have extended roles in an event’s lifetime so that they are not just internal creations for logging an event into the trace buffers.

The EVENT_TRACE_HEADER is involved right from the start for some types of event. It is the way that a so-called classic event provider describes an event to ETW through the documented user-mode API function TraceEvent and through the documented kernel export IoWMIWriteEvent. Note that these functions, though they are not formally deprecated, have some sense of legacy to them.

Both the documented structures have essential roles at the end of an event’s lifetime, no matter what type of header the event had while in the trace buffers or still has if persisting in an ETL file. When an event is described to an event consumer through callback functions that may be specified in the documented EVENT_TRACE_LOGFILE when calling the documented OpenTrace function, what the event consumer sees is a translation of the event. To get the callbacks called, the event consumer calls the ProcessTrace function, which is historically an ADVAPI32 export but is nowadays implemented in SECHOST. The older style of callback gets an EVENT_TRACE_HEADER as the beginning of an EVENT_TRACE. The newer callback gets an EVENT_HEADER as the beginning of an EVENT_RECORD.  

Layout

Several of the trace headers show signs of having developed from the WNODE_HEADER structure. Indeed, the EVENT_TRACE_HEADER has the same size and duplicates several members at the same offsets. What counts for differentiating the trace headers from the WNODE_HEADER and among themselves is just the first four bytes. Both the EVENT_TRACE_HEADER and the EVENT_INSTANCE_GUID_HEADER begin as: 

Offset Definition
0x00
USHORT Size;
0x02
union {
    USHORT FieldTypeFlags;
    struct {
        UCHAR HeaderType;
        UCHAR MarkerFlags;
    };
};

The EVENT_HEADER begins only a little differently:

Offset Definition
0x00
USHORT Size;
0x02
USHORT HeaderType;

It might not seem so at first, but the MESSAGE_TRACE_HEADER begins essentially the same way. See especially that the Size remains at the very beginning:

Offset Definition
0x00
union {
    ULONG Marker;
    struct {
        USHORT Size;
        UCHAR Reserved;
        UCHAR Version;
    };
};

The trace headers for system and kernel events, namely SYSTEM_TRACE_HEADER and PERFINFO_TRACE_HEADER, have the Marker but do not begin with the Size:

Offset Definition
0x00
union {
    ULONG Marker;
    struct {
        USHORT Version;
        UCHAR HeaderType;
        UCHAR Flags;
    };
};

For these, incidentally, the Size moves to offset 0x04 as one member of a WMI_TRACE_PACKET.

What’s common to all these headers, wherever they place the Size, are the bytes at offsets 0x02 and 0x03. The names change, such that what may seem most common to them is that even for the documented headers these bytes are documented only as Reserved. Still, they are the key to differentiating the headers.

Flags and Marker Flags

In the WNODE_HEADER, the whole of the first dword is a BufferSize and would be expected to have the most significant bit, or even bits, clear. The adaptation for event tracing makes bit fields of the high byte, whether that byte is named Flags, MarkerFlags or even Version, or isn’t even singled out. NTWMI.H defines the following as bits of the whole dword:

Value Name Remarks
0x80000000 TRACE_HEADER_FLAG set in all trace headers
0x40000000 TRACE_HEADER_EVENT_TRACE set in all trace headers except MESSAGE_TRACE_HEADER
0x10000000 TRACE_MESSAGE set only in MESSAGE_TRACE_HEADER

That MESSAGE_TRACE_HEADER is singled out has no known essential purpose and may be a sign of separate development. It dates from version 5.1 by which time the SYSTEM_TRACE_HEADER and EVENT_TRACE_HEADER were already established. Please be aware, however, that I do not mean to examine the history with any care any time soon.

Header Type

Given that the header’s first dword has both the TRACE_HEADER_FLAG and TRACE_HEADER_EVENT_TRACE bits set, the HeaderType, as a byte, tells what sort of header this first dword begins:

Value Name Continuation
0x01 TRACE_HEADER_TYPE_SYSTEM32 as SYSTEM_TRACE_HEADER followed by 32-bit event data
0x02 TRACE_HEADER_TYPE_SYSTEM64 as SYSTEM_TRACE_HEADER followed by 64-bit event data
0x03 TRACE_HEADER_TYPE_COMPACT32 as compact SYSTEM_TRACE_HEADER followed by 32-bit event data
0x04 TRACE_HEADER_TYPE_COMPACT64 as compact SYSTEM_TRACE_HEADER followed by 64-bit event data
0x0A TRACE_HEADER_TYPE_FULL_HEADER32 as EVENT_TRACE_HEADER followed by 32-bit event data
0x0B TRACE_HEADER_TYPE_INSTANCE32 as EVENT_INSTANCE_GUID_HEADER followed by 32-bit event data
0x0D TRACE_HEADER_TYPE_ERROR defined as used, but use presently not known
0x10 TRACE_HEADER_TYPE_PERFINFO32 as PERFINFO_TRACE_HEADER with 32-bit event data as Data array
0x11 TRACE_HEADER_TYPE_PERFINFO64 as PERFINFO_TRACE_HEADER with 64-bit event data as Data array
0x12 TRACE_HEADER_TYPE_EVENT_HEADER32 as EVENT_HEADER followed by 32-bit event data
0x13 TRACE_HEADER_TYPE_EVENT_HEADER64 as EVENT_HEADER followed by 64-bit event data
0x14 TRACE_HEADER_TYPE_FULL_HEADER64 as EVENT_TRACE_HEADER followed by 64-bit event data
0x15 TRACE_HEADER_TYPE_INSTANCE64 as EVENT_INSTANCE_GUID_HEADER followed by 64-bit event data

NTWMI.H defines 0x0F as TRACE_HEADER_MESSAGE but this value exists as the HeaderType only as a confection during user-mode translation of events so that the MESSAGE_TRACE_HEADER can be treated as just another case of the headers that have a HeaderType all along.

Marker

Also defined in NTWMI.H are combinations of the Flags and HeaderType, ready for use as a basis for the Marker in the internal headers:

Value (x86) Value (x64) Name
0xC0010000 0xC0020000 SYSTEM_TRACE_MARKER
0xC0030000 0xC0040000 COMPACT_TRACE_MARKER
0xC0100000 0xC0110000 PERFINFO_TRACE_MARKER

The NTETW.H header, also available only in some editions of the WDK for Windows 10, defines some other combinations similarly:

Value (x86) Value (x64) Name
0xC00A0000 0xC0140000 TRACE_HEADER_FULL
0xC00B0000 0xC0150000 TRACE_HEADER_INSTANCE

and yet more only as 16-bit integers for the headers that have a 16-bit HeaderType:

Value (x86) Value (x64) Name
0xC00D 0xC00D EVENT_HEADER_ERROR
0xC012 0xC013 EVENT_HEADER_EVENT

Version

To these markers must yet be added the Size or Version as the low word. The Size is straightforwardly 16 bits and is the total, in bytes, of the header and whatever variable-size data follows. Though NTWMI.H defines a 16-bit Version for the other headers, the kernel treats it as 8-bit and this is known also for so-called system events that are passed from user mode via the NtTraceEvent function: the Version is then the low 8 bits of the Flags argument, which NTWMI.H formalises by defining a macro ETW_SYSTEM_EVENT_VERSION_MASK as 0x000000FF. The high 8 bits of the Version can anyway be treated as more flags. NTWMI.H defines the following, again as bits for the Marker:

Value Name
0x00000700 TRACE_HEADER_PMC_COUNTERS_MASK
0x00008000 TRACE_HEADER_PEBS_INDEX_FLAG

The kernel sets these only for some very particular events that begin with the PERFINFO_TRACE_HEADER, but the translation by SECHOST recognises them in the SYSTEM_TRACE_HEADER also.