The Scenario Event Mapper (SEM)

The kernel’s involvement with the Windows Diagnostic Infrastructure (WDI) is mostly to implement the Scenario Event Mapper (SEM). For this purpose, a scenario is a sequence of events to log should they ever occur. The scenario has one start event and multiple end events. From start to end, or for some configurable time, selected events from selected event providers, here called context events and context providers, are also logged.

Programming

Though the WDI has “infrastructure” in its name, it is for its programmatic operation more a superstructure on Event Tracing for Windows (ETW). Scenarios can be started and stopped by undocumented variations EtwWriteStartScenario and EtwWriteEndScenario of the documented EtwWrite function. There is a user-mode interface too, with slightly different names: EventWriteStartScenario, etc., for the ADVAPI32 exports; and EtwEventWriteStartScenario, etc., for the NTDLL exports. Some very limited management can be exercised through two of the many function codes that are recognised by the NtTraceControl function.

That said, though the “scenario” functions take the same arguments as EtwWrite and retain its prefix, and though the only C-language declarations that Microsoft has disclosed in public are in a header named NTETW.H, section contributions in the public symbol files for the kernel show that these “scenario” variations are built not in the “etw” subdirectory with the rest of the EtwWrite family but in a “wdi” subdirectory—specifically in a source file that builds to an object file named “wdiapi.obj”. Plainly, Microsoft’s programmers regard these functions as an API for the WDI.

Configurability

How the SEM gives WDI its primary usefulness is in avoiding programmatic operation. The SEM is configured from the registry.

An unpublished early version of this page, from 2008 when WDI was new for Windows Vista, even listed the configuration for a newly installed Windows. A decade later, for Windows 10, this initial configuration is of course much too large for its listing here to be practical. But doesn’t its evidently increasing use by Microsoft make it all the more astonishing that nobody outside Microsoft seems to have detailed it for public review? I don’t say that what’s collected in the scenarios is a worry for anyone’s privacy, but neither is it credible that Microsoft has gone to this trouble for information that is not somehow useful or important. Or not, for surely the world’s experts in computer forensics must have picked over it by now to see what useful information Microsoft has Windows keep about how and when it started.

Registry settings that affect the SEM are in and under what will sometimes be referred to here as the WDI control key. Two settings in a subkey named Config govern overall behaviour:

Key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\WDI\Config
Value: SEMEnabled
Type: REG_DWORD

The default is that the SEM is enabled. To disable the SEM, SEMEnabled must be present and evaluate to zero: then, no more registry keys and values matter.

Key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\WDI\Config
Value: SEMTimeoutValue
Type: REG_DWORD

The data is the number of minutes after which the SEM may stop a scenario for which timing out is enabled. The default is 10. Zero means that no scenarios time out. While the SEM is enabled, it checks its state periodically. Ending scenarios by timing out is only one task. The period is one tenth of the timeout. If zero is set as the timeout, then the period is 1 minute, i.e., as if SEMTimeoutValue had defaulted.

Group Policy

Whether the SEM is enabled is subject to Group Policy, both overall and for particular scenarios. These settings are in and under what may as well be called the WDI policy key.

Key: HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows\WDI
Value: ScenarioExecutionEnabled
Type: REG_DWORD

Zero disables the SEM (such that no more registry keys and values matter). Anything else enables the SEM for all the scenarios that are defined under the WDI control key (see below).

Absence of ScenarioExecutionEnabled in the WDI policy key allows that scenarios can be selectively disabled. The affected scenarios each have their own subkey. The name of the subkey is the scenario’s GUID in string form between curly braces. A subkey that is not in this form is ignored, except that a subkey whose name exceeds 64 characters is an error and disables the SEM.

Key: HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows\WDI\{scenario}
Value: ScenarioExecutionEnabled
Type: REG_DWORD

The default is that the scenario is enabled. To disable the scenario, ScenarioExecutionEnabled in the scenario’s subkey must be zero. At most 64 scenarios can be disabled through these subkeys. Exceeding this disables the SEM.

Scenarios

The defined scenarios are listed under the Scenarios subkey of the WDI control key. The scenario’s own subkey is the scenario’s GUID in string form between curly braces. A subkey that is not in this form is ignored, except that a subkey whose name exceeds 64 characters is an error and disables the SEM.

Key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\WDI\Scenarios\{scenario}
  subkeys and values as scenario configuration and definition

If the scenario’s GUID is configured as disabled by Group Policy (see above), then everything else about the scenario’s subkey is ignored, as if the scenario were not defined. The scenario can also be disabled by a value in the Config subkey:

Key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\WDI\Scenarios\{scenario}\Config
Value: ScenarioExecutionEnabled
Type: REG_DWORD

The default is that the scenario is enabled. To disable the scenario, ScenarioExecutionEnabled must be zero.

The scenario’s Config subkey can also have a setting that specifies whether the scenario can end by timing out.

Key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\WDI\Scenarios\{scenario}\Config
Value: ScenarioTimeoutEnabled
Type: REG_DWORD

The timing out of scenarios is enabled by default. To stop a scenario from timing out, ScenarioTimeoutEnabled must be zero.

Instrumentation

How a scenario ever starts, let alone what effect it has between its start and end, is defined under a subkey named Instrumentation. A scenario that does not have this subkey is in effect not defined. Beneath the Instrumentation subkey, the definition can extend to three levels of subkeys. The first specifies a start event. For each, a second level differentiates end events and context providers. A third level lists the multiple end events and the multiple context providers. Providers are specified by GUID. Events are specified a 16-bit event ID relative to their provider. Both providers and events can have Level, Keyword and EnableProperty values. Providers can have a CaptureState value.

It is perhaps as well to note here that the implementation has hard-coded limits:

This last limit, of distinct providers, is only 128 in versions 6.0 and 6.1.

Start Event

A scenario’s hierarchical definition in its Instrumentation subkey starts with subkeys that specify start events:

Key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\WDI\Scenarios\{scenario}\Instrumentation
Subkey: {start-provider};start-event

The ideal is that the subkey’s name has three parts:

The start event, along with end events and context providers that might follow from it, is explicitly rejected if:

That white space immediately before start-event is ignored, as are characters other than decimal digits after, is here taken as incidental. Also disregarded here, but more as undefined than unintended, is the interpretation if the decimal digits continue too far for 16-bit evaluation. Possibly also unintended is that start-event evaluates as zero in default of being interpretable as a decimal number. That said, explicit allowance is made for start-event to be exactly an asterisk (see below).

The subkey for a start event can have any combination of Level, Keyword and EnableProperty values. So can subkeys for end events and for context providers: they are taken altogether, later.

End Events

To be useful, a start event must have at least one end event. The defined end events are listed as subkeys of an EndEvents subkey.

Key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\WDI\Scenarios\{scenario}\Instrumentation
Subkey: {start-provider};start-event \EndEvents\{end-provider};end-event

The end-provider and end-event are respectively the GUID of the event provider and the 16-bit event ID. Interpretation follows that of the start-provider and start-event, but now to define an end event. The subkey for an end event may also have any combination of Level, Keyword and EnableProperty values.

If the key for a start event has no EndEvents or if this subkey defines no end event, then the start event can never start a scenario.

If the key for a start event has an asterisk for its event ID, then its end events are parsed but it is rejected anyway (unless it has no end events). Since a newly installed Windows 10 (original) has two examples, Microsoft presumably has some purpose for the asterisk, perhaps beyond just marking a place: what, however, is unclear.

Context Providers

Much of the point to a scenario is to use the start event as signalling that events from other event providers may be especially useful to collect, whether for a time or until the occurrence of a matching end event. The SEM enables these providers for the WdiContextLog. Those that are defined for a given start event are listed as subkeys of the start event’s ContextProviders subkey:

Key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\WDI\Scenarios\{scenario}\Instrumentation
Subkey: {start-provider};start-event \ContextProviders\{context-provider}

The context-provider is the GUID of an event provider that the SEM is to enable for the duration of the scenario. How the provider is enabled can be configured through values named Level, Keyword, CaptureState and EnableProperty.

A start event need not have a ContextProviders subkey and this subkey need not define any context providers. The start event, along with its end events and other context providers, is explicitly rejected if:

Event and Provider Values

The SEM’s business with the start and end events and with the context providers is to trace events to either or both of the SEM’s dedicated loggers. Start and end events go to the DiagLog and possibly also to the WdiContextLog. Between a scenario’s start and end, events from the corresponding context providers go to the WdiContextLog. The SEM enables the providers of start and end events when the SEM starts. As scenarios start and end, the SEM enables and disables the applicable context providers. Mostly it won’t be wanted that the SEM’s loggers receive all events that the providers ever can send. The volume of analysis is very much reduced if whenever the SEM enables a provider for tracing to a logger, it can refine what gets traced.

All three levels of subkey allow for this through registry values that supply arguments for the EtwEnableTrace function. (That this function is undocumented is a story in itself for what it says of Microsoft’s attitude to ETW’s kernel-mode use by anyone other than Microsoft. Let’s just say it’s roughly a kernel-mode equivalent of the documented EventTraceEx.) Three registry values, Level, Keyword and EnableProperty correspond to the Level, MatchAnyKeyword and EnableProperty arguments.

Event providers define for each of their events an 8-bit level. When the provider is enabled for a logger, a level may be set for the logger so that it ignores events whose level is higher than the logger’s. The natural understanding is that numerically higher levels are for events that are less important. Whenever the SEM enables a provider, it sets the Level argument to the highest Level value from any applicable registry key.

Key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\WDI\Scenarios\{scenario}\Instrumentation
Subkeys: {start-provider};start-event
{start-provider};start-event \EndEvents\{end-provider};end-event
{start-provider};start-event \ContextProviders\{context-provider}
Value: Level
Type: REG_DWORD

Zero is reinterpreted as 0xFF. For anything else, only the low 8 bits matter. The default is 0xFF, as if the logger is to trace all levels of event.

Event providers also define for each of their events a 64-bit mask of flags that act as something like keywords. When the provider is enabled for a logger, a mask of keywords may be set for the logger so that it ignores events that have no keywords in common. Whenever the SEM enables a provider, it sets the MatchAnyKeyword argument to the bitwise-OR of the Keyword values from all applicable registry keys.

Key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\WDI\Scenarios\{scenario}\Instrumentation
Subkeys: {start-provider};start-event
{start-provider};start-event \EndEvents\{end-provider};end-event
{start-provider};start-event \ContextProviders\{context-provider}
Value: Keyword
Type: REG_QWORD

Zero is reinterpreted as setting all bits. The default is that all bits are set, as if the logger is to trace all events.

When a provider is enabled for a logger, it can be configured so that system-supplied data is added to every traced event as an extended data item. Whenever the SEM enables a provider, it sets the EnableProperty argument to the bitwise-OR of the EnableProperty values from all applicable registry keys.

Key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\WDI\Scenarios\{scenario}\Instrumentation
Subkeys: {start-provider};start-event
{start-provider};start-event \EndEvents\{end-provider};end-event
{start-provider};start-event \ContextProviders\{context-provider}
Value: EnableProperty
Type: REG_DWORD

The default is zero, as if to add no extended data items.

Context Provider Values

One more value is relevant but only in a subkey that defines a context provider:

Key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\WDI\Scenarios\{scenario}\Instrumentation
Subkey: {start-provider};start-event \ContextProviders\{context-provider}
Value: CaptureState
Type: REG_DWORD

The default is zero. The meaningful values are 1 and 2. The significance and usefulness is yet to be understood.

Installed Scenarios

There always have been too many scenario subkeys to interpret and list, and their number does not decrease as the years advance. What’s still just about practical, however, is to list the ones that define scenarios in the sense of having a start event that triggers the enabling of context providers until the occurrence of some end event. (There is also some merit that with this interpretation, the list hardly differs from what I prepared in 2008.)

The following such scenarios are already defined when Windows is installed. The format for the start and end events is: the event provider’s name; a semicolon; the event ID. The name is not that of the event provider, strictly speaking, but of the publisher as understood for such tools as WINEVT, here taken as the most readily useful interpretation of the GUID.

This still leaves four of the context providers to be given only by GUID. No name for them is known from the registry or from any manifest or MOF file. They are, for instance, not listed as publishers by Microsoft’s WEVTUTIL program or as event providers that the Reliability and Performance Monitor can add to any of its Event Trace Sessions. However, each GUID is named in Microsoft’s symbol files:

Provider GUID Symbolic Name
{41932CAB-7E12-40D6-A728-62D30E054593} MS_Kernel_BootDiagnostics_UserProxy_Provider
{7EFE4AB3-990D-4350-A878-CD8772888199} MS_Kernel_BootDiagnostics_SystemProxy_Provider
{AD5C7A10-4E08-45E1-81B5-CB5EB6EC8917} MS_Kernel_ShutdownDiagnostics_Proxy_Provider
{B27A2C15-40F4-4EA3-9637-628FC612A1D0} MS_Kernel_SecondaryLogonDiagnostics_Proxy_Provider

Scenario {2698178D-FDAD-40AE-9D3C-1371703ADC5B}

This scenario has two start events—in fact the same event but from different sources that once upon a time were one—but the end events and context providers are set up exactly the same for both:

Start Events: Microsoft-Windows-Wininit;6001
Microsoft-Windows-Winlogon;6001
Level: 4
Keyword: 0x00000000`00020000
End Event: Microsoft-Windows-Kernel-Power;48 Level: 4
Keyword: 1
Context Providers: Microsoft-Windows-Services Level: 4
Keyword: all
Microsoft-Windows-Wininit Level: 4
Keyword: 0x00000000`00010000
Microsoft-Windows-Kernel-Power Level: 4
Keyword: 0x00000000`00000008
{AD5C7A10-4E08-45E1-81B5-CB5EB6EC8917} Level: 85
Keyword: zero
Microsoft-Windows-Winlogon Level: 4
Keyword: 0x00000000`00010000
Microsoft-Windows-Subsys-Csr Level: 4
Keyword: 0x00000000`00010000

This is as good an example as any to take of what the Scenario Event Mapper can help with. It’s also an example of how the understanding of a Windows feature quickly ties you to Microsoft’s diagnostics tools for lack of independent knowledge of the particular events that are used here to start and end the scenario or of the levels and keywords for the events that are logged as context. Of course, the ideal would be that (if only as customer service for system administrators) Microsoft would document the many events that its many components can create. Next might be that the world’s legions of independent experts at Windows internals have produced a catalogue for general use. None of this happens in any sort of real world, of course.

Here, the scenario helps with diagnosing problems as Windows shuts down. The relevant event providers in WINLOGON and WININIT define event 6001 specifically as a signal to start this diagnostics: Microsoft’s public symbol files for WINLOGON show that the EVENT_DESCRIPTOR for the event with ID 6001 is named WLDiagEvt_ShutdownDiagnostics_Start. The scenario, and thus the shutdown diagnostics, ends when the kernel writes an event whose EVENT_DESCRIPTOR is shown in the kernel’s symbol files to be named POP_ETW_EVENT_GRACEFUL_SHUTDOWN. All the while, six event providers trace a neatly delimited and potentially useful record of what happened (and thus some support for inferring what did not happen) while Windows was shutting down.

Scenario {533A67EB-9FB5-473D-B884-958CF4B9C4A3}

Continuing the theme of the SEM automating diagnostics collection at times when Windows necessarily gives limited visibility even to user-mode diagnostics tools, this scenario traces across power transitions. Public symbol files for the kernel show the start and end events as having event descriptors that are named POP_ETW_EVENT_POWERTRANSITION_START and POP_ETW_EVENT_POWERTRANSITION_END.  

Start Event: Microsoft-Windows-Kernel-Power;1 Level: 1
Keyword: 0x00000000`00000001
End Event: Microsoft-Windows-Kernel-Power;2 Level: 1
Keyword: 0x00000000`00000001
Context Providers: Microsoft-Windows-Kernel-Power Level: 4
Keyword: 0x00000000`00000004
Microsoft-Windows-HAL Level: 2
Keyword: 0x00000000`00000001

Scenario {67144949-5132-4859-8036-A737B43825D8}

Start Event: Microsoft-Windows-Kernel-BootDiagnostics;501 Level: 4
Keyword: all
End Event: Microsoft-Windows-Winlogon;5007 Level: 4
Keyword: 0x00000000`00020000
Context Providers: Microsoft-Windows-Services Level: 4
Keyword: all
Microsoft-Windows-Kernel-Prefetch Level: 4
Keyword: 0x00000000`00000020
{7EFE4AB3-990D-4350-A878-CD8772888199} Level: 85
Keyword: zero
Microsoft-Windows-Kernel-PnP Level: 4
Keyword: 0xE0000000`000FFFFF
Microsoft-Windows-Winlogon Level: 4
Keyword: 0x00000000`00010000

Scenario {86432A0B-3C7D-4DDF-A89C-172FAA90485D}

Start Event: Microsoft-Windows-Winlogon;5001 Level: 4
Keyword: 0x00000000`00020000
End Event: Microsoft-Windows-Winlogon;5002 Level: 4
Keyword: 0x00000000`00020000
Context Providers: Microsoft-Windows-Services Level: 4
Keyword: all
Microsoft-Windows-Shell-Core Level: 4
Keyword: 0x00000000`04000000
{41932CAB-7E12-40D6-A728-62D30E054593} Level: 85
Keyword: zero
Microsoft-Windows-Winlogon Level: 4
Keyword: 0x00000000`00010000

Scenario {924F25BC-020A-4CAD-A53E-4A49CFABB6B6}

Start Event: Microsoft-Windows-Kernel-PnP;223 Level: 4
Keyword: 0x00000000`00040000
End Event: Microsoft-Windows-Kernel-PnP;224 Level: 4
Keyword: 0x00000000`00040000

Scenario {9F41811A-0429-42AA-81B7-CFD4D968411F}

Start Event: Microsoft-Windows-Winlogon;5003 Level: 4
Keyword: 0x00000000`00020000
End Event: Microsoft-Windows-Diagnostics-Performance;10001 Level: 4
Keyword: 0x00000000`02000000
Context Providers: Microsoft-Windows-Services Level: 4
Keyword: all
Microsoft-Windows-Shell-Core Level: 4
Keyword: 0x00000000`04000000
{B27A2C15-40F4-4EA3-9637-628FC612A1D0} Level: 85
Keyword: zero
Microsoft-Windows-Winlogon Level: 4
Keyword: 0x00000000`00010000

Scenario {AA526ADC-508B-448A-B2D1-C59511CB59D4}

Start Event: Microsoft-Windows-Kernel-PnP;216 Level: 4
Keyword: 0x00000000`00008000
End Event: Microsoft-Windows-Kernel-PnP;218 Level: 4
Keyword: 0x00000000`00008000

Scenario {FFC42108-4920-4ACF-A4FC-8ABDCC68ADA4}

Start Event: Microsoft-Windows-Kernel-Power;1 Level: 1
Keyword: 0x00000000`00000001
End Event: Microsoft-Windows-Kernel-Power;2 Level: 1
Keyword: 0x00000000`00000001
Context Providers: Microsoft-Windows-DriverFrameworks-UserMode Level: 4
Keyword: 0x00000000`00000001
Microsoft-Windows-Kernel-Power Level: 4
Keyword: 0x00000000`00000008

This scenario has the same start event (and end event) as another, such that the two are merged. One context provider is common to both (but with different keywords). When this start event starts a scenario, three context providers are enabled.