IRP Extensions

For Windows 8, an I/O Request Packet (IRP) can have an extension which can carry multiple new parameters to drivers that care (and back, too). It seems to be important to Microsoft that the delivery should by-pass drivers along the way, for although they see all the I/O requests go up and down the device-object stacks, they don’t know of the IRP extension that holds the extra parameters. Microsoft keeps IRP extensions secret.

Let’s be clear that Microsoft has gone to unusual trouble over this. More is going on here than just Microsoft keeping some structures and functions as internal, not documenting them and perhaps not even declaring them in the headers that Microsoft publishes in the Windows Driver Kit (WDK). More is going on than just Microsoft defining some structure as semi-opaque with members defined in plain sight but as being reserved for the system’s use. IRP extensions are more reserved than that. They count among the relatively few features that have Microsoft define some structure differently in public and private.

To support IRP extensions, Microsoft has two definitions of the age-old IRP structure: one public and one private. Symbol files for the kernel in Windows 8 and higher show clearly that the kernel is now built with an IRP whose Tail.Overlay ends with a pointer named IrpExtension. For everyone else, the end that shows in WDM.H from the WDK is the same OriginalFileObject that has been at the end since the definition in NTDDK.H from the Device Driver Kit (DDK) for Windows NT 3.51. The new IrpExtension isn’t blocked out of the IRP definition in WDM.H by conditional compilation: it just isn’t there.

That said, the IrpExtension does show in the NTOSP.H file that Microsoft disclosed relatively recently in the Enterprise WDK for Windows 10. Since this file’s in a subdirectory of a directory named “um”, as if for user-mode programming in contrast to “km”, its publication doesn’t seem intended for writing device drivers or file system drivers or anything else that ever sees an IRP. Yet it has the enhanced definition—along with comments that make clear that IrpExtension is edited out of WDM.H.

Of course, Microsoft does not play these games for fun or mischief. Especially as devices become ever more capable, there’s obvious merit to expanding what can be sent with an I/O request and what sort of detailed error information (such as a SCSI sense key and additional sense codes) can be returned. The example that brought IRP extensions to my attention is of hybrid disks that have non-volatile caches that can be accessed without the penalty of waiting for heads to seek or disks to spin. These devices can take a hint to use one form of storage rather than another. That Windows 8.1 can send such hints when writing to a paging file is surely a great help, but the hint needs to go with the data—not in a separate control request, but as control data in the same request that supplies the data that’s to be stored. The hint goes with the write request as an IRP extension, set by the kernel’s Memory Manager when forming the IRP for an asynchronous page write and then fished out when the request reaches such drivers as CLASSPNP.SYS, RDYBOOST.SYS and STORPORT.SYS. Perhaps only drivers from Microsoft could use the hint even if others knew of it.

Implementation

The IrpExtension is declared as pointing to void. What it actually points to, if anything, is an undocumented structure named IOP_IRP_EXTENSION that can hold multiple types of extension data. In Windows 8, just the one type is defined, for associating the IRP with an Activity ID to help with performance monitoring of I/O requests. But the definition of that structure, as known from published symbol files, makes clear that multiple types were planned from the start.

One type of extension that was added for Windows 8.1 is small enough to fit in the IRP in place of the IrpExtension pointer if it’s the only type of extension that is yet set. New bits in the AllocationFlags of the IRP distinguish the cases. (Microsoft’s names for these new bits are known from NTOSP.H.) When IRP_EXTENSION_GENERIC_ONLY (0x80) is set, the IRP has only a generic IRP extension whose four bytes overlay the IrpExtension pointer. When IrpExtension is not NULL, it points to an IOP_IRP_EXTENSION. If this is a separate allocation (from non-paged no-execute pool), the IRP_EXTENSION_ALLOCATED bit (0x40) is set in the AllocationFlags. The IOP_IRP_EXTENSION can instead be in the same memory block as the IRP, as an expansion of space allowed for the I/O stack locations.

An IRP that has an IOP_IRP_EXTENSION can have multiple types of extension data (though some types are mutually exclusive, if only for now). To each type there corresponds one bit in the TypesAllocated member. The bit is set if the extension has the type. Microsoft’s names for these bits are not known.

Functions

Of course, even Microsoft’s device driver programmers who know of IRP extensions do not access IrpExtension directly and depend on these implementation details, which Microsoft may change readily, and has already. Instead, they call functions, which Microsoft may change but rarely does. There are rather many:

None are documented (though an earlier set of functions such as IoGetActivityIdIrp, which use IRP extensions internally, are). Functions whose names are shaded orange above are declared in one or other of NTDDK.H and NTIFS.H from the ordinary WDK, even for the version that first exports the functions. The others are highlighted orange to draw attention to them. They too are declared but obscurely—in one or both of NTOSP.H and NTOSIFS.H from the “um” tree of the Enterprise WDK for an update of Windows 10.