Hosting a preview handler in WPF, correctly, part 2: interop

Continuing Hosting a preview handler in WPF, part 1: UI and file associations, we will do the major COM interop in this entry. Many people have failed to host the handler correctly because of careless implementation, so look carefully! Also, we will demonstrate some broken preview handlers, including ‘Adobe PDF Preview Handler for Vista’, ‘Microsoft Word previewer’ and ‘Microsoft PowerPoint previewer’.

TL; DR: Check out the code here.

Some UI tweaks

Before we continue, we will bind another command with keyboard accelerators. We’ll implement an ExitCommand that closes a preview window, and bind it to Esc. In the last episode, we forgot to implement PreviewStatusText in PreviewWindow, we’ll also need that. And lastly, we will need to handle PreviewWindow’s closing. These changes are reflected in this commit.

Finish COM interop

This is where we do the actual heavy work. We’ll need to read Building Preview Handlers and figure out how preview handlers are used, so that we can correctly use them.

The COM interfaces we will need to import are IInitializeWithStream, IInitializeWithFile, IInitializeWithItem, IObjectWithSite, IPreviewHandler, IPreviewHandlerVisuals and IPreviewHandlerFrame. We will also need RECT, LOGFONTW and ACCEL structures. For Win32 functions, we will need CreateAcceleratorTable.

To ease our managed host, I create an interface, IPreviewHandlerManagedFrame. It has GetAcceleratorTable and TranslateAccelerator methods, which are easier to implement (the CreateAcceleratorTable call is abstracted away). It also has two read-only properties, WindowHandle and PreviewerBounds, which are got when needed.

Finally, I created a class, PreviewHandler to wrap a preview handler. Its constructor takes the preview handler CLSID and an IPreviewHandlerManagedFrame, prepares the preview handler object, adapts the frame to an IPreviewHandlerFrame. Next, the object should be initialized. At our convenience, it has an InitWithFileWithEveryWay method, which has a string path parameter, and does the mumbo-jumbo for us (trying to initialize the object with each method until it succeeds). Then, DoPreview and UnloadPreview (which is also the Dispose method) are ready for us. During the preview, its ResetWindow and ResetBounds are used to move and resize it, Focus to set focus to it, and the managed wrapper for IPreviewFrameVisuals methods to align the scheme. A very important aspect is to call CoCreateInstance instead of using Activator.CreateInstance(Type.GetTypeFromCLSID(...)). The explanation is found here.

In the PreviewWindow, it is rather easy to implement the logic:

• When a preview is requested, query the preview handler CLSID, then hand it to PreviewHandler to construct a new preview handler, initialize it, set the color scheme and call DoPreview.
• When unloading is requested, call the UnloadPreview method and toss away our object.
• When our ContentPresenter receives focus, it tries to focus the preview handler, if any.
• Implement IPreviewHandlerManagedFrame as follows:
  • When asked for a list of accelerators, return Tab, B, F5 and Esc.
  • When asked to translate accelerators, if Tab is sent to us, move the focus away from the ContentPresenter (which represents the preview handler), direction decided by the asynchronous status of Shift, and deal with other accelerators similarly.

Explanation of important parts

We must create the object with CoCreateInstance P/Invoke and not Activator.CreateInstance(Type.GetTypeFromCLSID(...)), because according to the documentation, preview handlers always run out-of-process. However, Activator.CreateInstance allows in-process server. Creating the object in-process might cause the object fail to create because the object uses a different version of .NET Framework. The object might further assume it is running inside the surrogate, and might behave crazily if it is actually not, including crashing your process.

In PreviewHandler.SetupHandler, we try creating the object a second time if the first time failed with CO_E_SERVER_EXEC_FAILURE. Someone at (or previously at) Microsoft also noticed this. In the comment area, dcantemir reported that the presence of a debugger might cause the problem. In my experience, if the surrogate (server) process is not already running and you are trying to CoCreateInstance in a local server, the call will fail with CO_E_SERVER_EXEC_FAILURE after a period of time.

In PreviewHandler.InitWithFileWithEveryWay, we try IInitializeWithStream first because it provides the best security among the three options. We will create the stream read-only in a fully sharable way, so that the handler will not easily do harm to the file.

It is important to actually implement GetWindowContext because Excel previewer will actually need that. And no, you cannot simply fail the call, otherwise Excel will not send you any accelerators at all.

.NET will create a custom marshaler for our object when passed as a COM object and that marshaler will treat our object as free-threaded, we must delegate the calls to UI-related things to our Dispatcher. Therefore the delegation in PreviewWindow’s implementation of IPreviewHandlerManagedFrame.TranslateAccelerator. Note that even if we had access to the dispatcher, the action we decide to perform (browsing for another file, refreshing the preview handler or closing the window) must also be done asynchronously. Excel preview handler will encounter strange bugs if it is re-entered from TranslateAccelerator. Doing so asynchronously mitigates the problem.

Previous failure analysis

Mr Bradley Smith is a pioneer in trying to consume preview handlers. He has encountered with many obstacles during the process. He asked a question on StackOverflow why the IPreviewHandler implementation for TXT preview handler might throw uncatchable exception and crash the host. As we have seen, the reason is that preview handlers are not designed to run in every process, and unfortunately running TXT preview handler out of prevhost.exe might crash the process. The solution is to call CoCreateInstance.

In his blog, a lot of commenters seem to have similar problems of creating the handler object in-process. Here are some:

• Mr Smith said ‘64-bit processes cannot invoke 32-bit preview handlers, and vice versa’. This is wrong. Since preview handlers run out-of-process, they don’t have to match the host’s bitness.
• Mr Bill Boland found that sometimes an exception was thrown if Activator.CreateInstance was used. The reason is actually that preview handlers are not supposed to be created in-process. My guess is that MAPI preview handler is 64-bit on a 64-bit machine. Creating the object with Activator.CreateInstance from a 32-bit process resorts to using a surrogate process, which gives the good behaviour.
• Mr Jason Ching found that sometimes creation caused CO_E_SERVER_EXEC_FAILURE. This is why we would like to retry once before giving up. Moreover, the dreaded CO_E_SERVER_EXEC_FAILURE will almost certainly occur if we are being debugged (unless a suitable surrogate process is already running).

Some people failed to find a preview handler for images. The reason is that there isn’t any. Their ‘preview’ are extracted with thumbnail providers, which requires consumption of IExtractImage interface, IThumbnailProvider interface and the related.

Good preview handlers

I tested several preview handlers, ‘Windows TXT Previewer’ and ‘Microsoft Excel previewer’ are good.

Windows TXT Previewer does not use the GetWindowContext method, because it is designed to run at low integrity level. It forwards every keystroke to the host.

Microsoft Excel previewer uses the GetWindowContext method and will only send keystrokes upon which the host indicates interests. Looking at the registry (DisableLowILProcessIsolation of HKCR\CLSID\{00020827-0000-0000-C000-000000000046}), it opts itself out of low integrity level process isolation, therefore is able to perform this optimization (reducing number of keystrokes sent across process boundaries).

Misbehaving preview handlers

The other three preview handlers I tested behave worse.

Adobe PDF Preview Handler for Vista and Microsoft PowerPoint previewer ignores the location information provided with SetWindow and SetRect. They always show themselves at the top-left corner. The famous consumers of preview handlers, Windows Shell and Outlook, have a dedicated window (control) for the preview handlers, and the RECTs passed for the preview handlers are always the exact bounds of that window. That is the reason why this mistake is uncovered.

Moreover, Adobe PDF Preview Handler for Vista never sends any keystroke back to the host. Its SetFocus doesn’t really focus itself. Therefore, if you click the preview in our app and try to Tab away from the preview, you will not succeed. However, Windows Shell doesn’t seem to be affected by it. If you look closely, the moment you click the preview in Explorer, the Explorer window flashes, getting deactivated and reactivated. I am not sure whether this is a compatibility hack done to lick Adobe’s arse.

Microsoft Word previewer exhibits an interesting disorder. Its size is incorrect at first sight. But if you resize our window, the size of the preview handler will be correct, as shown below.

The reason is that Word previewer fails to follow the documentation that SetWindow sets both the parent window and the bounding box. Obviously the two famous consumers always call SetRect immediately after the first (and only) SetWindow, or the size doesn’t really matter for a borderless window (control). Word previewer seems to detect the size of the HWND passed into SetWindow instead of following the RECT.