OpenGL and WPF: Integrating an OpenGL Window inside a WPF Window

• Download source code - 11.4 KB

Introduction

This article uses OpenTK as the .NET library for accessing OpenGL on .NET. The library also has GLWpfControl, which is a Control to use for integration with WPF apps. There are other libraries which may or may not provide an option for WPF integration but describing the rationale for using OpenTK vs. other library, or for using WPF, OpenGL, etc. is not within the scope of this article - these are rather axiomatic: "Given WPF, OpenGL and OpenTK, then..."

The only thing that needs to be mentioned is that unfortunately, GLWpfControl didn't yield the performance I expected, at least not for my real-life application (see end of article). In fact, it was even worse than just using WPF custom animations with per-frame rendering. Hence, the challenge of directly using OpenTK's GameWindow in WPF and make it behave like a control. For this purpose, the following is a set of requirements or acceptance criteria for the window:

• The window must have no border, no title bar.
• The window must stick to the main window, i.e., it must not go behind the main window and no other window can get in-between.
• The window must stay "glued" to the area designated within the main window and must follow the main window when moved or resized.
• The window must be constrained (clipped) within the borders of the main window - it cannot extend beyond those borders (e.g., move/zoom).
• The window must not capture keyboard focus.
• The window must be transparent to mouse input, i.e., mouse input must pass through to the WPF content underneath.
• Optional: The window background must be transparent.
• The window must allow to be controlled by the application, for positioning, resizing, rendering.

Let's now list the advantages and disadvantages of this solution vs. using a dedicated control like GLWpfControl.

Advantages

• No performace degradation due to interoperability with WPF. Although I'm not an expert in this, from what I gather, all these interop WPF controls (also others like SharpGL's OpenGLControl or MonoGame's MonogameContentControl) have one thing in common: they use DirectX interop and work by copying the drawing results into buffers usable by WPF via its D3DImage class. There are extra buffer copy, flush operations that surely ought to have some performance impact... that doesn't apply when using the GameWindow directly.
• OpenTK's GameWindow doesn't require any OpenGL extensions, it only requires the core implementation. As opposed to that, GLWpfControl requires the NV_DX_interop extension, which, quite ironically, is not supported by my laptop's NVIDIA GPU but is supported by the integrated Intel graphics card! So if you don't have the extension, then you're out of luck.

Disadvantages

• Whereas a control is integrated in the WPF layout, with margins, horizontal/vertical alignments, etc., the GameWindow is not part of that layout so the positioning that is taken for granted with a control must be manually managed through the code. But it's not all that bad, as we will see a bit further.
• You cannot render anything in WPF over the GameWindow, like an overlay. Any rendering that occupies the same space as the window will go underneath it. I don't consider that use-case to be frequent, I rather consider a transparent window background a more frequent use-case. That is, the window is the overlay. Transparency is addressed in this article.

From here on, the discussion will be focused on the application contained in the zipped source code (link at the top). The basic appearance and behavior of the application is shown in the animated picture also at the top.

Dissecting the Code

Structure of the Application Code

The source code is in the form of a Visual Studio 2022 solution with a WPF Application project targeting .NET 6. The OpenTK library is installed as a NuGet package from nuget.org with the version set to 4.8.2 which is the latest stable version at the time of writing this article.

The code is based on the Hello Triangle OpenTK tutorial that is hosted on GitHub. The relevant classes for this article are MainWindow and GLWindow. I split the implementation of GLWindow into two source files. Not that I endorse this practice, but it was done to help staying focused on what is important for this article. The file GLWindow.article.cs contains the relevant part for this article - the logic for the creation and management of the OpenGL window. The file GLWindow.tutorial.cs, along with the shader files cover OpenGL-specific code that I won't insist on, because this is straight from and best covered by the mentioned OpenTK tutorial. There is also a small static class Interop that acts as a PInvoke wrapper to some required Win32 functions.

The GLWindow Constructor

GLWindow is the subclass of OpenTK's GameWindow responsible for the OpenGL rendering. In the introduction, we listed the basic requirements this window must satisfy, so let's see how those get accomplished in the code. We start off with the constructor:

private GLWindow(IntPtr hWndParent, Vector2i location, Vector2i size)
    : base(GameWindowSettings.Default,
            new NativeWindowSettings {
                Location = location,
                ClientSize = size,
                WindowBorder = WindowBorder.Hidden
            })
{
    unsafe {
        GLFW.HideWindow(WindowPtr);
        IntPtr ptr = GLFW.GetWin32Window(WindowPtr);
        uint childStyle = Interop.GetWindowLong(ptr, Interop.GWL_STYLE);
        childStyle |= Interop.WS_CHILD;
        childStyle &= ~Interop.WS_POPUP;
        _ = Interop.SetWindowLong(ptr, Interop.GWL_STYLE, childStyle);
        _ = Interop.SetWindowLong(ptr, Interop.GWL_EXSTYLE, 
            Interop.WS_EX_TOOLWINDOW | Interop.WS_EX_LAYERED | Interop.WS_EX_TRANSPARENT);
        _ = Interop.SetParent(ptr, hWndParent);
        _ = Interop.EnableWindow(ptr, false);
        _ = Interop.SetLayeredWindowAttributes(ptr, 0x00000000, 0, Interop.LWA_COLORKEY);
        GLFW.ShowWindow(WindowPtr);
    }
    _xpos = location.X;
    _ypos = location.Y;
}

The constructor invokes the base class version passing in some settings via NativeWindowSettings: the initial location and size of the window, which are received from the main window, and a very useful WindowBorder.Hidden setting to have a window without borders and title bar (one of the requirements). We don't specify any setting for the OpenGL version we want to use and let it be the default 3.3 version.

By the time the constructor body is entered, the window is already created by the base constructor. OpenTK gives us a low-level unsafe pointer to the window WindowPtr and low-level GLFW API (wrappers for OpenGL's glfw3.dll) which we can use. We use GLFW.HideWindow to hide the window before we alter it and GLFW.GetWin32Window to retrieve the Win32 HANDLE to the window, which we need further. So in the constructor body, we start making alterations to accomplish our requirements, which scream for the window to be set as a child of the main window. OpenTK doesn't give us any option for that, so we need to use the Win32 API function SetParent through the PInvoke wrapper Interop. According to MSDN though, we need to clear the WS_POPUP style and set the WS_CHILD style before calling this function. We do that with a call to GetWindowLong to get the current style and then to SetWindowLong to alter the style accordingly. We need to further alter the style through some extended style flags in order to satisfy more requirements. We do that with a second call to SetWindowLong with a combination of flags:

• WS_EX_TOOLWINDOW prevents it from appearing in the taskbar
• WS_EX_LAYERED | WS_EX_TRANSPARENT are used in conjuction with SetLayeredWindowAttributes to enable transparency

The next calls are as follows:

• EnableWindow, which disables the mouse and keyboard input to the window. The mouse events will pass through to the WPF window underneath, and the keyboard events will also be directed to the parent WPF window.
• SetLayeredWindowAttributes sets the transparency for the window. In this case, a color key (LWA_COLORKEY) passed as a 0x00bbggrr value sets the black color (0x00000000) as a transparent color. This matches the color used for background in the OpenGL rendering, which means the window background is transparent. All other rendered colors will be opaque, which is what we want.

Finally, we show the window with a call to GLFW.ShowWindow and we record the initial location of the window to update it when necessary.

That concludes the constructor; there is more to comment about SetLayeredWindowAttributes and OpenTK's support for transparent background, window opacity and mouse passthrough, but I will make those comments in the Conclusion section at the end.

The GLWindow Thread

At this point, we have a child window that is constrained within its parent, is borderless, disabled for mouse and keyboard input, and with transparent background. We are pretty close to satisfying the requirements from the introduction.

GLWindow has a rendering loop that is inherited from its parent class GameWindow - the Run method - which makes its calling thread busy. That means the thread cannot be the same as the main thread, which is the WPF UI thread. The window is therefore created and run in a Task that acquires a background thread from the thread pool. To be able to do this, we need to turn off a flag (property) in OpenTK: GLFWProvider.CheckForMainThread = false, otherwise, we get an exception when creating the window. We do this in a static method, along with creating and running the task:

public static GLWindow? CreateAndRun(IntPtr hWndParent, Rect bounds)
{
    GLFWProvider.CheckForMainThread = false;
    using var mres = new ManualResetEventSlim(false);
    GLWindow? glWnd = null;
    _GLTask = Task.Run(() => {
        glWnd = new GLWindow(hWndParent, ((int)bounds.X, (int)bounds.Y), 
                ((int)bounds.Width, (int)bounds.Height));
        mres.Set();
        using (glWnd) {
            glWnd?.Run();
        }
    });
    mres.Wait();
    return glWnd;
}

CreateAndRun is meant to be called from the main thread and starts the task stored in the _GLTask static member by calling its Task.Run method. The main thread passes the Win32 HANDLE wrapped in IntPtr of the parent window necessary for SetParent and the initial bounds of the child window. The task simply creates the child window through its constructor and then starts its rendering loop which is the inherited Run method. The rendering loop runs continuously from then on and doesn't return until the end of the application when the child window is closed.

There is a bit of a sync between the main thread and the task: the main thread doesn't return until the parallel task completes the creation of the child window. This is done through a ManualResetEventSlim, which the main thread waits on to be signaled after it starts the task, and the task signals after creating the child window.

Communicating with GLWindow

To complete the requirements, we need to be able to control the repositioning and resizing of the child window. As a bonus, the hiding and showing too. And it would be common to control what to render and when as well. This app follows the OpenTK tutorial, where everything that needs to be rendered (all vertex buffers) is determined at window loading time (OnLoad), but more realistically, it should be more fluid than that and under the control of the application.

That means we need to communicate with the child window - setting its location and size, calling methods for showing and hiding, setting parameters that affect the rendering in the render loop. Because this is a window created and running its render loop in a different thread, then thread-safety comes into focus. According to OpenTK's FAQ section although OpenTK is thread-safe, OpenGL itself is not. It's hard to gauge from this what exactly is thread-safe and what is not. Moving and hiding the window might be OK, since they are OS-level operations and don't involve OpenGL. However, to be on the safe side and be sure, we don't ask for trouble, we go by the general rule that any action we take that affects the window and its rendering must be a thread-safe action (even moving and hiding).

The pattern we go for is a loose command pattern where "commands" are sent from the main thread to the child window's thread through a thread-safe ConcurrentQueue object. GLWindow provides a bunch of public methods for such commands, which the main thread can directly call, but these methods only enqueue the commands (a command is an Action). Here are those methods:

public void Cleanup()
{
    EnqueueCommand(() => {
        Close();
    });
    _GLTask?.Wait();
}

public void SetBoundingBox(Rect bounds)
{
    EnqueueCommand(() => {
        ClientLocation = ((int)bounds.X, (int)bounds.Y);
        ClientSize = ((int)bounds.Width, (int)bounds.Height);
        _xpos = bounds.X; _ypos = bounds.Y;
    });
}

public void MoveBy(int deltaX, int deltaY)
{
    EnqueueCommand(() => {
        _xpos += deltaX; _ypos += deltaY;
        ClientLocation = ((int)_xpos, (int)_ypos);
    });
}

public void Show()
{
    EnqueueCommand(() => {
        unsafe {
            GLFW.ShowWindow(WindowPtr);
        }
    });
}

public void Hide()
{
    EnqueueCommand(() => {
        unsafe {
            GLFW.HideWindow(WindowPtr);
        }
    });
}

public void ToggleSpin()
{
    EnqueueCommand(() => {
        _isSpinStopped = !_isSpinStopped;
    });
}

private void EnqueueCommand(Action command) =>
    _commands.Enqueue(command);

I won't insist much on these methods, as they should be self-explanatory. Just to mention that Cleanup is the method getting called when the main WPF window is closed. The call to Close ends the rendering loop, so the task ends as well. The main thread waits for the task to end, making sure the child window closing (and the cleanup associated with it) completes. The cleanup code is in the overriden OnUnload from GLWindow.tutorial.cs. In our case, this is optional, as when the appplication exits the resources held by the process, such as vertex buffers, would be released by the OS anyway; it's nice though to have a disciplined shutdown where specific cleanup is performed.

Also, ToggleSpin is a simple example of controlling the rendering. While here, it just stops or restarts the spinning of the triangle, one can imagine this being expanded into something more advanced in a real-life application, such as controlling what objects to render.

GLWindow is the consumer of the ConcurrentQueue stored in the member _commands. But the question is what is the appropriate place where to consume this queue? OpenTK's tutorial shows that the overriden OnUpdateFrame is the place where keyboard and mouse input is processed. In our case, the window doesn't process any keyboard or mouse input, so we replace that processing with the processing of the "command input". Since OnUpdateFrame is called by the rendering loop Run (just as OnRenderFrame) then any Action from the queue is executed on the child window's thread, which is what we want. Here are the relevant methods:

protected override void OnUpdateFrame(FrameEventArgs args)
{
    base.OnUpdateFrame(args);
    while (TryDequeueCommand(out Action? command)) {
        command();
    }
}

private bool TryDequeueCommand([MaybeNullWhen(returnValue: false)] out Action command) =>
    _commands.TryDequeue(out command);

At this point, the child window satisfies all requirements from Introduction. Next is to review the main WPF window, which reveals how GLWindow is used.

The Main Window

<Window x:Class="WpfOpenTK.MainWindow"
      xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
      xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
      xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
      xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
      xmlns:local="clr-namespace:WpfOpenTK"
      mc:Ignorable="d"
      Title="OpenTK in WPF" Height="600" Width="650"
      Background="#000020"
      Loaded="Window_Loaded"
      Closing="Window_Closing"
      KeyDown="Window_KeyDown"
      >
  <Grid>
    <Grid.RowDefinitions>
      <RowDefinition/>
      <RowDefinition/>
      <RowDefinition/>
    </Grid.RowDefinitions>
    <Grid.ColumnDefinitions>
      <ColumnDefinition/>
      <ColumnDefinition/>
      <ColumnDefinition/>
    </Grid.ColumnDefinitions>

    <Rectangle Name="RenderArea" Grid.Row="1" 
    Grid.Column="1" Stroke="#008800" 
    SizeChanged="RenderArea_SizeChanged"/>

    <StackPanel Grid.Row="1" Grid.Column="2" 
    HorizontalAlignment="Left" VerticalAlignment="Center">
      <ToggleButton Width="100" Checked="HideShowButton_Checked" 
      Unchecked="HideShowButton_Unchecked">
        <ToggleButton.Style>
          <Style TargetType="ToggleButton">
            <Setter Property="Content" Value="Hide"/>
            <Style.Triggers>
              <Trigger Property="IsChecked" Value="True">
                <Setter Property="Content" Value="Show"/>
              </Trigger>
            </Style.Triggers>
          </Style>
        </ToggleButton.Style>
      </ToggleButton>
      <ToggleButton Width="100" Margin="0,5,0,0" Click="StopStartButton_Click">
        <ToggleButton.Style>
          <Style TargetType="ToggleButton">
            <Setter Property="Content" Value="Start spin"/>
            <Style.Triggers>
              <Trigger Property="IsChecked" Value="True">
                <Setter Property="Content" Value="Stop spin"/>
              </Trigger>
            </Style.Triggers>
          </Style>
        </ToggleButton.Style>
      </ToggleButton>
    </StackPanel>
  </Grid>
</Window>

The main WPF window is divided by a 3x3 grid; the cell in the middle is the area designated for the OpenGL rendering. In that cell, we place a Rectangle that plays no other role than defining the bounding box for the GLWindow, where the OpenGL rendering takes place. This bounding box is relative to the main window. As the main window is resized and the rectangle changes its position and size, so will the GLWindow. This layout choice is completely arbitrary, real-life apps will have their own layouts with a render area dictated by business requirements.

The other elements in the main window are two buttons that are used to send some commands to GLWindow - one for hiding and showing the window and another for stopping and resuming the rotation of the rendered triangle. The main window also defines event handlers for loading, closing and some keyboard input.

In reviewing the code behind, we will address in succession the following features:

• Creating and closing the child window
• Resizing the main window and the child window along with it
• Panning the triangle by moving the child window with the arrow keys
• Using buttons for hiding/showing and controlling the spin of the triangle

Creating and Closing the Child Window

private void Window_Loaded(object sender, RoutedEventArgs e)
{
    IntPtr hWndParent = new WindowInteropHelper(this).Handle;
    _glWnd = GLWindow.CreateAndRun(hWndParent, GetRenderAreaBounds());
}

private void Window_Closing(object sender, System.ComponentModel.CancelEventArgs e)
{
    _glWnd?.Cleanup();
}

Creating the child is done in the parent's Loaded event handler. Here, we obtain the Win32 HANDLE to the parent and call the static method CreateAndRun discussed before. The method GetRenderAreaBounds is discussed in the next paragraph. Closing the child is done in the parent's Closing event handler, which simply calls the child's Cleanup method. Remember that this method enqueues a command for closing the child and then waits for the child's thread to acknowledge it is about to end.

Resizing the Main Window and the Child Window Along With It

When the main window is resized, the grid cells are also resized, so we need to keep track of the position of the rectangle in the middle cell, so we can update the position and the size of the child window. For that, we handle the rectangle's SizeChanged event:

private void RenderArea_SizeChanged(object sender, SizeChangedEventArgs e)
{
    _glWnd?.SetBoundingBox(GetRenderAreaBounds());
}

private Rect GetRenderAreaBounds()
{
    Point location = RenderArea.TransformToAncestor(this).Transform(new Point(0, 0));
    return new Rect {
        X = location.X, 
        Y = location.Y, 
        Width = RenderArea.ActualWidth, 
        Height = RenderArea.ActualHeight
    };
}

The important method is GetRenderAreaBounds where the rectangle's position (top-left corner) is reacquired relative to the main window (this). This method was also used to get the initial position when the child window was created. For resizing, the event handler calls the child window's SetBoundingBox that we've already seen earlier, and which enqueues the command that ends up updating the ClientLocation and ClientSize properties of the child window.

Panning the Triangle by Moving the Child Window With the Arrow Keys

The main window's KeyDown event handler captures presses of the arrow keys and moves the child window in the corresponding directions. If the triangle reaches the borders of the main window, it is clipped and can dissapear completely. Also, if it overlaps the two buttons, then the buttons still react to mouse over and mouse clicks even if the mouse is directly above the triangle. All this is possible to the way we created the child window and gave it mouse transparency.

private void Window_KeyDown(object sender, KeyEventArgs e)
{
    int moveStep = 5;
    switch (e.Key)
    {
        case Key.Up:
            _glWnd?.MoveBy(0, -moveStep);
            break;
        case Key.Down:
            _glWnd?.MoveBy(0, moveStep);
            break;
        case Key.Left:
            _glWnd?.MoveBy(-moveStep, 0);
            break;
        case Key.Right:
            _glWnd?.MoveBy(moveStep, 0);
            break;
    }
}

As we've seen, the method MoveBy enqueues a command for the child window that ends up updating its ClientLocation and ClientSize. Also to note that although the arrow keys move the triangle off the center cell, as soon as the main window is resized the triangle snaps back to the center cell!

Note: Panning by moving the window around is done just to show that:

• we can send such commands to the window
• the main window (the parent) still receives keyboard input
• the window gets clipped by its parent when it reaches the parent's borders
• we have mouse transparency and we can click the buttons when the window covers them

A true (2D) panning requirement is much better accomplished by applying translation matrices to the rendered objects!

Using Buttons for Hiding/showing and Controlling the Spin of the Triangle

The event handlers of the two buttons send commands to the child window for hiding/showing and stopping/resuming the spin of the triangle:

private void HideShowButton_Checked(object sender, RoutedEventArgs e)
{
    _glWnd?.Hide();
}

private void HideShowButton_Unchecked(object sender, RoutedEventArgs e)
{
    _glWnd?.Show();
}

private void StopStartButton_Click(object sender, RoutedEventArgs e)
{
    _glWnd?.ToggleSpin();
}

Hiding and showing the child window may or may not be a useful ability, depending on the use-case. Controlling the rendering parameters, as with ToggleSpin can definitely be useful.

Conclusion and Points of Interest

I've shown how it is possible to take advantage of OpenGL rendering in a WPF window by embedding a plain OpenTK window right inside the WPF window as if it were a control. This doesn't require any DX interop, which adds layers of indirection and impacts (in my case, destroys) performance. Download the zipped source code, build the VS solution, and try it for yourself.

There are a few more thoughts on a few topics related to this article which are listed below as additional points of interest.

• Creating and running a GameWindow in a separate thread is something that OpenTK only recently has opened the door to, because it is not cross-platform - there is an interesting dicussion about that on their GitHub repository. On the other hand, this is about WPF, therefore, we only target Windows.

• One thing that I am not sure about is what happens if multiple OpenGL rendering areas, each with its own window, are created and coexist at the same time. For that matter, it seems GLWfpControl itself has some challenges with multiple instances. These are scenarios I din't have much interest delving into, as to me, they are less common use cases.

• You likely have noticed the unsafe blocks of code whenever the GameWinodw's unsafe pointer WindowPtr is used. The compiler requires the /unsafe compiler argument to compile the project, which can be specified through the project settings. If you feel uncomfortable with that, you can isolate the OpenGL related code into a separate library, not affecting the rest of the WPF application project.

• There isn't much of defensive programming like error checking in the source code, as this is a demo app rather than a real-life app. In a real-life app, we should at least check that the OpenGL version we want to use is supported on that system. That entails putting the window creation call in a try...catch block:

  C#

  try {
      glWnd = new GLWindow(hWndParent, bounds);
  }
  catch (GLFWException ex) when (ex.ErrorCode == ErrorCode.VersionUnavailable) {
      // Error - OpenGL version not supported
  }

• On the subject of transparency, the GLFW documentation states that GLFW has support for framebuffer transparency (which can be used to make transparent/tanslucid background) as well as whole window transparency. OpenTK brings this support to .NET as well: framebuffer transparency can be enabled either through a call like

  C#

  GLFW.WindowHint(WindowHintBool.TransparentFramebuffer, true);

  or by setting the NativeWindowSettings property TransparentFramebuffer = true. For whole window transparency GLFW provides:
  C#

  GLFW.SetWindowOpacity(WindowPtr, 0.5f); // e.g. half transparency

  So, all of the above is nice and true, but with the following twist: it only works for a top-level window, not a child window (I haven't been able to discover why). Hence, the use of the Win32 SetLayeredWindowAttributes discussed previously. This function has some interesting features and a limitation of its own. I won't dive into its features (for example LWA_COLORKEY vs. LWA_ALPHA), but I am going to mention its own twist because it is relevant to this article: although this function has been supported for child windows since Windows 8, it can only be used for our purpose on Windows 10 and later. On Windows 8.1, it has the nasty effect of "freeing" the child window from the borders of its parent window, i.e., the child can extend beyond the borders of the parent. Moreover, the transparency only seems to work with LWA_ALPHA, which makes the whole window tranparent/translucid, but not with LWA_COLORKEY, where the color used for key is still rendered as opaque. Therefore, it is better not to use window transparency on Windows 8/8.1 and be content with an opaque background.

  SetLayeredWindowAttributes undesired effect with LWA_COLORKEY and LWA_ALPHA on Windows 8.1:

  

• On the subject of mouse transparency, OpenTK could support that through a call like:

  C#

  GLFW.WindowHint(WindowHintBool.MousePassthrough, true);

  However, although the OpenTK version used here has the API, the underlying OpenGL version used here (3.3) does not support mouse passthrough, as it was added only in 3.4. It doesn't matter though, the Win32 function EnableWindow takes care of that.

Real Life Application

This is a low-res shot of a real life WPF application that uses the solution from this article:

The OpenGL rendering is only in the area of the piano roll, which is an OpenGL window, the rest of the piano (such as the keys, the frame) and main window is content rendered by WPF. The figure shows how that region moves with the whole piano and the main window, even when panning and zooming is involved, behaving pretty much like a control.

History

• 29^th February, 2024: Initial version