Unity Graphics Emulator for Native Plugin Development

• Download latest source (Github)

Introduction

Unity is a very popular game engine with quite an impressive toolbox. However, it is very hard to provide every feature a potential customer may need, so Unity supports extension mechanism in the form of native plugins implemented by dynamic libraries containing native code. These plugins may contain general-purpose functionality, as well as graphics commands in low-level API such as Direct3D11, Direct3D12, or OpenGL/GLES. Native rendering plugins communicate with Unity through IUnityGraphics interface.

Native plugin development may not always be straightforward due to a number of reasons. First, Unity does not support hot plugin reload. Once dynamic library is loaded, it is never offloaded. It may be possible to create a workaround for this issue (such as writing a proxy plugin that only loads real dynamic library), but this introduces additional complexity and may not work on all platforms. Second, when plugin is under development, it is very easy to crash Unity editor. Restarting the editor and reloading the scene increases iteration times. Finally, attaching to running editor to debug the plugin is certainly possible, but may not always be optimal.

Due to the reasons above, it would be much more convenient to have an isolated environment that emulates Unity interfaces to facilitate native plugin development. This article describes such environment. It emulates Unity graphics interfaces and currently supports Direct3D11, Direct3D12, and OpenGL on Windows Desktop platform, Direct3D11 and Direct3D12 on Universal Windows Platform, and OpenGLES on Android. The full source code is free for use and available on GitHub.

Background

Some understanding of low-level graphics APIs such as Direct3D11, Direct3D12, or OpenGL/GLES as well as Unity native plugin interface is desirable.

Unity Graphics Interfaces

This section describes Unity graphics interfaces that allow native plugins access low-level graphics API and issue draw commands. Detailed information can be found on Unity help pages.

To be recognized as a graphics plugin, the library should export the following two functions:

extern "C" void UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API 
    UnityPluginLoad(IUnityInterfaces* unityInterfaces);

extern "C" void UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API 
    UnityPluginUnload();

UnityPluginLoad() is automatically called by Unity when a plugin dynamic library is loaded. UnityPluginUnload() is apparently supposed to be called when the plugin is unloaded, but in my experiments, I've never seen the function called (Unity 2017.1.1f1, Windows 64-bit).

UnityPluginLoad() gets a pointer to IUnityInterfaces, which is the main interface for the plugin to interact with Unity. A typical implementation of this function stores the pointer, requests a pointer to IUnityGraphics interface, registers OnGraphicsDeviceEvent() callback and manually calls it:

static IUnityInterfaces* s_UnityInterfaces = nullptr;
static IUnityGraphics* s_Graphics = nullptr;

extern "C" void UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API 
    UnityPluginLoad(IUnityInterfaces* unityInterfaces)
{
    s_UnityInterfaces = unityInterfaces;
    s_Graphics = s_UnityInterfaces->Get<IUnityGraphics>();
    s_Graphics->RegisterDeviceEventCallback(OnGraphicsDeviceEvent);
    
    // Run OnGraphicsDeviceEvent(initialize) manually on plugin load
    OnGraphicsDeviceEvent(kUnityGfxDeviceEventInitialize);
}

IUnityGraphics interface does not provide access to the low-level API. Its first goal is to register the graphics device event callback OnGraphicsDeviceEvent(). The callback is called when one of the following events happen:

• Graphics device initialization (kUnityGfxDeviceEventInitialize)
• Graphics device shutdown (kUnityGfxDeviceEventShutdown)
• Graphics device is about to be reset (kUnityGfxDeviceEventBeforeReset). This event only happens with Direct3D9 API, and is thus irrelevant for us.
• Graphics device has just been reset (kUnityGfxDeviceEventAfterReset). Similar, this event Direct3D9-specific and is irrelevant.

The second goal of IUnityGraphics interface is to report the low level API used by Unity through GetRenderer() function. The function may return a variety of values, but we only support the following renderers: kUnityGfxRendererD3D11, kUnityGfxRendererD3D12, kUnityGfxRendererOpenGLCore, and kUnityGfxRendererOpenGLES30.

OnGraphicsDeviceEvent() thus needs to handle kUnityGfxDeviceEventInitialize and kUnityGfxDeviceEventShutdown events:

static UnityGfxRenderer s_DeviceType = kUnityGfxRendererNull;
void UNITY_INTERFACE_API OnGraphicsDeviceEvent(UnityGfxDeviceEventType eventType)
{
    // Create graphics API implementation upon initialization
    if (eventType == kUnityGfxDeviceEventInitialize)
    {
        // Get render API
        s_DeviceType = s_Graphics->GetRenderer();
        CreateRenderAPI(s_DeviceType);
        if (s_CurrentAPI)
            s_CurrentAPI->ProcessDeviceEvent(eventType, s_UnityInterfaces);
    }
    else if (eventType == kUnityGfxDeviceEventShutdown)
    {
        // Cleanup graphics API implementation upon shutdown
        // We must destroy all resources before releasing the API
        g_SamplePlugin.reset();
        if (s_CurrentAPI)
        {
            s_CurrentAPI->ProcessDeviceEvent(eventType, s_UnityInterfaces);
            s_CurrentAPI.reset();
        }
        s_DeviceType = kUnityGfxRendererNull;
    }
    else if (s_CurrentAPI)
    {
        // Let the implementation process the device related events
        s_CurrentAPI->ProcessDeviceEvent(eventType, s_UnityInterfaces);
    }
}

CreateRenderAPI() is the function that initializes the plugin to work with the specific low-level API. We support Direct3D11, Direct3D12, and OpenGL/GLES, so the function looks like this:

static std::unique_ptr<RenderAPI> s_CurrentAPI;
void CreateRenderAPI(UnityGfxRenderer apiType)
{
#if SUPPORT_D3D11
    if (apiType == kUnityGfxRendererD3D11)
    {
        s_CurrentAPI.reset( CreateRenderAPI_D3D11() );
    }
#endif // if SUPPORT_D3D11

#if SUPPORT_D3D12
    if (apiType == kUnityGfxRendererD3D12)
    {
        s_CurrentAPI.reset( CreateRenderAPI_D3D12() );
    }
#endif // if SUPPORT_D3D9

#if SUPPORT_OPENGL_UNIFIED
    if (apiType == kUnityGfxRendererOpenGLCore || apiType == kUnityGfxRendererOpenGLES30)
    {
        s_CurrentAPI.reset( CreateRenderAPI_OpenGLCoreES(apiType) );
    }
#endif // if SUPPORT_OPENGL_UNIFIED
}

N.B.: When processing graphics device event, it is very important to check s_CurrentAPI for null, because Unity may call OnGraphicsDeviceEvent() with kUnityGfxRendererNull (which will result in no render API initialized) before calling it second time with the actual renderer. We will look at how to initialize the specific API little later, but now let's take a look at one more function that a graphics plugin needs to export:

extern "C" UnityRenderingEvent 
        UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API GetRenderEventFunc()
{
    return OnRenderEvent;
}

This function is called by Unity to query the function that is called when render event for this plugin is issued. The function should be declared as follows:

static void UNITY_INTERFACE_API OnRenderEvent(int eventID);

eventID is the integer passed to IssuePluginEvent() on Unity side. This is what a minimalistic Unity script calling native rendering plugin may look like:

public class UseRenderingPlugin : MonoBehaviour
{
    [DllImport("GhostCubePlugin")]
    private static extern IntPtr GetRenderEventFunc();

    void OnRenderObject()
    {
        // Issue a plugin event with arbitrary integer identifier.
        // The plugin can distinguish between different
        // things it needs to do based on this ID.
        // For our simple plugin, it does not matter which ID we pass here.
        GL.IssuePluginEvent(GetRenderEventFunc(), 1);
    }
}

Render API Initialization

Let's now talk about the steps a plugin needs to take to initialize for a specific low-level graphics API. CreateRenderAPI() function creates an instance of RenderAPI_D3D11, RenderAPI_D3D12, or RenderAPI_OpenGLCoreES class, depending on the render type. As class names imply, they handle specific low-level API. The full source code can be found on GitHub, so I am not going to post it here.

For every low-level API, Unity exposes specific interface (IUnityGraphicsD3D11, and IUnityGraphicsD3D12 are of main interest for us) that can be queried through IUnityInterfaces, for example:

IUnityGraphicsD3D11* d3d11 = interfaces->Get<IUnityGraphicsD3D11>();

Let's take a closer look at API-specific interfaces.

Direct3D11

For Direct3D11 renderer, Unity exposes IUnityGraphicsD3D11 interface that allows the plugin to get a pointer to D3D11 device. Immediate context can then be requested from the device:

IUnityGraphicsD3D11* d3d = interfaces->Get<IUnityGraphicsD3D11>();
m_d3d11Device = d3d->GetDevice();
CComPtr<ID3D11DeviceContext> d3d11ImmediateContext;
m_d3d11Device->GetImmediateContext(&d3d11ImmediateContext);

This is all an application needs to issue D3D11 rendering commands. Other methods of the IUnityGraphicsD3D11 interface allow the application to get access to internal D3D11 objects of a Unity render buffer or a native texture object.

OpenGL/GLES

There is no IUnityGraphicsGL interface as one may expect. The reason is that in OpenGL/GLES, everything is an internal global state, so there is really nothing that an interface would be able to return. To initialize the plugin in GL mode, Unity just calls OnGraphicsDeviceEvent() from the thread with active GL context. The plugin may call whatever gl function it needs to initialize itself.

NB: The most important thing about OpenGL mode is that Unity uses multiple GL contexts initialized by different threads. As a result, OnRenderEvent() may not be called from the same thread as OnGraphicsDeviceEvent(), which means that GL context-specific objects such as VAO, FBO, and program pipelines cannot be initialized in OnGraphicsDeviceEvent().

Direct3D12

Direct3D12 is (not surprisingly) the most involved case, for which Unity 2017.1.1f1 exposes five different interface versions. The emulator currently supports version 2:

UNITY_DECLARE_INTERFACE(IUnityGraphicsD3D12v2)
{
    ID3D12Device* (UNITY_INTERFACE_API * GetDevice)();

    ID3D12Fence* (UNITY_INTERFACE_API * GetFrameFence)();
    // Returns the value set on the frame fence once the 
    // current frame completes or the GPU is flushed
    UINT64(UNITY_INTERFACE_API * GetNextFrameFenceValue)();

    // Executes a given command list on a worker thread.
    // [Optional] Declares expected and post-execution resource states.
    // Returns the fence value.
    UINT64(UNITY_INTERFACE_API * ExecuteCommandList)
          (ID3D12GraphicsCommandList * commandList, 
        int stateCount, UnityGraphicsD3D12ResourceState * states);
};

The interface exposes the following functions:

• GetDevice() - returns a pointer to D3D12 device
• GetFrameFence() - returns a pointer to the fence used to synchronize GPU execution
• GetNextFrameFenceValue() - returns the value set on the frame fence once the current frame completes or the GPU is flushed
• ExecuteCommandList() - executes a given command list

The Emulator

Overview

Unity graphics emulator system contains the following main components:

• Unity graphics interface emulators (UnityGraphicsD3D11Emulator, UnityGraphicsD3D12Emulator, and UnityGraphicsD3D11Emulator) which are responsible for mimicking unity graphics interfaces (IUnityGraphicsD3D11 and IUnityGraphicsD3D12).
• Scene emulator that is responsible for simulating scene objects (such as RenderTexture). The plugin uses Diligent Engine to facilitate API-agnostic cross-platform graphics object management. The engine connects to Unity interfaces through adapters (DiligentGraphicsAdapterD3D11, DiligentGraphicsAdapterD3D12, and DiligentGraphicsAdapterGL). Scene emulator also calls plugin-specific functions such as setting transformation matrices or time.
• Unity plugin. The plugin communicates with the simulator (and Unity) through Unity interfaces. The plugin connects to the graphics API through RenderAPI_D3D11, RenderAPI_D3D12, and RenderAPI_OpenGLCoreES classes. It also communicates with scene to get other information such as current time and transformation matrices.

The system diagram is shown in the image below:

Unity Graphics Emulators

Unity graphics emulators are derived from UnityGraphicsEmulator class shown in the listing below and mostly implement standard boilerplate code that is not worth posting here.

class UnityGraphicsEmulator
{
public:
    virtual void Release() = 0;
    virtual void Present() = 0;
    virtual void BeginFrame() = 0;
    virtual void EndFrame() = 0;
    virtual void ResizeSwapChain(unsigned int Width, unsigned int Height) = 0;
    virtual bool SwapChainInitialized() = 0;
    virtual bool UsesReverseZ()const;
    virtual IUnityInterface* GetUnityGraphicsAPIInterface() = 0;
    IUnityInterfaces &GeUnityInterfaces();

    static UnityGraphicsEmulator& GetInstance() { return *m_Instance; }
    void RegisterInterface(const UnityInterfaceGUID &guid, IUnityInterface* ptr);
    IUnityInterface* GetInterface(const UnityInterfaceGUID &guid);
    virtual UnityGfxRenderer GetUnityGfxRenderer() = 0;

    void RegisterDeviceEventCallback(IUnityGraphicsDeviceEventCallback callback);
    void UnregisterDeviceEventCallback(IUnityGraphicsDeviceEventCallback callback);
    void InvokeDeviceEventCallback(UnityGfxDeviceEventType eventType);

private:
    UnityGraphicsEmulator(const UnityGraphicsEmulator&) = delete;
    UnityGraphicsEmulator(UnityGraphicsEmulator&&) = delete;
    UnityGraphicsEmulator& operator = (const UnityGraphicsEmulator&) = delete;
    UnityGraphicsEmulator& operator = (UnityGraphicsEmulator&&) = delete;

    std::vector< std::pair<UnityInterfaceGUID, IUnityInterface*> > m_Interfaces;
    static UnityGraphicsEmulator *m_Instance;
    std::vector<IUnityGraphicsDeviceEventCallback> m_DeviceEventCallbacks;
};

Every emulator returns corresponding Unity graphics interface. In case of D3D11, the code is shown below:

UnityGraphicsD3D11Impl* UnityGraphicsD3D11Emulator::GetGraphicsImpl() 
{ 
    return m_GraphicsImpl.get(); 
} 

static ID3D11Device* UNITY_INTERFACE_API UnityGraphicsD3D11_GetDevice()
{
    auto *GraphicsImpl = UnityGraphicsD3D11Emulator::GetGraphicsImpl();
    return GraphicsImpl != nullptr ? GraphicsImpl->GetD3D11Device() : nullptr;
}

IUnityInterface* UnityGraphicsD3D11Emulator::GetUnityGraphicsAPIInterface()
{
    static IUnityGraphicsD3D11 UnityGraphicsD3D11;
    UnityGraphicsD3D11.GetDevice = UnityGraphicsD3D11_GetDevice;
    return &UnityGraphicsD3D11;
}

UnityGfxRenderer UnityGraphicsD3D11Emulator::GetUnityGfxRenderer()
{
    return kUnityGfxRendererD3D11;
}

The two methods worth mentioning are BeginFrame() and EndFrame(), which, as their names suggest, are called at the beginning and end of each frame and perform some API-specific actions. BeginFrame() sets the default render target and depth-stencil buffer, clears them and sets the viewport. EndFrame() does nothing for D3D11 and OpenGL/GLES cases, and for D3D12, it transitions render target to present-compatible state and discards frame resources.

Unity Scene Emulator

Unity scenes contain lots of different objects and duplicating all of them in the emulator is neither practical nor useful. However, some objects do need to be duplicated in the emulation environment (such as mirror RenderTexture in our example project). Since the emulator supports multiple low-level APIs, it would be necessary to implement scene objects in multiple ways if the low-level APIs were used directly. To avoid this problem, scene emulator uses Diligent Engine, a cross-platform graphics API abstraction library. Diligent Engine connects to Unity interfaces through adapters (DiligentGraphicsAdapterD3D11, DiligentGraphicsAdapterD3D12, and DiligentGraphicsAdapterGL) that handle all API-specific functionality. The required scene objects can then be created in a graphics API-agnostic way (see GhostCubeScene.cpp for example).

Source Code

The emulator's full source code is available at GitHub and is free to use. The repo contains sample Unity project that uses native plugin to render reflection of a ghost cube in the mirror:

The emulator creates a render texture using the scene emulator and uses the same native plugin to render the cube:

The main reason why the cube is only visible in the mirror is obviously because it is a ghost cube. The other reason is that in D3D12 mode, there is no way that I am aware of to get the render target view of the main back buffer. It is possible to do this in D3D11 and OpenGL/GLES, but I wanted the plugin to look the same on all APIs. At the same time, Unity provides access to native handle of a render texture, which allows to set it as render target in D3D12 plugin.

The unityplugin folder is organized as follows:

• UnityEmulator folder contains implementation of the main emulator components (Unity graphics emulators, Diligent Engine adapters, base scene emulator, platform-specific functionality)
• GhostCubeScene folder contains implementation of the scene-specific objects (RenderTexture)
• GhostCubePlugin/PluginSource folder contains implementation of the native plugin that renders the cube using Diligent Engine
• GhostCubePlugin/UnityProject folder contains Unity project
• build folder contains Visual Studio solution files for Windows Desktop and Universal Windows platforms

Building and Running

Windows Desktop

To build the project for Windows Desktop platform, open UnityPlugin.sln solution in unityplugin/build/Win32 folder, select the desired platform and configuration, and build the project. Select GhostCubeScene as startup project and run it. You can use mode={GL|D3D11|D3D12} command line argument to select the graphics API.

Universal Windows Platform

To build the project for Windows Desktop platform, open UnityPlugin.sln solution in unityplugin/build/Win32 folder, select the desired platform and configuration, and build the project.

Android

To build for Android, you will need to first setup your machine for Android development.

Navigate to /unityplugin/GhostCubeScene/build/Win32/ folder and run android_build.bat.