(untagged)

Accessing WPF apps over a Web browser via WebSockets and HTML5 Canvas

Anoop Madhusudanan

0.00/5 (No votes)

12 Apr 2011

Shows how to push your WPF apps to users via WebSockets and HTML5 Canvas.

Introduction

This is a hack for pumping WPF windows over WebSockets and rendering it using HTML5 canvas - also with basic mouse input support. For each user connecting to the server, a new window instance is created, and the screen will be pumped over web sockets to the browser as base64 encoded images. These are rendered on an HTML5 canvas, almost real time.

What does this mean?

Once the HTML5 specification is final - it is almost certain that it'll be the most widely adopted platform for delivering applications to end users. With WebSockets in HTML5, we are going beyond the unidirectional, stateless HTTP to a full duplex, high speed connection infrastructure between the server and the client. This will definitely bring up lot of interesting possibilities, including exciting and feature rich user experiences. WPF is already a widely adopted, feature rich desktop application development platform, and this hack is a quick POC to bring the WPF experience over the web to the end user using HTML5 web sockets and canvas.

You can see that I'm accessing my WPF application over multiple browsers in the below video. Just to clarify, it is rendered in the browser purely using HTML5 canvas - No Silverlight or Flash.

Rendering the screen in the browser

First, let me explain a bit about WebSockets. The HTML5 draft specification includes WebSockets - here:

WebSocket is a technology providing for bi-directional, full-duplex communications channels, over a single Transmission Control Protocol (TCP) socket. It is designed to be implemented in web browsers and web servers but it can be used by any client or server application. The WebSocket API is being standardized by the W3C and the WebSocket protocol is being standardized by the IETF.

Here is a pretty minimized implementation of the page that opens a web socket, retrieves the image data, and draws it on a canvas:

var socket = new WebSocket("ws://localhost:8181/main");
socket.onopen = function () {

}

socket.onmessage = function (msg) {
    var canvas = document.getElementById("canvas");
    var ctx = canvas.getContext("2d");
    var img = new Image;
    img.onload = function () {
        ctx.clearRect(0, 0, canvas.width, canvas.height)
        ctx.drawImage(img, 0, 0); // Or at whatever offset you like
    };
    img.src = "data:image/gif;base64," + msg.data; ;
}

The server

I was planning to implement a quick socket server based on draft specifications - but I found that there are already a couple of .NET based implementations out there. One among them is Nugget - http://nugget.codeplex.com/ - I have a thin layer on top of Nugget for creating a window instance whenever a client socket connects to the server.

var srv = new WebSocketServer(port, origin, "ws://" + server + ":" + port);
srv.RegisterHandler<ElementSocket<T>>("/" + name);
srv.Start();
return srv;

Most of the logic required for creating window instances and pumping them back to the connected client socket goes in my ElementSocket implementation.

Once connected, the client WebSocket can send string commands to our server, requesting operations like start, stop, create etc. Have a look at the ProcessCommand method in the ElementSocket class, which actually does the command processing. When we receive a 'create' request from the WebSocket, we'll create a WPF Window instance based on the input key, and will setup a timer to pump the window as base64 encoded images to the WebSockets. This will be rendered on an HTML5 canvas as shown above.

Here is the method that encodes the WPF screens as GIF images. Have a look at this method:

public static string FrameworkElementToBase64(FrameworkElement c)
{
    using (MemoryStream ms = new MemoryStream())
    {
        int Height = (int)c.ActualHeight;
        int Width = (int)c.ActualWidth;

        RenderTargetBitmap bmp = new RenderTargetBitmap(
                 Width, Height, 96, 96, PixelFormats.Pbgra32);
        bmp.Render(c);
        var encoder = new GifBitmapEncoder();
        encoder.Frames.Add(BitmapFrame.Create(bmp));
        encoder.Save(ms);
        byte[] imageBytes = ms.ToArray();
        // Convert byte[] to Base64 String
        string base64String = Convert.ToBase64String(imageBytes);
        return base64String;
    }
}

Now, a WebSocket can subscribe to a visual element, and from that point onwards we'll send the Visual Element to the WebSocket based on the interval, as a base64 encoded image.

void SetupTimer(int interval)
{
    t = new Timer { Interval = interval };
    t.Elapsed += (e, s) =>
    {
        try
        {
            Send(element.GetContextAsBase64());
        }
        catch (Exception ex)
        {
            Debug.WriteLine(ex.Message);
        }
    };

    t.Start();
}

Capturing user inputs

Presently, the support for input devices (mouse, keyboard) etc., is minimal. Only the mouse-button-down and mouse-button-up events are supported. To do this, we capture the mouse inputs using jQuery, and then pump it back to the server.

$("#canvas").mousedown(function (e) {
    var point = getXY(e);
    socket.send("mousedown;" + point.X + ";" + point.Y);
});

$("#canvas").mouseup(function (e) {
    var point = getXY(e);
    socket.send("mouseup;" + point.X + ";" + point.Y);
});

On the server side, the incoming messages are used to simulate the events on the WPF window instance that corresponds to the socket, like:

var pnt=this.TranslatePoint(new Point(x, y), this);
var res=VisualTreeHelper.HitTest(this, pnt);

if (res != null && res.VisualHit != null && res.VisualHit is UIElement)
{
    SimulateEvent(res, action);
}

In SimulateEvent, we'll invoke the corresponding action on the target element after performing a hit test.

Challenges

There are a number of challenges - ideally, we should push only the region copies once it is modified. As of now, I'm pushing the entire screen image (bad) as that's pretty easy (this is a weekend hack, remember). Now, another challenge is proper user input simulation on the instance windows.

I still need to find a good way to trigger user inputs to the specific windows, without affecting the actual input devices. For example, when a user moves the mouse, the move message should be triggered to the corresponding server side window instance, without affecting the actual mouse cursor. I may need to have a look at the WM_ messages, not sure to what extend it can be applied on WPF top level windows.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here