A Dynamic Rest Client Proxy with the DLR

• Download latest source code from GitHub
• NuGet Package

Introduction

When I come across an interesting Rest web service that I want to explore or maybe integrate into an app, the first thing that needs to be done is to create a bunch of wrapper classes around the http communication so that the meat of the service can be invoked. This usually looks something like this:

• Read the API docs
• Look at pre-supplied .NET library (if any) and decide it doesn't fit the rest of the programming model, so write a wrapper
• Create some service classes to mirror the endpoints of the API
• Create a bunch of POCO objects to represent the data going back and forth
• Fiddle around with that for a bit until data is flowing
• Actually do something interesting with the API

Even with great tools like RestSharp and Json2CSharp, I always find myself writing a lot of boilerplate code before getting down to the fun.

This little project grew out of my boredom with the boilerplate, plus a desire to explore the Dynamic Language Runtime (DLR). The result is a convention based, dynamic rest client and transparent proxy that can use RestSharp or the portable Microsoft HttpClient for transport. The goal is to make it easy to interact with Rest Services with minimal startup overhead.

Background

The basic premise is that the RestProxy is a DynamicObject that translates property and method invocations into Rest endpoint Uri's and allows basic http verb invocation. A DynamicObject generates its members at runtime and it's this capability that is used to build up the request and execute it.

One downside of dynamic objects is the lack of IntelliSense since the IDE does not know what members the object has or will have. It feels more like JavaScript than C#.

Using the Code

Client Conventions

• All communication is via http or https
• Data transfer is always JSON
• The vast majority of API access can be accomplished with GET, PUT, POST, PATCH or DELETE
• Unnamed arguments passed to a verb invocation are serialized to the request body
• Named arguments are passed as request parameters (either query params or form encoded)
• Outputs are dynamic objects by default, but serialization to a static type is supported
• All Rest calls are asynchronous and awaitable (they always return a Task)

Calling Conventions

Calls to the dynamic client all take the following pattern:

client.{optional chain of dot separated property names}.verb({optional parameter list});

• each property name represents a Url segment relative to the root url
• verb must be one of get, put, post, patch or delete
• unnamed arguments to the verb invocation will be serialized into the request body
• named arguments to the verb invocation will be added as named parameters

So getting up and running with a new service endpoint is three steps:

1. Create a DynamicRestClient to represent the API root
2. Chain together members of the client object to build up the endpoint Uri
3. Invoke away!

Example

So let's try a simple GET example using SunLight Labs API:

dynamic client = new DynamicRestClient("http://openstates.org/api/v1/");

dynamic result = await client.metadata.mn.get(apikey: "your_api_key_goes_here");

Assert.IsNotNull(result);
Assert.AreEqual("Minnesota", result.name);

So what's going on here? The first line is pretty self explanatory; create the DynamicRestClient specifying the root Uri. The DynamicRestClient uses the BCL HttpClient library for the request and response communication but that is pretty well hidden behind the scenes.

The second line is where all the dynamic stuff is happening but in just two lines of code, a rest endpoint is defined, accessed and its response deserialized and returned.

The endpoint we are ultimately invoking is this: http://openstates.org/api/v1/metadata/mn/.

See the pattern? metadata.mn gets translated to metadata/mn/ and the get() defines the type of http request invoked. And then the method arguments apikey: "your_api_key_goes_here", are turned into parameters. Were you to look at the http request that line creates, you'd see:

GET http://openstates.org/api/v1/metadata/mn/?apikey=your_api_key_goes_here HTTP/1.1
Accept: application/json, text/json, text/x-json, text/javascript
Accept-Encoding: gzip, deflate
Host: openstates.org

When a DynamicObject has a property accessed, a method called TryGetMember is invoked. It's in here, and the DynamicObject's other overridable methods, that we create a chain of DynamicObjects to represent the complete endpoint Uri. There are similar methods for invoking a dynamic method or invoking a dynamic object as a delegate. These overrides are also used to help create a complete endpoint Uri.

public override bool TryGetMember(GetMemberBinder binder, out object result)
{
    // this gets invoked when a dynamic property is accessed
    // example: proxy.locations will invoke here with a binder named locations
    // each dynamic property is treated as a url segment
    result = CreateProxyNode(this, binder.Name);

    return true;
}

But Wait! Urls can have spaces and all sorts of other crazy characters in them!!!

That's where we introduce an escape method. In order to add a uri segment that is not a valid C# identifier, escape it by passing it as an argument to any method on the dynamic proxy object. Escaped segments can be chained and intermixed with property segments in any combination.

dynamic client = new DynamicRestClient("http://openstates.org/api/v1/");

var result = await client.bills.mn("2013s1")("SF 1").get(apikey: "your_api_key_goes_here");

Assert.IsNotNull(result);
Assert.IsTrue(result.id == "MNB00017167");

This has the added advantage of allowing us to add segments that are data not code. For instance, perhaps part of the endpoint is determined by user selection (picking one state over another in the examples above) or it is a value returned from a previous call.

string billId = GetBillIdFromUser();
var result = await client.mn("2013s1")(billId).get();

Segment Chaining

Notice the somewhat odd proxy.bills.mn("2013s1")("SF 1"). What is going on there?

Until one of the five http verbs is invoked, every method call or property access on the DynamicRestClient returns another instance of a DynamicRestClient that are chained together, forming the entire uri.

• proxy.bills returns a new proxy object from the property accessor "bills"
• bills.mn("2013s1") invokes the dynamic method "mn" on the bills instance, passing "2013s1" as an argument
• both mn and "2013s1" are returned as proxy object instances
• the final part mn("2013s1")("SF 1") is invoking the 2013s1 instance as if it were a delegate. This too returns another instance of a proxy object added to the chain

Each client instance represents a segment in the endpoint uri. Segment names are defined as the dynamic property names, method names and/or arguments passed to dynamic method invocations.

Passing Parameters

Named parameters are passed to the verb method using C#'s named argument syntax. Here's an example using Bing's Locations API:

dynamic client = new DynamicRestClient("http://dev.virtualearth.net/REST/v1/");

dynamic result = await client.Locations.get
                 (postalCode: "55116", countryRegion: "US", key: "bing_key");

Assert.AreEqual(200, result.statusCode);

The http request for the above looks like this:

GET http://dev.virtualearth.net/REST/v1/Locations?postalCode=55116&countryRegion=US&key=bing_key& HTTP/1.1
Accept: application/json, text/json, text/x-json, text/javascript
Host: dev.virtualearth.net
Accept-Encoding: gzip, deflate
Connection: Keep-Alive

Again, you can see that named paramters passed to the dynamic method are converted to name value pair parameters on the endpoint uri.

Escaping Parameter Names

Parameter names are not always going to be valid C# identifiers (though in practice they are most of the time). Since we are using C#'s named argument syntax for request parameters, this represents a problem. Take for instance, this endpoint:

It has a parameter name with a "." in it. In order to escape parameter names, they can be passed to the invoke functions in a dictionary. Any named parameter that is an IDictionary<string, object> will have each key/value pair added as a parameter. This example code will generate the rest request above:

dynamic client = new DynamicRestClient("http://congress.api.sunlightfoundation.com");

var parameters = new Dictionar<string, object>()
{
    { "chamber", "senate" },
    { "history.house_passage_result", "pass" } 
};

dynamic result = await client.bills.get(paramList: parameters, apikey: "sunlight_key");

foreach (dynamic bill in result.results)
{
    Assert.AreEqual("senate", (string)bill.chamber);
    Assert.AreEqual("pass", (string)bill.history.house_passage_result);
}

There are also instances where a parameter name conflicts with a C# reserved keyword. Those can also be escaped by passing them as a dictionary but you can also use the C# argument identifier escape syntax using an @.

dynamic client = new DynamicRestClient("http://openstates.org/api/v1/");

//escape the reserved word "long" with an @ symbol
var result = await client.legislators.geo.get
             (apikey: "sunlight_key", lat: 44.926868, @long: -93.214049);
Assert.IsNotNull(result);
Assert.IsTrue(result.Count > 0);

Passing Content Objects

Putting, patching and posting often requires an object in the request body. In order to accomplish that, pass unamed arguments to the verb (named and unnamed arguments can be used together but all unnamed arguments must proceed the named ones).

In this example, a new Google calendar is created using a POST method. We're using an ExpandoObject because we don't have to use static POCO types. That way both input and output objects can be completely dynamic. Static types can also be passed in this way, and most objects will be serialized as Json.

dynamic google = new DynamicRestClient
("https://www.googleapis.com/calendar/v3/", null, async (request, cancelToken) =>
{
    // this demonstrates how t use the configuration callback to handle authentication 
    var auth = new GoogleOAuth2("email profile https://www.googleapis.com/auth/calendar");
    var token = await auth.Authenticate("", cancelToken);
    Assert.IsNotNull(token, "auth failed");

    request.Headers.Authorization = new AuthenticationHeaderValue("OAuth", token);
});

dynamic calendar = new ExpandoObject();
calendar.summary = "unit_testing";

var list = await google.calendars.post(calendar);

Assert.IsNotNull(list);
Assert.AreEqual(list.summary, "unit_testing");

In the resulting http request, notice the serialized object in the body:

POST https://www.googleapis.com/calendar/v3/calendars HTTP/1.1
Authorization: OAuth xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Accept: application/json, text/json, text/x-json, text/javascript
Content-Type: application/json; charset=utf-8
Host: www.googleapis.com
Expect: 100-continue
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
Content-Length: 26

{"summary":"unit_testing"}

By default, any unnamed object passed to a verb invocation will be serialized as Json into the content body. There are a couple exceptions:

Specific Content Types

There are a few object types that will serialize in specific ways:

• HttpContent: Because the client uses the portable HttpClient internally, a preconfigured HttpContent object will be passed directly to the resulting request
• Stream: A Stream object will be serialized as a stream content type. This is useful for file upload APIs.
• Byte array: A byte[] will be passed as byte array content.
• string: A string will be passed as string content.
• ContentInfo and StreamInfo: These custom classes wrap a content object and allow a specific content MIME type and other content headers to be set.
• IEnumerable<object>: A collection of objects will be sent as multi-part content, with each constituent object being serialized by the above rules

Reserved Types

There are a handful of types, that when an unnamed argument is passed that is that type, will not be serialized as content but will trigger specific behavior during the request.

• CancellationToken: Since all of the rest requests are async, cancellation is supported by passing an unnamed CancellationToken.
• JsonSerializationSettings: Deserialization of response content is handled internally by json.net, an object of this type can be passed to customize how content is deserialized.
• System.Type: By default, content is returned as a dynamic object. To deserialize the response to a static type, pass the desired return type as an unnamed argument (see below).

Return Types

By default, responses are returned as dynamic objects. I find this very easy to use without needing to cobble together static DTO types to match the API. However, static type deserialization is also supported. In order to specify the type of the deserialized response, pass an instance of the desired Type to the rest invocation:

public class Bucket
{
    public string kind { get; set; }
    public string id { get; set; }
    public string selfLink { get; set; }
    public string name { get; set; }
    public DateTime timeCreated { get; set; }
    public int metageneration { get; set; }
    public string location { get; set; }
    public string storageClass { get; set; }
    public string etag { get; set; }
}

[TestMethod]
public async Task DeserializeToStaticType()
{
    dynamic google = new DynamicRestClient("https://www.googleapis.com/");
    dynamic bucketEndPoint = google.storage.v1.b("uspto-pair");

    // by default a dynamic object is returned
    dynamic dynamicBucket = await bucketEndPoint.get();
    Assert.IsNotNull(dynamicBucket);
    Assert.AreEqual(dynamicBucket.name, "uspto-pair"); 
            
    // but if we really want a static type that works too
    Bucket staticBucket = await bucketEndPoint.get(typeof(Bucket));
    Assert.IsNotNull(staticBucket);
    Assert.AreEqual(staticBucket.name, "uspto-pair");
}

Why Not Use Generic Syntax for that?

Dynamic objects support generic type arguments, and it would feel more natural to use that syntax to specify the return type. This can actually be made to work:

dynamic google = new DynamicRestClient("https://www.googleapis.com/");
dynamic bucketEndPoint = google.storage.v1.b("uspto-pair");
            
Bucket staticBucket = await bucketEndPoint.get<Bucket>();
Assert.IsNotNull(staticBucket);
Assert.AreEqual(staticBucket.name, "uspto-pair");

The problem is that, when implementing a custom DynamicObject, the generic type arguments are not made visible to your derived class. To provide custom method handling in a dynamic object, which this library does to acheive its core functionality, you override TryInvokeMember. This method is passed an instance of a InvokerMemberBinder. If you look at the instance passed to your overload, you will find that it is an DLR internal type CSharpInvokeMemberBinder. This class has a private field which holds the generic type arguments, and that field does not have a public accessor.

I am not sure why they chose to support generic dynamic methods but not for classes outside of the DLR, but the only way to get at the type arguments is by reflecting into that private field. I've tried it and it does work, but that could break at any time with an update to the DLR implementation. Hence, the slightly more cumbersome typeof syntax.

Other Supported Return Types

It is also possible to retreive the response in an un-serialized format by passing a type argument of string, byte[], Stream or HttpResponseMessage. For string and byte[] the content will be read and returned in those formats in their entirety. Retruning the Stream and HttpResponseMessage require the caller to dispose of them after use. Retreiving the HttpResponseMessage is another escape mechansim that allows inspection of the response itself, not just its content.

Defaults

It is also possible to initilize the DynamicRestClient with a defaults object that let's you specify auth tokens, a user agent string and other default request parameters and headers. These defaults will be used for any request made using the client instance.

In this example, we first use a helper class to authenticate against a Google account and get an OAuth token. This is then set in a DynamicRestClientDefaults object and any subsequent call to the client will be authenticated with that token. The following code uploads a file to a specific bucket in Google cloud storage.

var auth = new GoogleOAuth2
("email profile https://www.googleapis.com/auth/devstorage.read_write");
var token = await auth.Authenticate("");
Assert.IsNotNull(token, "auth failed");

var defaults = new DynamicRestClientDefaults()
{
    AuthScheme = "OAuth",
    AuthToken = token
};

dynamic google = new DynamicRestClient("https://www.googleapis.com/", defaults);

using (var stream = new StreamInfo(File.OpenRead(@"D:\temp\test2.png"), "image/png"))
{
    dynamic metaData = new ExpandoObject();
    metaData.name = "test2";
    dynamic result = await google.upload.storage.v1.b.unit_tests.o.post
                     (metaData, stream, uploadType: new PostUrlParam("multipart"));
    Assert.IsNotNull(result);
}

Verb Invocation

The invoking of the verbs results in a dynamic method call, which ultimately calls TryInvokeMember. The arguments passed to TryInvokeMember contain the details about the dynamic method, its name and arguments. It is in this method where the request is created, formatted, invoked and its response deserialized. Pretty much all of the heavy lifting happens in this method.

public override bool TryInvokeMember(InvokeMemberBinder binder, object[] args, out object result)
{
    if (binder.IsVerb()) // the method name is one of our http verbs - invoke as such
    {
        var unnamedArgs = binder.GetUnnamedArgs(args);

        // filter our sentinel types out of the unnamed args to be passed on the request
        var requestArgs = unnamedArgs.Where(arg => !arg.IsOfType(_reservedTypes));

        // these are the objects that can be passed as unnamed args that 
        // we use intenrally and do not pass to the request
        var cancelToken = 
            unnamedArgs.OfType<CancellationToken>().FirstOrDefault(CancellationToken.None);
        var serializationSettings = 
            unnamedArgs.OfType<JsonSerializerSettings>().FirstOrNewInstance();

#if EXPERIMENTAL_GENERICS
        // dig the generic type argument out of the binder
        // evil exists within that method
        var returnType = binder.GetGenericTypeArguments().FirstOrDefault(); 
#else
        var returnType = unnamedArgs.OfType<Type>().FirstOrDefault();
#endif
        // if no return type argument provided there is no need for late bound method dispatch
        if (returnType == null)
        {
            // no return type argumentso return result deserialized as dynamic
            // parse out the details of the invocation and have the derived class create a Task
            result = CreateVerbAsyncTask<dynamic>(binder.Name, 
                          requestArgs, 
                          binder.GetNamedArgs(args), 
                          cancelToken, 
                          serializationSettings);
        }
        else
        {
            // we got a type argument (like this: client.get(typeof(SomeType)); )
            // make and invoke the generic implementaiton of the CreateVerbAsyncTask method
            var methodInfo = 
                this.GetType().GetTypeInfo().GetDeclaredMethod("CreateVerbAsyncTask");
            var method = methodInfo.MakeGenericMethod(returnType);
            result = method.Invoke(this, 
                        new object[] { 
                            binder.Name, 
                            requestArgs, 
                            binder.GetNamedArgs(args), 
                            cancelToken, 
                            serializationSettings });
        }
    }
    else // otherwise the method is yet another uri segment
    {
        if (args.Length != 1)
            throw new InvalidOperationException
                  ("The escape sequence can have 1 unnamed parameter");

        // this is for when we escape a url segment by passing it 
        // as an argument to a method invocation
        // example: proxy.segment1("escaped")
        // here we create two new dynamic objects, 1 for "segment1" which is the method name
        // and then we create one for the escaped segment passed as an argument 
        // - "escaped" in the example
        var tmp = CreateProxyNode(this, binder.Name);
        result = CreateProxyNode(tmp, args[0].ToString());
    }

    return true;
}

Transparent Proxies

All of the above examples use a DynamicRestClient that encapsulates the use of HttpClient for communication. This does limit the configurability of the http conversation to whatever is implemented by the library.

There is an abstract RestProxy base class that implements the dynamic Uri creation and invocation logic but does not specify the http client library used for communication. From this base class, a transparent proxy for other http client libraries can be created. The attached code includes a transparent proxy that uses RestSharp for instance. This project actually started with RestSharp but since it doesn't have a portable version, I switched to HttpClient instead which is the version I use most.

An example using RestSharp:

var client = new RestClient("http://openstates.org/api/v1");
client.AddDefaultHeader("X-APIKEY", "you_api_key");

dynamic proxy = new RestSharpProxy(client);

dynamic result = await proxy.metadata.mn.get();
Assert.IsNotNull(result);
Assert.IsTrue(result.name == "Minnesota");

Points of Interest

DynamicObject really has some interesting possibilities and once you wrap your head around is very simple to use. All of the above does not really require all that much code.

The Unit Tests

The unit tests have further examples of the various verbs, dynamic OAuth2 and miscellaneous other stuff. If you try to run the unit tests take a close look at the CredentialStore class in the unit test project. It's pretty straightforward and you can use it to supply your own API keys while keeping them out of the code. Virtually all of the integration tests require an API key for the endpoints they hit, so most will fail spectacularly without them.

History

• 20^th April, 2014 - Initial version
• 24^th April, 2014 - Parameter name escape mechanism
• 21^st June, 2014 - HttpClient addition
• 25^th December, 2014 - Rewrite to use DynamicRestClient as the primary examples
• 7^th February, 2015 - Add section about return types