Preface
This article and the associated code should really be part of the main distribution of
RaptorDB but since it is a proof of concept and I want feedback from users I have decided to create a separate article, if all goes well then I will integrate it into the main release, so please tell me what you think.
Introduction
The programming world is changing and more and more leaning towards languages like javascript and web application (to the dismay of old programmers like yours truly), and if you want to survive you must adapt, so this article and the associated code is the implementation of a REST web interface for RaptorDB.
There is a couple of reasons for wanting to use RaptorDB with web applications:
- You already have .net applications using
RaptorDB and you want to leverage your existing infrastructure.
- Even if you don't use .net then using a high performance nosql database like
RaptorDB is quite appealing.
This article uses the following references:
What is REST?
At it's root a REST interface is simply the implementation of HTTP's POST GET PUT DELETE instructions on URL paths. This means that you can supply paths and parameters and query information for that path. Data is usually returned in XML or JSON format to the caller.
What does RaptorDB REST do?
RaptorDB REST will start a HTTP listener and process incoming request from a routing table you provide. The processing consists of saving json documents and retrieving the same or data from views which have been created from those documents.
Limitations & Requirements
The main limitation in using this REST interface is that you loose polymorphism because the json to object deserializer cannot determine nested types since that information is not contained in the json input string (fastJSON's $type extension).
To POST json documents you must supply a string GUID value, this is needed to uniquely identify that document, if you are doing multi-user applications then you should not have a problem with this since generating a client side ID is preferred to round tripping to a server.
How to use RaptorDB REST
You can easily test the REST interface by running the testing.exe application form the output folder. This console application will startup the REST server on port 8080 and generate 100,000 Salesinvoice entities and insert them into the RaptorDB database.
The code to do this is simply :
var rest = new RestServer(8080, @"..\RaptorDBdata");
But before this you should get to know the following concepts.
Concepts
There are the following key concepts which you should be aware of when using
RaptorDB REST :
- REST specific : URLs, routing tables, entity containers
Also a determining factor is the existence of already defined entity classes, since both routes and views depend on entities or data containers.
You can use RaptorDB REST in the following two ways:
- In your compiled code by initializing through code
- With a route script file
The script file method is really powerful and is the preferred way to do things, since you can change the implementation at run-time.
URL
All REST services start off from a URL path so you should create paths that make sense with meaning full names, an endpoint to a URL can either be one of the following :
- a
POST end point for inserting/updating data - a
GET end point to a RaptorDB View - a
GET end point to a aggregate function on a RaptorDB View
Routing table
All the goodness is in the routing table, this is similar to the Express framework for nodejs in that you create URL paths and supply a closure function to do the work. This routing table is filled by
*.route files in the "
Datafolder\Routing" folder, this allows you to break-down your application into logically separate units and manage them in isolation and the system will assemble the pieces at run-time.
Useful tools
To test a REST interface I have found the following tool indispensable :
Sample route code
The sample crm.route file consists of the following sections :
using System;
using System.Collections.Generic;
using System.Linq;
...
The above snippet is from the top of the code which allows you to load existing DLL's with the ref comment and link to your script file to access already defined types.
...
namespace crm
{
public class Customer
{
public string Name;
public string Address;
}
... With-in the namespace above we have defined a
Customer class to map the incoming json object POSTs since
RaptorDB needs .net classes to operate upon, if you have existing infrastructure and applications in place with entity types you can link with those .net types instead of defining here.
...
public class crmrouting : IRDBRouting
{
public void Initialize(IRouteAPI api)
{
api.AddRoute(new RDBRoute { URL = "crm/customer", EntityType = typeof(Customer) });
api.AddRoute(new RDBRoute { URL = "crm/customerview", Viewname = "customer"});
...
The above code is a routing table definition which is inherited from the IRDBRouting interface. Within the Initialize() method we have defined a POST-ing handler for the "crm/customer" URL and maps the incoming json to the Customer type.
The second line maps GET request to the "crm/customerview" URL to a RaptorDB view by the name of "customer" that view is defined later in the file.
...
api.AddRoute(new RDBRoute
{
URL = "testing/function",
function = (rdb, filter) =>
{
var q = rdb.Query<SalesItemRowsViewRowSchema>(filter);
var res = from x in q.Rows
group x by x.Product into g
select new
{
Product = g.Key,
TotalPrice = g.Sum(p => p.Price),
TotalQTY = g.Sum(p => p.QTY)
};
return res.ToList<object>();
}
});
...
The above route in the Initialize() method is a very powerful closure to compute an aggregate on a RaptorDB view.
You are free to do anything with the function with the full power of .net LINQ at your disposal, the filter parameter is what is given from the REST call by the user.
Examples of all the above are shown below.
..
[RegisterView]
public class CustomerView : View<Customer>
{
public class CustomerRowSchema : RDBSchema
{
public string Name;
public string Address;
}
public CustomerView()
{
this.Name = "Customer";
...
To the end of the
crm.route file we have defined a
RaptorDB view for the
Customer entity which we defined at the start of the file.
So in one script file we have created a handler and logic for processing data all in a type-safe way which will get compiled and loaded at run-time so we have the ease of change of scripts and the full power and performance of compiled languages.
Running the sample application
Lets get started working with the sample application. First lets just query:
http:
From the above we can see the basic structure for doing queries in that we have a query filter based on the schema for a RaptorDB view 'serial<100' and the paging extension 'start=100' and 'count=10' you can omit any of these and you will get back the related data.
In the results we can see the data rows returned and also how many rows the query had in the TotalCount field and how many were return in the Count field.
To see the count of all the data in a view we can do the following which is a short cut and the results are in the TotalCount field :
http:
We can call the aggregate in a similar way which is computing the group by on all the rows :
http:
or with a filter:
http:
Posting is a little more involved since we can't do it through the address box in the browser, so open PostMan in Google Chrome and select POST from the drop down and type the following :
http:
We are supplying the following json document as "raw" data, by pressing "Send" we should get a reply of "posted :..." :
{"name":"aa","address":"safasdfasd","age":9090,"id":"A5AE46F0-E114-4659-A4AF-F285CD00A93D"}
A requirement of RaptorDB REST is that you must supply the GUID for that document when posting and presumably the json document contains the same but you can omit it.
To get back the original json we can GET from the POST URL supplying the GUID :
http:
If all went well, we can query our "customer" RaptorDB view which we have defined and see our document in there:
http:
The inner workings
At the heart of
RaptorDB's REST interface is the
HTTPListener, this is a great piece of work done by the .net team which is under-rated. To get the power of this class lets see how we use it:
_server = new HttpListener();
_server.Prefixes.Add("http://*:" + _port + "/");
_server.Start();
while (_run)
{
var context = _server.BeginGetContext(new AsyncCallback(ListenerCallback), _server);
context.AsyncWaitHandle.WaitOne();
} That is all the code! and it is async and high performance (although the sync version is about the same). The HTTPListener takes a prefix which it will listen on and you handle the AsyncCallBack like the following :
private void ListenerCallback(IAsyncResult ar)
{
var listener = ar.AsyncState as HttpListener;
var ctx = listener.EndGetContext(ar);
string path = ctx.Request.Url.GetComponents(UriComponents.Path, UriFormat.Unescaped).ToLower();
RDBRoute r = null;
ctx.Response.ContentEncoding = UTF8Encoding.UTF8;
if (_routing.TryGetValue(path, out r))
{
string mth = ctx.Request.HttpMethod;
if (mth == "POST" || mth == "PUT")
ProcessPOST(_rdb, ctx, path, r);
else
ProcessGET(_rdb, ctx, path, r);
}
else
WriteResponse(ctx, 404, "route path not found : " + ctx.Request.Url.GetComponents(UriComponents.Path, UriFormat.Unescaped));
ctx.Response.OutputStream.Close();
}
So in the above code we are checking the routing table for a matching path and handling the request.
How the RestServer works
At a high level the
RestServer() works in the following way:
As you can see the RestServer takes a json document and maps that to a .net object and sends it to RaptorDB as it expects, since there might be more in the original json document that you did not anticipate the RestServer also saves that original to a Key/Value store.
All get requests to the json document will be processed from the original json stored, not the mapped version in RaptorDB.
Queries are processed from views stored in RaptorDB.
Appendix v1.0.1 - the weird and wonderful world of the web
While trying to get the REST interface working with a javascript client, I have run into some of the weirdness of the http protocol and how browsers handle requests. The following fix took me a day of frustration to find out, which I was only able to do when I enabled the debugger in the Chrome browser since the browser silently does nothing and you only see the error message if you have enabled the debugger.
The problem is that the browsers require the json payload to be from the same domain address (for cross site scripting problems) as the page, to overcome this you need to add the following line on the REST server:
Response.AppendHeader("Access-Control-Allow-Origin", "*"); Now for the wonderful, since json payloads can get really big you need some sort of compression on it otherwise transferring that much data can be slow and block the browser. To do compression all that is required is the following :
byte[] b = Encoding.UTF8.GetBytes(msg);
if (b.Length > 100*1024)
{
using (var ms = new MemoryStream())
{
using (var zip = new GZipStream(ms, CompressionMode.Compress, true))
zip.Write(b, 0, b.Length);
b = ms.ToArray();
}
ctx.Response.AddHeader("Content-Encoding", "gzip");
}
ctx.Response.ContentLength64 = b.LongLength;
The great thing about the above is that it is totally transparent for the client javascript and it is handled by the server and the browser. As you can see if the message being sent is greater than 100Kb then it will be compressed. This saves a lot of time and bandwidth.
Also in this release the following internal routes have been added:
- RaptorDB/GetRoutes : will give you a list of the available registered routes
- RaptorDB/GetViews : will give you a list of
RaptorDB views registered - RaptorDB/GetSchema : will give you the schema for a view you query
testpage.html
To help test the REST interface using a real javascript client instead of using POSTMan I have added the testpage.html file which you can see below:
The above page uses zepto.js [http://zeptojs.com/] as a replacement for jQuery which is smaller and does the same thing.
By pressing the buttons on this page you will get a pretty [arguably] table representation of the json sent from the server. You can change the values in the textboxes and get different results.
Appendix v1.1.0 - Sorting
In this version you can now do sorting while paging the data, to do this you append an OrderBy clause to your URL calling :
http:
As you can see from the above the query will return 10 items sorted in descending order on the serial column.
Also a nice feature in this release is the ability to serve static content, so now if you call the base URL for the server you will get the testpage.html back, as you can see from the image below I have changed the html so you can easily change the prefix and call the functions from other systems :
Previous Versions
You can download previous versions here:
History
-
Initial Release v1.0.0 : 5th November 2013
- Update v1.0.1 : 16th November 2013
- post a Guid ID with or without single or double qoutes
- access control to everyone on http header
- compress json if over a threshold (default 100kb)
- start and count in query with ? or & deliminator
- added information routes under RaptorDB/GetViews, RaptorDB/GetRoutes, RaptorDB/GetSchema
- added testpage.html
- Update v1.1.0 : 5th December 2013
- added sort to queries
- the rest server can now serve static content and the testpage.html by calling the root url
- removed extra query overloads in favour of the new model
