Combining Liberator and Korma to Build a Web-Service in Clojure

In a recent episode of the Code Speak Loop podcast I mentioned two Clojure projects: Liberator, designed to build REST services, and Korma, allowing to talk to relational databases easily. I’ve been working with these libraries lately and it turns out they play quite nice together. In this post and the related repository on GitHub I will show the way I combined Liberator and Korma to build a simple RESTful application so that anyone who wants to do something similar has an example. I did not put much effort into separating concerns and making code clean in this sample, still I think it conveys the general ideas properly.<o:p>

Here we will set up a task-list application, which would allow to view, add and edit tasks over HTTP. There is a bit more to it on Github, but I won’t cover many of the details here. For a database I used a local Postgres installation with a very simple table structure – there is a schema.sql script in the repository. It should not matter much whether you use Postgres or not, although if you pick some other DBMS you will have to change the DB connection configuration in the application (see below). Besides, some problems may arise with timestamps.<o:p>

Let us start with the database. To talk to it we use Korma, and Korma in turn uses entities. These are the descriptions of the database tables written in Clojure with a defentity macro. Entity definition normally includes a set of keys, a list of fields to select from the corresponding table by default and possibly a name of the table – if it differs from the name of the entity. Additionally – and that’s the coolest part – entities might include relationships, which allow to extract linked entities seamlessly.<o:p>

(declare tag)

(defentity task
  (pk :task_id)
  (entity-fields :task_id
           :title
           :description
           :is_done
           :is_cancelled
           :created_time
           :finished_time)
  (many-to-many tag :tasktag))

(defentity tag
  (pk :tag_id)
  (entity-fields :tag_id
           :tag)
  (many-to-many task :tasktag))

(defentity tasktag
  (entity-fields :task_id :tag_id))

In our application there are only three entities – task, tag and tasktag. Both in task and tag we specify that tasks are related to tags with a many-to-many link – to do this we only need to specify the second entity and the name of the linking table (:tasktag) in our case. We don’t define any relationships for the tasktag entity – that’s because we need it only to insert and delete records, which link tasks and tags together. To achieve our other goals the relationships defined on task and tag are pretty enough. (Note however, that I don’t show tag and tasktag entities at work in this post – take a look at the code on Github.) You can find a lot of info regarding entities on the Korma site.

Once we defined the entities, we have to tell Korma where we want to get them from – that is specify a database connection. In Korma you do this by means of defdb macro passing it a connection description generated from a dictionary. I use postgresfunction provided by Korma that will setup all the required parameters for connecting to a Postgres database. There are plenty of other helpers like that in Korma – check them in the docs.<o:p>

(def dbcon 
  (postgres {:db "libekorma" :user "postgres" :password "Aw34esz"}))

(defdb dbconnection dbcon)

Now that we defined the entities it’s time to give access to them through a resource. Resource is the fundamental concept in Liberator, which binds together various handlers and parameters, that define what will it do under which conditions. We create a resource with defresrouce macro and for our simple case we will specify only the :available-media-type – we deal with JSON, :allowed-methods – GET is enough so far, and a function to :handle-ok – this one will get tasks from the database and encode them in JSON format – that’s what users will get. Even this simple example shows that Liberator allows to manage a lot of HTTP-stuff without much ceremony. The most important part of the resource for now is the :handle-ok function, called when the resource thinks it should respond with 200 HTTP code – that’s what happens when user sends GET request because we don’t have any restrictions and in this case we should respond with a list of tasks.<o:p>

(defresource tasks-r
  :available-media-types ["application/json"]
  :allowed-methods [:get]
  :handle-ok (fn [_]
        (json/write-str (select task (with tag))))) 

To make it all work we have to do only one more thing: define a Compojure route that will expose the tasks-r resource. Its definition starts with ANY, which means that the route accepts any HTTP method. Liberator handles allowed methods through its own mechanism (:allowed-methods) and thus there is usually no need to make Compojure expect a specific verb.<o:p>

(defroutes app
  (ANY "/tasks" [] tasks-r)) 

(You can check the project.clj for a list of libraries that we use.)

However, if you try to request some data from /tasks, you will likely run into an error message telling that the app can’t produce JSON output because it doesn’t know what to do with timestamps. Even though this does sound scary, thanks to the extensibility of clojure.data.json this problem is pretty easy to deal with – we just extend the Timestamp type with a simplistic implementation of the JSONWriter protocol:<o:p>

(extend-type java.sql.Timestamp
  json/JSONWriter
  (-write [date out]
  (json/-write (str date) out)))

Now you can check that everything, including this last trick, works fine. That means one can retrieve the list of tasks with GET request and observe the records stored in the database (be sure to insert some for testing – there is a sample.clj, which can do this for you). This is not too impressive though, so let’s proceed and allow for tasks creation with POST in the same resource:<o:p>

(defn tasks-request-malformed?
  [{{method :request-method} :request :as ctx}]
  (if (= :post method)
    (let [task-data (util/parse-json-body ctx)]
      (if (empty? (:title task-data))
        [true {:message "Task title missing or empty"}]
        [false {:task-data task-data}]))
    false))

(defresource tasks-r
  :available-media-types ["application/json"]
  :allowed-methods [:get :post]
  :malformed? tasks-request-malformed?
  :post!
    (fn [{task-data :task-data}]
      (let [data (into task-data {:created_time (util/cur-time)})]
        (insert task (values data))))
  :handle-ok (fn [_]
        (util/to-json
          (select task (with tag)))))

Here we add 2 things. First, when someone’s posting data to us we want to check that it complies with our requirements. Validation of this kind can be done in the malformed? handler of the corresponding resource. Particularly, for tasks we don’t allow empty :title, so for requests with bad title our tasks-request-malformed? function returns a vector of true (yes, the request is malformed) accompanied by an error message. If, on the other side, a proper title is present in the posted data, the function will return false – not malformed – and a dictionary including the parsed request data under the :task-data key. This illustrates the proper way to pass data between various decision points in Liberator: if along with the result of the check (true or false) you return a map from the handler, liberator will merge it into the context and downstream handlers will have access to whatever there is in the dictionary.<o:p>

In our example the data provided by the malformed? is used by post! handler, which gets it from the context by means of destructuring. Beside this, in post we add the :created_time field to the same dictionary and call Korma’s insert with it. That’s it, we enabled creating tasks – core functionality is here!<o:p>

One particular thing to note are the calls to the cur-timefunction. There is nothing magical about it – I just use it to abstract away instantiation of the Timestamps for the time columns in the database:<o:p>

(defn cur-time []
  (Timestamp. (.getTime (Date.))))

If you take a closer look at the malformed? handler above, you’ll notice that it uses the parse-json-body utility function. This one combines two other functions and json/read-str to get the JSON body of the request from context, turn it into a Clojure map and transform its string keys into keywords. In other words, the function creates an easy to handle dictionary from a raw stream buried deep in the context. Be aware that the keywordify function used here is not recursive, so only the top-level keys will become keywords, while nested dictionaries will still have string keys.<o:p>

(defn body-as-string [ctx]
  (if-let [body (get-in ctx [:request :body])]
  (condp instance? body
    java.lang.String body
    (slurp (io/reader body)))))

(defn keywordify [dict]
  (into {}
    (map (fn [[k v]] [(keyword k) v]) dict)))

(defn parse-json-body [context]
  (if-let [body (body-as-string context)]
    (keywordify (json/read-str body))
    {}))

Liberator allows to define a lot of various handlers thus opening doors for managing any particular condition in proper place and time. The general idea is that when processing a request Liberator will navigate the decision graph and execute handlers defined for visited nodes. In the example above we used only the malformed? decision point to parse and check the incoming request. Next, we will implement a separate resource for deleting and updating individual tasks and see how one can implement other handlers.<o:p>

Let us start with something simple – deletes. We actually need to define only two handlers: delete! and exists? As a bonus, we will also implement one under :handle-ok to allow for getting tasks by ID – just because it is very easy:<o:p>

(defresource one-task-r [task-id]
  :available-media-types ["application/json"]
  :allowed-methods [:get :delete :put]
  :exists?
    (fn [_]
      (if-let [task
        (first
          (select task
            (with tag)
            (where {:task_id task-id})))]
        [true {:task task}]
        [false {:message "Task not found"}]))
  :delete!
    (fn [{{task-id :task_id} :task}]
      (delete task
        (where {:task_id task-id})))
  :handle-ok
    (fn [{task :task}]
      (json/write-str task)))

Here we remove tasks in the delete! handler with a simple call to Korma’s delete with task entity and a where clause restricting the ID of the task. However, the function provided under the :delete! keyword gets called only in case the one specified with :exists? yields truth or a vector starting with truth – there is little sense to deleting missing tasks. Our implementation of the exists? handler attempts to select the task from the database by its ID and upon success returns it together with true. Here the pattern is the same as in the malformed? handler – we use the dictionary to pass data around so that , for example, downstream handlers don’t have to query database once more. In case you update or delete records it makes a lot of sense to retrieve them in the exists? handler and then use them when they are needed.<o:p>

What makes this resource very different from the previous one is that it takes an argument – task-id. This might seem strange because otherwise resources look more like dictionaries, but that’s the thing that Liberator handles without any work required from us – we just accept this gift. As for passing the parameter in, we do it in the route definition like this:<o:p>

(defroutes app
  (ANY "/task/:task-id" [task-id] (one-task-r (Integer/parseInt task-id)))

Now we can remove the tasks – and only the existing ones. Deletes, however, are very simple in comparison to updates, which we are going to implement next. First thing that we need is a malformed? handler that will parse and validate the request – we have already seen something like this in the previous resource:<o:p>

(defn task-update-request-malformed?
  [{{method :request-method} :request :as ctx}]
  (if (= :put method)
    (let [task-data (util/parse-json-body ctx)]
      (cond
        (empty? task-data)
          [true {:message "No new values specififed"}]
         (and (contains? task-data :title)
            (empty? (:title task-data)))
          [true {:message "Empty title is not allowed"}]
        true
          [false {:task-data task-data}]))
    false))

The new thing is the conflict? handler. In our case, it is pretty simple and just verifies that the task doesn’t end up completed and cancelled at the same time – this is a forbidden state:<o:p>

(defn task-update-conflict? [{new-task :task-data old-task :task}]
  (let [combined (into old-task new-task)]
    (if (and (:is_done combined) (:is_cancelled combined))
      [true {:message "Invalid state after update"}]
      false)))

As you might guess (or discover from the decision graph), Liberator invokes the conflict? handler for put requests somewhere between the exists? and put! handlers. This means that you already have access to data extracted by exists? and can check whether the update can cause any problems here, without turning the actual put! handler into a state validation mess. Note that the OK return value here is false – meaning no conflict. <o:p>

Having this handler separated is cool because there are usually quite a few other things that you have to decide upon when executing update, so it might end up messy by itself:<o:p>

(defn update-task [{new-task :task-data old-task :task}]
  (let [just-finished?
      (or
        (and (:is_done new-task)
           (not (:is_done old-task)))
        (and (:is_cancelled new-task)
           (not (:is_cancelled old-task))))
      finished-time-dict
      (if just-finished?
        {:finished_time (util/cur-time)}
        {})
      updated
      (into finished-time-dict
        (filter
          (fn [[k _]] (#{:title :description :is_cancelled :is_done} k))
          new-task))]
    (update task
      (set-fields updated)
      (where {:task_id (:task_id old-task)}))))

Our update-task function evaluates the :finished_time field if needed and passes it to Korma’s update together with the values coming from the user. It is important to filter the latter and exclude the fields that should not be updated under any conditions – e.g. :task_id and :created_time. That’s what we do when assembling the updated dictionary. <o:p>

(defresource one-task-r [task-id]
  :available-media-types ["application/json"]
  :allowed-methods [:get :delete :put]
  :can-put-to-missing? false
  :malformed? task-update-request-malformed?
  :conflict? task-update-conflict?
  :put! update-task
  ; + a bit more - see above
  )

In the resource we specify these routines under the malformed?, conflict? and put! keys. Additionally, we have :can-put-to-missing? set to false, which prevents updates to non-existent tasks. The update-related functions do look quite complex, but they’d be absolutely dreadful without the separation of concerns offered by Liberator and that is its main strength.<o:p>

Another cool thing about this library is that it automatically manages the execution flow in a fully HTTP-aware way. That is, when building an app, you don’t need to think, for instance, of producing appropriate error codes and giving them back to user in the required form – Liberator will do everything on its own. On the other side, it doesn’t stand in your way and allows to fine-tune the resources the way you need them.<o:p>

As for Korma, it’s key advantege is simplicity – at least that’s what I love about it. It isn’t an overweight ORM or something – it just allows to run queries against a database from Clojure, but it does it in a very natural way. However, I also can’t leave out the relationships feature of Korma, which permits linking the entities together in a straightforward manner.<o:p>

Our simple example also shows that these two libraries work great together thanks to how Liberator makes you consider only one thing at a time and how Korma simplifies access to data. I began using these about a month ago and I must admit that I truly love this way of producing REST services.<o:p>

The sample project that I show here is a bit bigger than I managed to stick into the post. For example, it allows to add tags to the tasks and to ask for a full list of tasks tagged with a particular word. This doesn’t introduce many new concepts apart from demonstrating Korma’s relationships in action – I just tried to go further along the road of showing Korma and Liberator. You can find full sources on GitHub – do clone the repo and play with the code!<o:p>

I will appreciate your comments – both regarding the examples here and about your experience at building web services with Clojure! If you spot an error in the code or just can’t figure out what a particular piece does – please let me know. I’ll be happy to fix the mistakes and explain what I meant!<o:p>

CodeProject