Querying Last.fm web API with F#

Introduction

Let's imagine that you have an edgy musical taste so you would like to recommend to your friends only those artists which are the most mainstream. If you have a profile on last.fm, then you could write a small tool which would query and process your listening statistics to automate this task.

My tool for this job is F# programming language and I'll show you some benefits of it such as type providers or the ease of unit testing functions comparing to objects.

The complete source code can be accessed here.

Workflow

The task is as follows:

1. Get top 50 artists from last.fm chart.
2. Transform them to the collection of their names.
3. Remove those ones which were already recommended.
4. Replace non-url characters in artist names to make further API call.
5. For each artist, make additional API call to get a number of listeners.
6. Transform given information to a more minimalistic data type which would contain only artist name and listeners count.
7. Order artists by the count of listeners, implying that those which have more listeners are more mainstream.

Note how this workflow is expressed by F# pipeline in a truly idiomatic way.

let result = getTopArtists
                |> getTopArtistNames
                |> removeAlreadyRecomendedArtists
                |> getUrlEncodedArtistNames 
                |> mapArtistNamesToArtistInfo getArtistInfo
                |> getArtistsShortInfo
                |> orderArtistsByListenersCount          

Making Use of Type Providers

Type providers are arguably the most advertised feature of F#. Type providers allow us to access a lot of contexts such as Web APIs, database schemas, etc. as strongly typed entities which allow us to get compiler time support and some nice perks as IDE autocomplete.

To make use of it in our application, we:

1. Import FSharp.Data
2. Declare snippet of our API response:
   F#

   let [<Literal>] TopArtistsSample = """{  
      "topartists":{  
         "artist":[  
            {  
               "name":"Porcupine Tree",
               //skipped for the sake of breivety
            }
         ],
         "@attr":{  
            "user":"Morbid_soul",
            "page":"1",
            "perPage":"2",
            "totalPages":"165",
            "total":"330"
         }
      }
       }"""

3. Construct type from our sample via JsonProvider:
   F#

   type TopArtists = JsonProvider<TopArtistsSample>

4. Enjoy compile time support of strongly typed response.

   

Using Higher Order Functions to Improve Unit Testing

Let's take a close look at the following function:

let mapArtistNamesToArtistInfo getArtistInfoFn artists = 
    artists
        |> Array.map (fun i -> getArtistInfoFn i) 

getArtistInfoFn responds for interaction with remote web API. Here's how the unit testing of such a scenario is performed.

let getArtistInfoStub input = 
        match input with
        | "Nokturanl Mortum" -> 1
        | "Krobak" -> 2
        | _ -> 3

[<Fact>]
let mapArtistNamesToArtistInfo_returns_expected_result() =
    let result = mapArtistNamesToArtistInfo getArtistInfoStub 
                   [| "Nokturanl Mortum"; "Heinali"; "Krobak"|]
    Assert.Equal(result.[0], 1)
    Assert.Equal(result.[1], 3)
    Assert.Equal(result.[2], 2)    

This is far more elegant than the typical testable OO-solution which would require introducing an interface, injecting it into caller class and introducing some heavyweight mocking library inside the test project.
One may argue that injecting an impure function into pure is not truly functional way but F# is quite a forgiving language and allows us not to come with some clever concepts as free monads, etc.

Error Handling

The attentive reader may have noticed that we rely on web API working faultlessly which is not a sign of robust programming. For proper handling, we will employ the concept of railway oriented programming.
The main idea is to encode successful and unsuccessful execution of function into return type so that all functions in a pipeline would handle successful result with some useful business logic and the unsuccessful result would be excluded from further execution.

But I strongly encourage you not to take my word for it but read the original article instead which explains this concept in far more detail.

The recipe is the following:

type Result<'TSuccess,'TFailure> = 
    | Success of 'TSuccess
    | Failure of 'TFailure

let switch switchFunction1 switchFunction2 input = 
    match switchFunction1 input with
    | Success s -> switchFunction2 s 
    | Failure f -> Failure f

let (>=>) switchFunction1 switchFunction2 input = 
    switch switchFunction1 switchFunction2 input

Now we can wrap our return value into the provided type:

let getTopArtists () = 
    try
        let path = String.Format(getTopArtistsPattern, baseUrl, userName, apiKey)
        let data = Http.Request(path)
        match data.Body with
        | Text text -> Success(TopArtists.Parse(text).Topartists.Artist)
        | _ -> Failure "getTopArtists. Unexpected format of reponse message"
    with
    | ex -> Failure ex.Message

So with this, the pipeline would transform to:

let pipeline = 
    getTopArtists
        >=> getTopArtistNames
        >=> removeAlreadyRecomendedArtists
        >=> getUrlEncodedArtistNames 
        >=> mapArtistNamesToArtistInfo getArtistInfo
        >=> getArtistsShortInfo
        >=> orderArtistsByListenersCount

Let's also take a look at unit test snippet to get an overall feeling how the caller works with function output:

[<Fact>]
    let orderArtistsByListenersCount_returns_expected_result() =
        let Satie = {name = "Erik Satie"; listeners = 750000}
        let Chopin = {name ="Frederic Chopin"; listeners = 1200000}
        let Barber = {name = "Samuel Barber"; listeners = 371000}
        let artists = [|Satie; Chopin; Barber|]
        let result = orderArtistsByListenersCount artists
        match result with
        | Success s -> 
            Assert.Equal(s.[0], Chopin)
            Assert.Equal(s.[1], Satie)
            Assert.Equal(s.[2], Barber)
        | Failure _ -> Assert.True(false)

Using Built-in Result type

F# comes with a built-in Result type which allows us to ditch ROPHelper.

Our pipeline now looks as follows:

let pipeline = 
    getTopArtists()
        |> Result.bind getTopArtistNames
        |> Result.bind removeAlreadyRecomendedArtists
        |> Result.bind getUrlEncodedArtistNames 
        |> Result.bind (mapArtistNamesToArtistInfo getArtistInfo)
        |> Result.bind getArtistsShortInfo
        |> Result.bind orderArtistsByListenersCount

Note that we had to make our getTopArtists() to accept unit to be accepted by Result.bind. We pattern match the result as below:

[<Fact>]
    let getUrlEncodedArtistNames_returns_expected_result() =
        let result = getUrlEncodedArtistNames [|"Bohren & Der Club Of Gore"; "Цукор Біла Смерть"|]
        match result with
        | Ok s ->
            Assert.Equal(s.[0], "Bohren+%26+Der+Club+Of+Gore")
            Assert.Equal(s.[1], 
         "%d0%a6%d1%83%d0%ba%d0%be%d1%80+%d0%91%d1%96%d0%bb%d0%b0+%d0%a1%d0%bc%d0%b5%d1%80%d1%82%d1%8c")
        | Error _ -> Assert.True(false)

Conclusion

I hope those who had the first encounter with F# today didn't find the uncommon syntax too complex to appreciate such benefits of language as type providers or easy unit testing due to function composition. And I also hope that those who are already on a solid ground found the technique of railway oriented programming quite useful.

Revision History

• 7.11.2017 - Initial version
• 24.03.2019 - Using built-in Result type