FETCH! Retrieve data from a remote web server with one line of code (with MVC reCAPTCHA example)

	Source (RedCell.Net on GitHub)
	Binary Component (RedCell.Net.dll)
	Compiled Help (RedCell.Net.chm)

Introduction

This is one of those classes I've had in my toolbox for a many years, and I still use it regularly. It was inspired by the FreeBSD fetch command, which is more intuitive than curl. It compiles against version 2.0 of the .NET Framework, so it can be used in almost any desktop or web application. Its only dependency is System.dll.

Background

Sometimes you just want to get the result of an HTTP request and use it. Maybe you want to download a whole webpage to scrape its content for some data, or maybe your're dealing with a web service that just returns a number, string, some XML, or some JSON. Fetch was create for these cases. It features:

• Make simple GET or POST requests with a single line of code.
• Include credentials for authentication.
• Configurable retry and timeout.
• Modifyable headers.
• Get the result returned in the type of your choice.

Using the code

To use the Fetch class in your code, either add Fetch.cs to your project, or reference RedCell.Net.dll in your project.

Simple Use (static)

There are static Get and Post methods for simple requests. These come in two varieties, generic and binary. The binary methods return byte[]. Here are some examples of static usage:

using RedCell.Net;

string url = "http://example.com/api";

// Retrieve some binary data.
byte[] image = Fetch.Get(url);

// Retrieve a string.
string slogan = Fetch.Get<string>(url);

// Post some data and expect an int back.
var formData = new Dictionary<string,string> {
    { "firstname", "Yvan" },
    { "lastname", "Rodrigues" }
};
int age = Fetch.Post<int>(url, formData);

The static methods will return null if an error occurs. Some default values can be set for static invocation:

Fetch.DefaultRetries = 5;         // Number of times to retry.
Fetch.DefaultTimeout = 60000;     // Time to wait for a response (ms).
Fetch.DefaultRetrySleep = 10000;  // Pause between retries (ms).

Instance Use

Instantiation of the class is required to access additional features, and for lower-level access to options.

using System.Net;  
using RedCell.Net;

string url = "http://example.com/api";

// Create a fetcher.
var f = new Fetch(url);

// Set some options.
f.Retries = 1;
f.Timeout = 10;

// Include authentication credentials
f.Credential = new NetworkCredential("yvan", "happy-fun-password");

// Make the request.
// The result will be stored as byte[] in the ResponseData property.
f.Get(); 

// Exceptions are caught, so we have to check for success.
if(!f.Success)
{
    // If the fetch failed, examine f.Response to find out why.
}

float response = f.ConvertResponse<float>();

Applied Example: Recaptcha MVC

Today I wanted to add the new reCAPTCHA to a web form in an MVC web application. There are tons of libraries to do this on Nuget and GitHub, but I really wanted to cut-and-paste something dead simple. Here's the simple approach, using Fetch. In this example, my form uses the same view for showing and handling the form, like:

@if (Request.HttpMethod == "POST")
{
    <p>Thanks!</p>
}
@else
{
    <form> 20 questions </form>
}

Three steps

1. Sign up and get your keys.

2. Add to your web form view before the submit button:

<script src="https://www.google.com/recaptcha/api.js"></script>
<div class="g-recaptcha" data-sitekey="put-your-site-token-here" ></div>

protected bool VerifyRecaptcha()
{
    const string key = "your-secret-key";
    const string url = "https://www.google.com/recaptcha/api/siteverify";
    string recaptcha = Request.Form["g-recaptcha-response"];
    string ip = Request.ServerVariables["REMOTE_HOST"];
    var values = new Dictionary<string, string>
    {
        { "secret", key },
        { "response", recaptcha },
        { "remoteip", ip }
    };
    string response = Fetch.Post<string>(url, values);
    if (response == null) return false; // You may wish to do a more sophisticated handler.
    var data = JObject.Parse(response);
    return data.Value<bool>("success");
}

The g-recaptcha-response field from the form is POSTed to the web service along with the private key and the IP address. We use Fetch.Post with string as the type argument, knowing we are expecting a JSON object. Json.NET is then used to parse the response and check for success.

A nice approach is to create a base class for your controllers e.g. RecaptchaController that contains this class, and have your other classes subclass it.

4. Finally, invoke it from your controller:

public ActionResult MoreInformation()
{
    if (Request.HttpMethod == "POST" && VerifyRecaptcha())
        SendConfirmation("Thanks, we'll be in touch);

    return View();
}

Points of Interest

Fetch is a simple wrapper for HttpWebRequest and HttpWebResponse. A basic request with these is normally about 20 lines of code. The main reason is that getting the data content into or out of these classes requires handling of Streams. Internally, Fetch creates a MemoryStream, copies the web stream, and turns it into byte[].

One of the nice features is the ability to use the generic methods to get strongly typed data. This is performed internally by the ConvertResponse<T> method. The tricky part is that arbitrary types cannot be cast to T (unless the where constraint is used); however object can be cast to a generic type, so values are boxed before casting.

History

• August 13, 2015 · v1.0.0 · Original article
• August 20, 2015 · v1.0.1 · Bug fix