Introduction
This article demonstrates how to integrate the Google Search REST API into a .NET project using managed code (C#). The REST API requires no license key, and queries are unlimited.
The article also covers the deserialization of a JSON response into .NET objects, and workarounds for some of the restrictions / features that come with the use of the API.
Queries to the REST API can be made in any web browser by entering the URL and search terms in the address bar. A call such as the following...
http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=Earth%Day
returns...
{"responseData":
{"results":[{"GsearchResultClass":"GwebSearch",
"unescapedUrl":"http://www.earthday.net/",
"url":"http://www.earthday.net/",
"visibleUrl":"www.earthday.net",
"cacheUrl":"http://www.google.com/search?
q\u003dcache:szJHSzSCm38J:www.earthday.net",
"title":"\u003cb\u003eEarth Day\u003c/b\u003e Network",
"titleNoFormatting":"Earth Day Network",
"content":"Get information on \u003cb\u003eEarth
Day\u003c/b\u003e events, activities and actions, and learn how you
can join the \u003cb\u003eEarth Day\u003c/b\u003e Network
and Green Generation to make a difference
in \u003cb\u003e...\u003c/b\u003e"},
{"GsearchResultClass":"GwebSearch",
"unescapedUrl":"http://en.wikipedia.org/wiki/Earth_Day",
"url":"http://en.wikipedia.org/wiki/Earth_Day",
"visibleUrl":"en.wikipedia.org",
"cacheUrl":"http://www.google.com/search
?q\u003dcache:57bkwmGRNFkJ:en.wikipedia.org",
"title":"\u003cb\u003eEarth Day\u003c/b\u003e - Wikipedia,
the free encyclopedia","titleNoFormatting":"Earth
Day - Wikipedia, the free encyclopedia",
"content":"\u003cb\u003eEarth Day\u003c/b\u003e
is celebrated in the US on April 22
and is a day designed to inspire
awareness and appreciation for
the Earth\u0026#39;s environment.
\u003cb\u003e...\u003c/b\u003e"},
{"GsearchResultClass":"GwebSearch",
"unescapedUrl":"http://www.epa.gov/earthday/",
"url":"http://www.epa.gov/earthday/",
"visibleUrl":"www.epa.gov",
"cacheUrl":"http://www.google.com/search?
q\u003dcache:3IS9GTb-r0IJ:www.epa.gov",
"title":"\u003cb\u003eEarth Day\u003c/b\u003e | US EPA",
"titleNoFormatting":"Earth Day | US EPA",
"content":"Includes history of the celebration,
listing of EPA-sponsored events, and other resources."},
{"GsearchResultClass":"GwebSearch",
"unescapedUrl":"http://holidays.kaboose.com/earth-day/",
"url":"http://holidays.kaboose.com/earth-day/",
"visibleUrl":"holidays.kaboose.com",
"cacheUrl":"http://www.google.com/search?
q\u003dcache:v3MWPNh2yBkJ:holidays.kaboose.com",
"title":"\u003cb\u003eEarth Day\u003c/b\u003e 2009:
Crafts, Environmental Games, and
Recycling \u003cb\u003e...\u003c/b\u003e",
"titleNoFormatting":"Earth Day 2009:
Crafts, Environmental Games, and Recycling ...",
"content":"Jun 1, 2009 \u003cb\u003e...\u003c/b\u003e
\u003cb\u003eEarth Day\u003c/b\u003e is a special day to learn
about to take care of the planet.
Kids can learn with fun \u003cb\u003eEarth
Day\u003c/b\u003e activities, environmental projects,
\u003cb\u003e...\u003c/b\u003e"}],
"cursor":{"pages":[
{"start":"0","label":1},
{"start":"4","label":2},
{"start":"8","label":3},
{"start":"12","label":4},
{"start":"16","label":5},
{"start":"20","label":6},
{"start":"24","label":7},
{"start":"28","label":8}],
"estimatedResultCount":"40300000",
"currentPageIndex":0,
"moreResultsUrl":"http://www.google.com/search?
oe\u003dutf8\u0026ie\u003dutf8\u0026source\u003duds\u0026start\
u003d0\u0026hl\u003den\u0026q\u003dEarth+Day"}},
"responseDetails": null, "responseStatus": 200}
This article demonstrates how to make such a call and use the response in a .NET project.
One of Google's requirements in their terms of use is that attribution to Google for the search results is made. The web controls in this project include JavaScript that print the 'powered by Google' logo on the results page.
The focus here is on standard web search results, although the API can return images and other types of search results. Modification of the code is required if other search result types are required.
More about interaction with the Google Search REST API can be read here: http://googlesystem.blogspot.com/2008/04/google-search-rest-api.html.
Background
The REST API does not have to be used server side, the results can equally be parsed client side using JavaScript. We, however, had a need to use the Google Search API to generate and record information about the search results returned, so a server side solution was what we came up with. An added advantage is that we have been able to build reusable .NET web controls that can be placed in any web project, with next to no configuration.
One of the advantages of using the Google Search REST API is that no API key is required and there is no limitation on the number of queries, making the code and controls here easy to reuse in new applications.
One disadvantage of the REST API is that it will only return results in groups of 8 per query, and there is only access to the top 32 results for any search term. The code here partly solves that problem by creating multiple queries and grouping results where more than 8 results are required, so that issue will not have to be dealt with by the developer implementing search functionality.
Google Search is a powerful tool, and integration server side has many practical applications, so we decided to contribute our code to help other .NET developers who want to make use of the REST API. The code published along with this article uses the API to create a simple web site search box, which redirects the user to a results page, although the modular structure of the solution is designed to allow for use of the code in other applications.
Screenshots
Integration on a live site can be viewed at: Cognize.co.uk.
Points of Interest
The Google Search REST API returns a JSON response. One of the more interesting bits of code in this article is the deserialisation of the JSON response to .NET objects. This required the construction of objects that represent the JSON response from the API. The objects were constructed following testing and observing the response data. The code makes the call using a WebClient
. This is because the Google Search REST API requires a valid HTTP header, and this was the easiest way of achieving this.
The code then makes use of DataContractJsonSerializer
to deserialize the response data into .NET objects:
public static ResponseData GetSearchResultsChunk( string siteName,
string searchString, int resultCountStartPos )
{
ResponseData responseData = null;
try
{
searchString = HttpUtility.UrlEncode( searchString.Trim() );
using (WebClient client = new WebClient())
{
client.Headers.Add( "user-agent",
"Mozilla/4.0 (compatible; MSIE 6.0; " +
"Windows NT 5.2; .NET CLR 1.0.3705;)" );
string siteSearchString = String.Empty;
if (!String.IsNullOrEmpty( siteName ))
{
siteSearchString = "site:" + siteName + " ";
}
string resultSize = "large";
string searchRequestURL = "http://ajax.googleapis.com/" +
"ajax/services/search/web?v=1.0&start=" +
resultCountStartPos.ToString() + "&rsz=" +
resultSize + "&q=" + siteSearchString + searchString;
DataContractJsonSerializer jsonSerializer =
new DataContractJsonSerializer( typeof( GoogleAPISearchResults ) );
GoogleAPISearchResults searchResultsObj =
(GoogleAPISearchResults)jsonSerializer.ReadObject(
client.OpenRead( searchRequestURL ) );
responseData = searchResultsObj.responseData;
}
}
catch (Exception ex)
{
throw ex;
}
return responseData;
}
Objects representing the JSON response require the attribute [DataContract]
at the class level, and [DataMember]
for each property.
using System;
using System.Runtime.Serialization;
namespace Cognize.GoogleSearchAPIIntegration
{
[DataContract]
public class Pages
{
[DataMember]
public int start
{
get;
set;
}
[DataMember]
public int label
{
get;
set;
}
}
}
The management of multiple queries and grouping of results (due to restriction where the API will only return 8 results per call) is managed by the following code. If more than 8 results are required, additional calls are made using a different start index, so for example, if 10 results are requested, the second call will be made using 8 as the result start parameter (chunkResultCountStart
).
One odd bit of behaviour in the Google Rest API is that if only a small number of results are returned for a search term, say for example 1, then a second call to the API for the second chunk will return the same search result, even if the start parameter is set to 8 - for the 8th result. To get around this, on reading the first query result, the response data is read for the estimated result count (estimatedResultCount
). If this value is lower than the maximum search results requested, then maxChunksRequired
is modified within the for
loop (not great practice I know, but necessary) to prevent excess calls to the API and duplicate results.
public static SortedList<int, Results>
GetSearchResults( string siteName,
string searchString, int requestedResults )
{
SortedList<int, Results> searchResultsLst =
new SortedList<int, Results>();
if (requestedResults > 64)
{
requestedResults = 64;
}
int maxChunksRequired = CalculateChunksRequired( requestedResults );
for (int chunkIndx = 0; chunkIndx < maxChunksRequired; chunkIndx++)
{
int chunkResultCountStart = (chunkIndx * 8);
ResponseData responseData = GetSearchResultsChunk( siteName,
searchString, chunkResultCountStart );
if (responseData != null)
{
if (chunkIndx == 0)
{
int realChunksRequired = CalculateChunksRequired(
responseData.cursor.estimatedResultCount );
if (maxChunksRequired > realChunksRequired)
{
maxChunksRequired = realChunksRequired;
}
}
for (int resultIndx = 0; resultIndx <
responseData.results.Length; resultIndx++)
{
searchResultsLst.Add( searchResultsLst.Count,
responseData.results[resultIndx] );
}
}
}
return searchResultsLst;
}
Using the Code
To use the code, you can use the web controls included if you just want to plug it straight into your web app.
Should you want to use the code for other purposes, the key method to use is:
public static SortedList<int, Results>
GetSearchResults( string siteName,
string searchString, int requestedResults )
As it says 'on the tin', this will retrieve search results for a given search term up to the number of results requested (maximum in testing was 64).
An overload that does not restrict results to a particular site is available, that can be used in a similar way.
History
- 30th December 2009: First version published.