Introduction
The goal of this article is to simplify Windows Azure and event-driven architecture by walking through the process of building a .NET event driven system in the clouds. Let's start with some basic and boring definitions:
- Event-driven architecture: "A pattern promoting the production, detection, consumption of, and reaction to events." - Wikipedia
- Windows Azure: "A cloud services Operating System that serves as the development, service hosting, and service management environment for the Windows Azure platform. Windows Azure provides developers with on-demand compute and storage to host, scale, and manage web applications on the internet through Microsoft® datacenters." - Microsoft.
Now you know why the stick figure above is scratching its head. While those definitions are accurate, it is hard to abstract a concrete implementation of event-driven architecture (EDA) and understand the true value of Windows Azure without a real world example.
To demonstrate Azure and EDA beyond text book definitions, this article will show you how to build a flight delay management system for all US airports in clouds. The format of this article is intended to be similar to a hands on lab, walking you through each step required to develop and deploy an application to the clouds. This flight delay management system will be built with a mix of technologies including SQL 2008 Azure, Visual Studio 2010, Azure Queue Service, Azure Worker/Web Roles, Silverlight 3, Twitter SDK, and Microsoft Bing mapping. And, while the implementation of this article is based on a flight delay system, the concepts can be leveraged to build IT systems in the clouds across functional domains.
Flight Delay Manager: Event Driven Architecture (EDA)
Information on event driven architecture is well documented and widely available through web and print publications. For background into the event driven systems and the architectural pattern, check out this EDA article written by Gregor Hohpe, a software architect at Google.
This article will focus on what is not well documented, an example of how to implement EDA with emerging technologies such as Visual Studio 2010 and Windows Azure. In order to design and build the Flight Delay Manager, we do need to establish the EDA basics:
- EDA at a high level has four logical layers: event generation, event transportation, event processing, and downstream event activities.
- EDA is driven by the asynchronous flow of events, a direct contrast to the request/response pattern of typical SOA based systems.
- Humans are a good example of a perfect EDA system; we can generate, process, and react to millions of events on a daily basis.
For the Flight Delay Manager, the EDA model will be:
As you can see, each of the four logical layers of EDA are translated to Azure components that we will build to power our flight delay system. The design is based on simple event processing (SEP) that could be extended to a complex event processing (CEP) by plugging in additional event sources such as delay streams from airlines and weather feeds. CEP could be used to predict flight delays and cancellations before they happen based on real-time analysis of incoming event streams. While CEP is not covered in this article, a future version of the flight delay system that uses CEP is definitely possible.
Flight Delay Manager: Windows Azure Setup
To configure and build the solution in this article, you will need a Windows and SQL Azure account. Microsoft is currently offering a free trial of these services. Sign up for your account at: http://www.microsoft.com/windowsazure/pricing/.
Now that you have your Azure account, you will need to setup the component “Event Transport: Queue” noted in the diagram above. Log into the Azure developer portal at http://windows.azure.com/, and then create a new service configured as queue storage:
Continue following the Azure prompts such as queue name and other basic metadata; once complete, Azure will display security tokens and a public HTTP endpoint for the Queue. This Azure Queue will be used for event transport of raw flight delay messages between the message generator and the processor.
Flight delays are translated into data points by the event processor, and used by the downstream Silverlight client. These data points are stored in SQL 2008 Azure. To create a database, drill into the SQL portal and click Create Database, DB Name = FlightDelay:
It’s important to note when working with SQL Azure, Management Studio R2 is required to use an object explorer against an Azure database. And, don't forget to add your development machine IP addresses to the firewall list (second tab above) and check the box “Allow Microsoft Services access to this server”. That’s it! The Windows Azure backend setup is complete.
Flight Delay Manager: Development Machine Setup
In the next section, we will move on to application development. To develop in Windows Azure, you will need these components on your development machine:
After your development machine is Azure ready, you will need to setup the Flight Delay solution:
- Execute FlightDelayManager.sql on your SQL 2008 Azure DB created in the section above.
- Open FlightUpdates.sln in Visual Studio 2010 and build the solution.
- In the projects MessageGeneratorService/ MessageProcessorSerivce/ Web, change the configuration (app/web.config) of the “
FlightUpdates” connection string to the Azure database created above. Connecting strings can be copied directly from the Azure developers portal.
Twitter Message Generator and Message Processor
The message generator will be responsible for getting flight delays from Twitter (http://twitter.com/FltAdvisor) and pushing those messages on to an Azure Queue. To get started with the Message Generator, load Visual Studio 2010 and open FlightDelayManager.sln. Drill into the Twitter Message Generator solution folder (Azure project creation step also shown):
The blue icon denotes an Azure cloud project containing two basic components: Roles and Service Configuration. Roles map back to .NET projects containing code for execution in the clouds. Service configuration are application settings outside the cloud deployment packaging, allowing for dynamic configuration changes in the clouds.
The Message Generator requires access to an Azure Queue to transport Twitter flight delay messages. To do this, drill into the project “FlightUpdates.MessageGenerator”, open ServiceConfiguration.cscfg, and update this setting to your Windows Azure Queue (key available @ http://windows.azure.com/):
<Setting name="FlightUpdatesQueue"
value="DefaultEndpointsProtocol=http;AccountName=flightupdates;
QueueEndpoint=http://yourqueuename.queue.core.windows.net/;
AccountKey=fakekey...C++
EbamYR7Jv/9Dz9900p9g5FeTlvb9C0DUCeAcXKLch
5PEfQafrIqksuUVaGHAWZCCZqkEiw==" />
Next, open TwitterMonitorService.cs:
using Microsoft.WindowsAzure.Diagnostics;
using Microsoft.WindowsAzure.ServiceRuntime;
using FlightUpdates.Utility.Constants;
using FlightUpdates.TwitterMonitorService.Resource_Files;
namespace FlightUpdates.TwitterMonitorService
{
public class TwitterMonitorService : RoleEntryPoint
{
public override void Run()
{
try
{
Trace.Listeners.Add(new TraceWriterUtil());
Trace.WriteLine(Messages.ServiceStart);
FlightDelayMessageGenerator flightMessageGen =
new FlightDelayMessageGenerator();
while (true)
{
Trace.WriteLine(Messages.CheckForDelayMessages);
flightMessageGen.RetrieveFlightDelayMessages_PutInQueue();
…}}}
The implementation of RoleEntryPoint defines this class as the Azure Worker Role which will poll the Twitter Feed http://twitter.com/FltAdvisor for flight delays. Let’s take a look at how to get Twitter messages in .NET:
using RareEdge.Twitteroo;
core = new TwitterooCore(TwitterSettings.userName,
TwitterSettings.password);
core.PublicTimelineUrl =TwitterSettings.timeline;
users = core.GetTimeline(Timeline.Public);
Trace.WriteLine(Messages.TwitterConnectSuccess);
foreach (User message in users)
{
int postAgeMin = (int)DateTime.Now.Subtract
(message.Status.CreatedAt).TotalMinutes;
if ((postAgeMin < TwitterSettings.runInterMin) || debugMode)
{
string flightDelayMessage = message.Status.Text.ToLower().Trim();
InsertMessageIntoAzureQueue(flightDelayMessage);
}
}
For this piece, you will need to create a Twitter account and add your credentials to the class FlightUpdates.Utility.Constants.TwitterSettings. The code above is pretty simple, thanks to TwitterooCore, a free Twitter .NET SDK available at http://rareedge.com/twitteroo/. In about 10 lines of code, we can get all the current Tweets posted to FltAdvisor. Now that we have the messages, let’s insert the Tweets into an Azure Queue:
using Microsoft.WindowsAzure.StorageClient;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.ServiceRuntime;
var storageAccount = CloudStorageUtil.GetFlightUpdatesQueue();
CloudQueueClient queueStorage =
storageAccount.CreateCloudQueueClient();
CloudQueue queue = queueStorage.GetQueueReference
(CloudQueues_QueueReferenceNames.FlightDelayMessages);
queue.Create();
queue.AddMessage(new CloudQueueMessage(flightDelayMessage));
CloudQueueClient references your Azure queue instance, CloudQueue is the specific Queue within that instance, and CloudQueueMessage is the outgoing message. The method CloudStorageUtil.GetFlightUpdatesQueue() is the standard boiler plate code to reference the Azure Queue we configured earlier in ServiceConfiguration.cscfg.
Now that we are code complete for the Twitter Message Generator, set the project “FlightUpdates.MessageGenerator” as the start up project and run the application. This will run the application within your local Azure development fabric which will display the trace information for each message being added to the queue:
Now, onto the message processor, the component responsible for flight delay message identification and transformation. The Message Processor requires access to the Azure Queue configured as the event transport for Twitter flight delay messages. To do this, drill into the project “FlightUpdates.MessageProcessor”, open ServiceConfiguration.cscfg, and update this setting to your Windows Azure Queue:
<Setting name="FlightUpdatesQueue"
value="DefaultEndpointsProtocol=http;AccountName=flightupdates;
QueueEndpoint=http://yourqueuename.queue.core.windows.net/;
AccountKey=fakekey...C++
EbamYR7Jv/9Dz9900p9g5FeTlvb9C0DUCeAcXKLch5PEfQafrIqksuUVaGHAWZCCZqkEiw==" />
With the queue configured, the application can now read messages from the queue via this block of code:
using Microsoft.WindowsAzure.StorageClient;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.ServiceRuntime;
CloudQueueClient queueStorage = storageAccount.CreateCloudQueueClient();
CloudQueue queue = queueStorage.GetQueueReference
(CloudQueues_QueueReferenceNames.FlightDelayMessages);
CloudQueueMessage message = queue.GetMessage();
Trace.WriteLine(Messages.ConnectQueueDone);
while (message != null)
{
AddFlightDelayToDB(message.AsString);
queue.DeleteMessage(message);
message = queue.GetMessage();
}
After the queue messages are retrieved, the application can now filter out bad messages and transform good messages into usable flight delay data:
Now, the message transformation is complete, and the Flight Dashboard can communicate delay information to downstream users. In the next section, we will create the Flight Delay dashboard that detects these data updates from the message/event processor via a pooling timer.
It is worth mentioning, this is a simplified implementation of an event processor. Typically, this component would notify downstream clients of the data change in near real time. When Microsoft re-releases Azure Workflow (pending .NET 4.0), Flight Delay Manager R2 will be refactored to use .NET Workflow and send real-time messages to downstream clients.
Silverlight 3 Flight Dashboard with Microsoft Bing Mapping
This dashboard is actually fairly simple. It is made possible thanks to Microsoft’s Silverlight Bing Map control. Before doing anything, you should download the control and SDK. At your leisure, you should peruse the SDK. It contains some pretty useful information to get you started. In addition, you will also need a developer’s license/key. This key is not optional, and will be required to access the Bing Map control and its functionalities.
Now, before we continue on, I would like to just briefly mention the data access portion of the dashboard. The method is fairly trivial, so I won't go into too much detail. However, it does warrant a brief mention. The dashboard, in this case, has a service reference to a relatively simple web-hosted WCF service. The service itself is rather unspectacular, and only has two methods, to retrieve airports and to retrieve the delay feed. The only thing to note is, to avoid creating a cross-domain policy, the service is hosted in the same web which hosts the Silverlight dashboard application.
Now, off to the good stuff. Prior to using the map control, you will have to reference the assemblies installed by the SDK download. The assemblies should be located within the location where you installed the SDK. Once you have the assemblies referenced, you can add the control to your XAML (or through code) much like you would any other Silverlight control:
<Border Margin="0" Grid.Column="1"
Grid.Row="1" BorderBrush="White"
BorderThickness="2">
<m:Map HorizontalAlignment="Stretch"
VerticalAlignment="Stretch"
x:Name="BingMap" Margin="0,0,-4,-4">
</m:Map>
</Border>
Now, remember that the key we had generated earlier, that will have to be assigned to the CredentialsProvider attribute on the control:
<Border Margin="0" Grid.Column="1"
Grid.Row="1" BorderBrush="White"
BorderThickness="2">
<m:Map HorizontalAlignment="Stretch"
VerticalAlignment="Stretch"
CredentialsProvider="Ap0NlORXtMcQuszz1GvM_wiZ8Y3Pq-
YGnHQN9SqV9qhkTnrVFEJ1q9JKbkQcFYXA"
x:Name="BingMap"
Margin="0,0,-4,-4">
</m:Map>
</Border>
On map load, you most likely will want to center your map on a particular region. There are a few ways to do this. You can either supply the center point of the map along with a zoom level, or specify a bounding rectangle and center point. All points are measured in longitude and latitudes, and a quick Google search can typically help you find the longitude and latitude of any area you are looking for. In this project, we utilized a bounding rectangle and a center point that encompass the continental United States:
Location center = new Location(39.3683,-95.2734,0.0000);
LocationRect boundingRec = new LocationRect(center,6, 6);
BingMap.SetView(boundingRec);
Now that the map is setup, we would want to display some meaningful information on it. Since we are displaying airport delays, it would be meaningful to display the locations of each of the airports. In the project, it was fairly simple to add some pushpins of the location of airports. All we need is the latitude and longitude of the airport. In our case, this information is returned to us from the WCF Service as a list:
foreach (AirportInfo item in e.Result)
{
Pushpin pin = new Pushpin();
pin.Location = new Location();
pin.Location.Latitude = item.LatitudeDegree;
pin.Location.Longitude = -item.LongitudeDegree;
…
if (item.DelayTime <=30)
pin.Background = new SolidColorBrush(Colors.Green);
else if (item.DelayTime > 30 && item.DelayTime <= 60)
pin.Background = new SolidColorBrush(Colors.Yellow);
else
pin.Background = new SolidColorBrush(Colors.Red);
BingMap.Children.Add(pin);
}
The pushpin also has some additional attributes. For our purposes, we wanted to color code the pins to the size of the delays at the airport. Green pins would represent relatively small delays, yellow moderate delays, and red extensive delays. The one peculiarity is that the built-in pushpin feature does not have rollover text. The best practice would be to create a custom control containing the pushpin as the base. However, for a quick workaround, add a text block control to the MapLayer. The MapLayer allows for the positioning of its child element using geographical coordinates:
<m:Map HorizontalAlignment="Stretch" VerticalAlignment="Stretch"
CredentialsProvider="Ap0NlORXtMcQuszz1GvM_wiZ8Y3Pq-
YGnHQN9SqV9qhkTnrVFEJ1q9JKbkQcFYXA"
x:Name="BingMap" Margin="0,0,-4,-4">
<m:MapLayer x:Name="ml_MapText" Margin="0">
<Grid x:Name="grd_Rollover"
HorizontalAlignment="Right"
Margin="0"
VerticalAlignment="Bottom"
Visibility="Collapsed"
Canvas.ZIndex="1000">
<Border Margin="0" Background="Black"
BorderThickness="1"
CornerRadius="5" Opacity="0.785"/>
<Border Margin="0" BorderBrush="White"
BorderThickness="2" CornerRadius="5">
<TextBlock x:Name="tbk_AirportNameLabel"
Text="TextBlock" TextWrapping="Wrap"
HorizontalAlignment="Center"
VerticalAlignment="Center"
Foreground="White" Padding="10"/>
</Border>
</Grid>
</m:MapLayer>
</m:Map>
To complete this rollover text, we will add some events to each of the pushpins that will make the text visible and display the proper text. In addition, we will store the airport item as part of the pins data context:
foreach (AirportInfo item in e.Result)
{
Pushpin pin = new Pushpin();
pin.Location = new Location();
pin.Location.Latitude = item.LatitudeDegree;
pin.Location.Longitude = -item.LongitudeDegree;
pin.MouseEnter += new MouseEventHandler(pin_MouseEnter);
pin.MouseLeave += new MouseEventHandler(pin_MouseLeave);
pin.MouseLeftButtonUp += new MouseButtonEventHandler(pin_MouseLeftButtonUp);
if (item.DelayTime <=30)
pin.Background = new SolidColorBrush(Colors.Green);
else if (item.DelayTime > 30 && item.DelayTime <= 60)
pin.Background = new SolidColorBrush(Colors.Yellow);
else
pin.Background = new SolidColorBrush(Colors.Red);
pin.DataContext = item;
BingMap.Children.Add(pin);
}
The MouseEnter event will make the TextBlock visible, set the current position of the TextBlock and extract the current information from the data context and populate the TextBlock with it:
void pin_MouseEnter(object sender, MouseEventArgs e)
{
Pushpin selectedPin = sender as Pushpin;
AirportInfo selectedPinInfo = selectedPin.DataContext as AirportInfo;
Location loc = new Location(selectedPinInfo.LatitudeDegree,
-selectedPinInfo.LongitudeDegree);
tbk_AirportNameLabel.Text = selectedPinInfo.AirportName +
" (" + selectedPinInfo.AirportCode + ") ";
MapLayer.SetPosition(grd_Rollover, loc);
grd_Rollover.Visibility = System.Windows.Visibility.Visible;
}
The code to note above is MapLayer.SetPosition(grd_Rollover, loc);. This little beauty will place positioning coordinates for most, if not all, of your Silverlight elements. This attribute can also be set within XAML. The MouseLeave event simply makes the TextBlock invisible:
void pin_MouseLeave(object sender, MouseEventArgs e)
{
grd_Rollover.Visibility = System.Windows.Visibility.Collapsed;
}
Now, let’s get it to do something a bit more meaningful. The last event, the MouseLeftButtonUp (oddly enough, there wasn't a Click event at the time of this writing), will go back to the service to retrieve the flight delays at the airport that the user selected. Much like the rollover text, the airport information is retrieved from the pin’s data context, and that information is sent back to the service.
void pin_MouseLeftButtonUp(object sender, MouseButtonEventArgs e)
{
VisualStateManager.GoToState(this, "FlightDelayHidden", false);
Pushpin selectedPin = sender as Pushpin;
AirportInfo selectedPinInfo = selectedPin.DataContext as AirportInfo;
AirportCodeServiceClient airportService = new AirportCodeServiceClient();
airportService.GetFlightDelayFeedCompleted+=
new EventHandler<GetFlightDelayFeedCompletedEventArgs>
(airportService_GetFlightDelayFeedCompleted);
airportService.GetFlightDelayFeedAsync(selectedPinInfo.AirportCode);
tbk_Airport.Text = selectedPinInfo.AirportName +
" (" + selectedPinInfo.AirportCode + ") ";
}
The resulting event handler will then populate a DataGrid residing beside the map. The end result will hopefully help you manage your travels better….
Flight Delay Manager: Windows Azure Deployment
The FlightUpdates solution contains a Windows Azure project per component, allowing for the deployment of the Flight Dashboard, Message Processor, and Message Generator as individual Azure services. In addition, the solution includes a master Azure project that bundles all three components (Azure Roles) into a single Azure service. From my initial experiences with Azure, I would recommend an Azure project per component approach. This makes deployments easier to test and troubleshoot, with the added benefit of faster Azure service start up.
Let’s use the Azure project per component approach and deploy the Flight Delay Manager dashboard to the clouds. Log into the Azure developer portal at http://windows.azure.com/ and create a new service with the name FlightManager_code:
Now that the service is created, go to Visual Studio, open the Silverlight configuration file “ServiceReferences.ClientConfig”, and change the endpoint to the URL created above (http://myappname.cloudapp.net).
Next, highlight FlightUpdates.AzureWeb and click Publish from the Build menu. Then, in the Windows Azure portal, upload the service package and service configuration:
That’s it; you now have an application running in the clouds. Click Start to enable your deployment, and then click the Azure link to run the flight delay manager. You should see a map showing all US flight delays posted to Twitter, with the ability to drill into a city for flight details.
Flight Delay Manager R2
Flight Delay Manager R1 is an example of a simplified Event Driven Architecture (EDA) implementation. Flight Delay Manager R2 will extend the code base and fundamental Azure skills established in the article. Flight Delay Manager R2 will shift to a CEP (complex event processing) based implementation. The current ideas on the table are (feel free to post what you would like to see):
- Event transport refactor to the AppFabric Service Bus with multiple real-time flight delay event sources.
- Event processor refactor to Azure Workflow, handling multiple downstream activities such as mobile flight delay alerts.
- Silverlight delay dashboard to receive message notifications when new flight delays are available.
- iPhone Objective-C application to allow airline passengers to enter and share flight delays based on their travel experiences. Application would also support the same functionality as the Silverlight delay dashboard via Google Earth. (…this one is already in development).
- Cloud to cloud through the integration of a Google App Engine application as a possible event source or downstream application.
Flight Delay Manager R2 will be posted on CodeProject and linked to this article when complete.
Wrapping Up
With the flight delay manager complete, you have now developed and deployed an event driven system to the clouds using SQL 2008 Azure, Visual Studio 2010, Azure Queue Service, Azure Worker/Web Roles, Silverlight 3, Twitter SDK, and Microsoft Bing mapping. You have dynamically provisioned hardware such as web, database, and queue servers in the clouds. And, you have executed production deployments in a single click. Hopefully, these skills provide you with solid foundation in Windows Azure, allowing you to start planning and creating your future cloud based IT systems.
History
- 29 January, 2010: Initial version.
- 5 February, 2010: Update article text.
