Realtime Twitter Stream Visualization with .NET Core, Emitter and JavaScript

• Download Source Code - 2.6 MB
• Source Code on GitHub
• View Live Demo

Introduction

This article presents a visualization of a Twitter sample stream with a C#/.NET Core publisher application which captures the stream from Twitter, processes and re-publishes text and avatars through emitter messaging broker (also on GitHub), which allows broadcasting the stream to subscriber clients, written in Javascript.

[View Live Demo]

Background

The architecture of the application is very simple, yet powerful. We have a publisher application which consumes tweets from Twitter API and publishes it in Emitter Broker. The broker, which handles multi- and broad-casting of the messages, is used to transfer the processed tweets from the publisher to the HTML5 consumers.

Tweet Publisher (C#/.NET Core)

1. Establishes connections with Twitter Streaming API and Emitter Broker.
2. Every time a new tweet is received, processes the tweet and publishes it into tweet-stream channel.

Javascript Client

1. Subscribes to tweet-stream channel.
2. Every time it receives a tweet through tweet-stream channel, processes it and binds it to the view.

Tweet Publisher - C#/.NET Core App

Below is most of the code I had to write for the application that consumes Twitter Stream, processes it and republishes into Emitter.

1. First, it uses a very nice, and .NET Core compatible Twitter API: https://github.com/linvi/tweetinvi to hook up the Stream and a TweetReceived event which will be invoke every time the message is received from Twitter.
2. It establishes the connection with Emitter and performs Twitter Authentication. If you want to try out for yourself, make sure to get the 4 security keys required to authenticate your app, which can be found here https://apps.twitter.com.
3. We use a rate limiter, allowing us to have a steady and visually pleasing rate at which we show tweets to the user.
4. For each tweet received, we skip if it’s a retweet and simply re-publish the Text and ProfileImageUrl400x400 in JSON format into tweet-stream channel of Emitter.

public class Program
{
    public static void Main(string[] args)
    {
        // Connect to emitter
        var emitter = Connection.Establish();

        // Set up your credentials (https://apps.twitter.com)
        Auth.SetUserCredentials(
            Environment.GetEnvironmentVariable("CONSUMER_KEY"),
            Environment.GetEnvironmentVariable("CONSUMER_SECRET"),
            Environment.GetEnvironmentVariable("ACCESS_TOKEN"),
            Environment.GetEnvironmentVariable("ACCESS_TOKEN_SECRET")
            );

        // Setup a rate limiter, only re-publish one tweet per 350 milliseconds
        var limiter = new Throttle(1, TimeSpan.FromMilliseconds(350));

        // Using the sample stream
        var stream = Stream.CreateSampleStream();
        stream.AddTweetLanguageFilter(LanguageFilter.English);
        stream.FilterLevel = Tweetinvi.Streaming.Parameters.StreamFilterLevel.Low;
        stream.TweetReceived += (sender, t) =>
        {
            // Skip retweets
            if (t.Tweet.IsRetweet)
                return;

            // Limiter allows us to have a steady, visually pleasing rate of tweets
            var ct = new CancellationToken();
            limiter.Enqueue(() =>
            {
                // Publish the tweet to the broker
                emitter.Publish(
                    "IsQ7z18uGEFpjOJpt4K6ij49klT3PGzu",
                    "tweet-stream",
                    JsonConvert.SerializeObject(new
                    {
                        avatar = t.Tweet.CreatedBy.ProfileImageUrl400x400,
                        text = t.Tweet.Text
                    }));
            }, ct);
        };

        // Start
        stream.StartStream();
    }
}

Tweet Publisher - Deploying with Docker

We are going to deploy our little .NET Core publisher using Docker. Docker containers wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries – anything that can be installed on a server. This guarantees that the software will always run the same, regardless of its environment.

One of the easiest ways to get started is to use a Dockerfile along with Docker Hub Build Server which would build an image we could deploy on the target machine. Docker Hub allows us to automatically build a Docker Image (think of a Virtual Machine Image here) which we could deploy. The image can be triggered to automatically re-build on every commit to the git repository.

Our docker file, as shown below, does several things and essentially represents a script which is used to create the Docker Image:

1. It starts with the Docker image provided for .NET Core by Microsoft.
2. Updates the apt-get sources and clones the Git repository.
3. Performs dotnet restore which essentially does a nuget restore and loads all the dependencies.
4. Performs dotnet publish which creates final binaries for our image.
5. Hooks a deploy.sh script, which will be executed every the image starts. This script simply runs our application.

FROM microsoft/dotnet:1.0.0-preview2-sdk
MAINTAINER Roman Atachiants "roman@misakai.com"

# Make sure we have S3 & additional libraries
RUN apt-get update -qq \
    && apt-get install -y git \
    && mkdir /tmp/emitter \
    && cd /tmp/emitter \
    && git clone "https://github.com/kelindar/twitter-stream.git" "/tmp/emitter" \
    && cd /tmp/emitter \
    && dotnet restore -v Minimal \
    && cd /tmp/emitter/src/Server \
    && dotnet publish -c Release --output /app \
    && rm -rf /tmp/emitter

# Application will be in app folder
WORKDIR /app
ADD deploy.sh /
CMD ["/bin/bash", "/deploy.sh"]

Once the image is built, we can start it by typing docker run -it kelindar/twitter-stream, assuming you have docker installed. It will automatically download the image from Docker Hub and run it for us interactively.

Javascript Client

Similarly to the C# app we just wrote, the client is extremely simple and straightforward. In fact, all of the javascript (excluding dependencies) is shown below. The client does several things:

1. Connects to emitter publish/subscribe service.
2. Sets up our ViewModel using VueJS.
3. Once we’re connected to emitter, it subscribes to tweet-stream channel.
4. Every time a message is received, we push it into the data property of our ViewModel, which is bound to the HTML DOM.

// Connect to emitter broker
var emitter = emitter.connect({
    secure: true
}); 

var key = 'jtdQO-hb5jfujowvIKvSF41NeQOE8IoF';
var vue = new Vue({
    el: '#app',
    data: {
        messages: []
    }
});

emitter.on('connect', function(){
    // once we're connected, subscribe to the 'tweet-stream' channel
    console.log('emitter: connected');
    emitter.subscribe({
        key: key,
        channel: "tweet-stream"
    });
})

// on every message, print it out
emitter.on('message', function(msg){
    // If we have already few messages, remove the oldest one (first)
    if (vue.$data.messages.length >= 8){
        vue.$data.messages.shift();
    }

    // Push the message we've received
    vue.$data.messages.push(msg.asObject());
});

Most of the UI here comes from one of my previous articles where emitter was used to create a simple group chat. Feel free to check the live demo and the corresponding CodeProject Article if you’re interested.

History

• 12/04/2016 - Initial Version of the article