Develop a Windows 8 app in 30 days
Music videos are evolving. What used to be the paradigm of music videos on channels like MTV and VH1, are now being eclipsed by the sheer volume of music videos being released and featured on the web. Unlike traditional media – such as television – where one music video is played after another, music videos on the web compete with one another for attention. So how do artists cut through the noise and get noticed, especially if you’re an up and coming star like Jasmine Villegas?
This is the challenge Jasmine and the team at Internet Explorer sought to solve. Her vision was to create an interactive experience to accomplish two things, 1) engage her existing fan base, and 2) help her reach new fans. To get this done right, Microsoft called in HTML5 experts at creative agencies Digital Kitchen and Bradley and Montgomery to create justafriend.ie The result is an experience that not only meets Jasmine’s objectives, but also showcases the power of HTML5 on the web.
Layered throughout the entire music video are crazy amounts of technical sparklers working to bring it life. Whether you’re Jasmine Villegas’ BFF or just want to learn more about HTML5, we want to provide you with more info on how we brought Just A Friend to life online.
Setting-up the site with HTML5 standards
We started the project by making sure modern browsers will recognize and use HTML5 video instead of plug-ins. This was easy:
<!DOCTYPE html>
Connect Facebook data to your web app
Facebook Connect is an integral component of JustaFriend.ie as it really helps to draw the user into the experience. When you first start the video, users are asked to connect to their Facebook account. This allows us to pull images from their FB account and project them directly into the video and also gives us the ability to display their name contextually throughout the experience. As a result, fans of Jasmine can feel part of the video, alongside their favorite singer.
Project images on an HTML5 canvas and video
Image projection in HTML5 is not a new concept. In fact, much of the early work on the project was based on what Steven Wittens provided at http://acko.net/files/projective-canvas/index.html. But canvas projection in IE9 performed much better in tests than other modern browsers, so the code had to be altered to work cross-platform.
A good example are the steps taken to get the scoreboard to display the user’s name pulled from Facebook Connect. To achieve this, the projection code was updated to work with a canvas buffer instead of an image. This allowed various elements to be composited to the buffer before it was projected. Starting with a blank PNG of the scoreboard, the names were drawn using the canvas drawing API. Another PNG was laid over the top to provide a slight glare. The whole piece was taken a step forward by having this projection matched to frames of video instead of just a motionless, flat surface.
Layering of PNGs was also used in projecting the user’s Facebook photos onto Jasmine’s bedroom wall at the beginning of the video. PNGs provided shadow effects to make the photos blend seamlessly with the existing photos on the wall. This technique was again used on the photo on Jasmine’s desk at the end of the video. Here’s the final experience and the code that powered it.
Before:
After:
function scoreboardProjection(frame_offset) {
if (frame_offset == null) frame_offset = 0;
var canvas, proj;
var score_img_1,
score_img_2,
overlay,
board_canvas,
score_canvas_1,
score_canvas_2;
var self = view.extend({
template:'<div class="scene HD" id="scoreboard_projection"></div>&',
name:'scoreboardProjection',
pageName: 'Bowling Scoreboard',
onInit:onInit
});
return self;
function onInit() {
score_img_1 = media.getImage('img/bowlingscore1.png');
score_img_2 = media.getImage('img/bowlingscore2.png');
overlay = media.getImage('img/bowlingscore_overlay.png');
var scale_down_to_blur = .75;
board_canvas = canvasUtil.create(Math.floor(800 * scale_down_to_blur), Math.floor(450 * scale_down_to_blur));
score_canvas_1 = canvasUtil.create(800, 450);
score_canvas_2 = canvasUtil.create(800, 450);
canvas = canvasUtil.create(10, 10);
proj = projection(board_canvas, canvas, null, {
wireframe:false,
subdivisionLimit:3,
patchSize:32
});
}
}
view raw scoreboardProjection.example.js This Gist brought to you by GitHub.
Using this code source with the techniques described above will produce image projections that blend seamlessly into your project.
Source code for the Scoreboard scene is here: http://www.justafriend.ie/cdn/js/jv/scenes/scoreboardProjection.js
To learn more about developing with HTML5 canvas, try these links:
Pull video frames with pixels
HTML5 video is still an evolving technology. Although a lot of work has been done to ensure frame accuracy, it is still not perfect. So we had to hack it together with some help:
The HTML5 MediaElement API provides a getter for ‘currentTime’ that is supposedly accurate to a fraction of a second, but this does not always match the current frame being rendered to the screen. This is especially a problem when doing computationally expensive processing while the video is being rendered.
To get around this issue, the frame number was pulled out of the image using the black and white pixels that have been burned into the video and placed just off screen. This is not the standard NTSC scan display you see in videos. Instead, it is an original bit of binary data created and burned into the video manually to keep everything correctly synced. While seemingly simplistic in nature it is an incredibly useful technique that is used during the interactive moments in the video. Without it, the videos would not be properly synced which would cause the overlays to not match their corresponding frames.
function getCurrentFrameFromTimecode() {
frame_canvas_ctx.drawImage(video, 961, 0, 1, 16, 0, 0, 1, 16);
var timeBitmap = frame_canvas_ctx.getImageData(0, 0, 1, 16);
var timeData = timeBitmap.data;
var frame = 0;
var value;
for (var i = timeData.length - 1; i >= 0; i -= 4) {
value = (timeData[i - 3] + timeData[i - 2]
+ timeData[i - 1]) > 125 ? 1 : 0;
if (Math.floor(i / 4) == 15) {
frame = value << Math.floor(i / 4);
} else {
frame |= value << Math.floor(i / 4);
}
}
current_frame_text = "Frame: "+frame;
return frame;
}
view raw currentFrameFromTimecode.example.js This Gist brought to you by GitHub.
Use this code source to help keep video and interactive elements in sync. You can get the source for the "frame code reader"module here:
http://www.justafriend.ie/cdn/js/jv/frameCodeReader.js
Blowing up PNGs for the good of performance
During the dance game, the graphic elements needed to match the beat pattern and explode once they hit the paddle for a cool effect. For the explosions, the PNG sprites were converted to canvas image data, and then a particle system was created where each particle represented a single pixel in the image. The particle system is pretty basic and was developed using the 2D Vector class found in the Javascript port of toxiclibs.
Get the code sample here: http://haptic-data.com/toxiclibsjs/
In early testing, performance was an issue when too many particles were rendered at once. The simplest solution was to create sprites with lots of transparency within Photoshop. Then, when the particle system is created, only particles are created for those pixels that exceed a certain opacity threshold. By fading out and destroying the particles quickly, better performance was obtained.
An early prototype can be seen here: http://www.justafriend.ie/cdn/dev/proto13b.html
Sequence and media loading
Throughout the experience, there are different outcomes based on user interaction. Because of these different outcomes, a lot of different videos had to be created. So, the media loader needed to work in a way that made the transitions look seamless.
Initially Popcorn.js and the Popcorn.sequence module that was used in the other IE9 project for Cut The Rope was used. However, it was quickly discovered that this framework was overkill for this project and did not give the precision needed for sequencing. For starters, the sequence breaks when using non-integer in and out points, and the timed callbacks are not really in sync.
In the end, a system of scenes that had frame-specific in and out points were created. On each frame, all scenes are rendered that should be visible on that frame. Each scene can be thought of as an overlay, as the HTML5 video is rendered in the background.
Each scene uses the custom media loader to preload the required assets at startup. There are various media loaders for HTML5, but one couldn’t be found that met the project’s specific needs. So, a custom media loader was created that works with RequireJS and supports images, video, and audio. The loader includes support for onComplete and onError callbacks as well as application-level status notifications. All requests are queued and limited to a certain number of simultaneous “threads” to avoid many of the most common HTTP pipeline issues in some browsers, fortunately, not an issue for modern browsers like Internet Explorer 9.
One issue was that HTML5 video clips would not fully preload due to the way modern browsers naturally load segments of video in a progressive manner. To ensure that videos are cached, they are loaded via Ajax-style XHR. The main video is the only one that is allowed to stream in the traditional sense. The main video is the one element that doesn’t change depending on the user’s interactions. If it was preloaded, the initial load time at the start of the experience would be incredibly long. Using this combination of preloaded and progressively loading video elements, traditional buffer times are eliminated.
Adapt this code source to load your assets so that they transition seamlessly.
Source code for the media loader is here: http://www.justafriend.ie/cdn/js/jv/media.js
Dial Your Digits
One of the hidden elements within the experience is the use of the Tropo API that allows you to type in your phone number on a phone lying on the table at the end of the video. By doing so, Jasmine calls the number you dialed and leaves one of 6 random voice messages. The API is easy to implement, and adds a nice layer of surprise and delight to the individual going through the experience.
Wrapping it up
Thanks for reading the Behind the Scenes developer tear down on Jasmine Villegas’ Just A Friend.
To learn more about developing cross-browser code for modern browsers, start with MSDN: http://msdn.microsoft.com/ie
HTML5 Video Resources
