Augmented reality - XNA Models

• Download SlXnaGeoMotion.zip - 35.1 KB

Introduction

The app combines the motion API with Geocoordinatewatcher, Xna 3D models and Silverlights PhotoCamera class.

Place your model, calibrate your phone and walk around to admire your 3D work from all sides and angles...

None of these techniques are really new. They are just put together in such a way as to easily create a starting point for some interesting apps we, hopefully, soon will be served.

Background

Would it not be nice to walk around in some ancient ruins and be able to watch thru your phone how the site would have looked like at the height of it's glory? Or in this case, walk around some teapots floating quietly in the air, creating an eary feeling... but looking kind of displaced, and just out of context to be honest. As always the sky is the limit and the possibillities are endless...

Prerequsites

You will at least need some kind of Visual Studio 2010 with the Windows Phone Software Development Kit (SDK 7.1).

How to...

You start with creating a new project from template "Windows Phone Silverlight and XNA application". You can call your project SlXnaGeoMotion, SlXnaTeapots or whatever.

If you are not going to use your own 3D models of type *.fbx or *.x you can delete the two extra created "SlnaGeoMotionContent" and "SlXnaGeoMotionLib" projects the template creates. Be sure to remove the reference to the "SlXnaGeoMotionLib" in your remaining project.

Create a directory called Primitives3D and add excisting Items either from the attached SlXnaGeoMotion.zip file or from:

http://xbox.create.msdn.com/en-US/education/catalog/sample/primitives_3d

You will use these to later create your flying teapots or whatever 3Dimensional geometric xna model you are going for.

Now the only other files we will bother ourselves with in this small project is the GamPage.xaml and its "codebehind" GamePage.xaml.cs.

XAML

Let's start with the xaml:

<Grid x:Name="LayoutRoot" Background="Transparent">
    <Canvas x:Name="viewfinderCanvas" Height="800" Margin="-110,0,-110,0">
        <Canvas.Background>
            <VideoBrush x:Name="videoBrush">
                <VideoBrush.RelativeTransform>
                    <CompositeTransform x:Name="viewfinderTransform"
                            Rotation="90" CenterX="0.5" CenterY="0.5">
                    </CompositeTransform>
                </VideoBrush.RelativeTransform>
            </VideoBrush>
        </Canvas.Background>
    </Canvas>
    <TextBlock x:Name="Info"
               Text=""
               Style="{StaticResource PhoneTextLargeStyle}"
               Grid.Row="0">    
    </TextBlock>   
</Grid>

Using a VideBrush to display output from the Microsoft.Devices.PhotoCamera class in a container type of control is the usual and prefered way of displaying camera output from your Windows Phone 7.5 Mango device.

Thing to notice here is the 90 degree rotation so as to get your view in portrait mode. You should also notice that we use a Left and right Margin of -110 which will then give our Canvas (named viewfinderCanvas) a total width of 110 + 480 + 110 = 600. This gives us an aspect ratio of 600 / 800 or 3/4 which is what the physical camera in our phones actually are providing. We will thereby maintain the correct proportions on real world objects viewed.

And that's all we have to do regarding xaml stuff.

C# - Codebehind

I like to remove all comments explaining what is obvious and isolate (in regions) template generated statements. It is also very useful to group statements regarding different subjects within different "#region" tags every time we implement new functionality.

In this program the main subjects are:

- Silverlight

- PhotoCamera

- Motion

- 3D

- GeoCoordinateWatcher

I will try to explain what is going on here in the same sequence, and you will be able to reference the actual "#region"s in the source in the attached zip file.

Silverlight

In order to render Silverlight within Xna we must first define and instanciate a Microsoft.Xna.Framework.Graphics.UIElementRenderer.

UIElementRenderer silverlightRenderer;
silverlightRenderer = new UIElementRenderer(this, (int)ActualWidth, (int)ActualHeight);

If you then in the "Ondraw" section of your program renders and draws this object as a Xna Sprite, you should be able to display the real world as a background in your Xna world.

spriteBatch.Begin();
silverlightRenderer.Render();
spriteBatch.Draw(silverlightRenderer.Texture, Vector2.Zero, Color.White);
spriteBatch.End();

For the simple 3D models we use here, the next statements are hardly necessary, but you should get used to include them as a habbit.

SharedGraphicsDeviceManager.Current.GraphicsDevice.BlendState =
                                              BlendState.Opaque;
SharedGraphicsDeviceManager.Current.GraphicsDevice.DepthStencilState =
                                              DepthStencilState.Default;
SharedGraphicsDeviceManager.Current.GraphicsDevice.SamplerStates[0] =
                                              SamplerState.LinearWrap;

This will reconfigure your GraphicsDeviceManager to better handle more advanced "Skinned" 3D textured models often created with tools like Autocad, Maya or Blender.

PhotoCamera

Nothing out of the ordinary here:

PhotoCamera photoCamera;
photoCamera = new PhotoCamera();
videoBrush.SetSource(photoCamera);

The two last lines here is in the "#region Silverlight" in the "OnNavigatedTo" thingie, sorry about that, and in the last line our defined and instanciated "photoCamera" is set as source to the "viewfinderCanvas"s VideoBrush "videoBrush" in the xaml code in GamePage.xaml.

Motion

A "new" class in WP Mango basically handling the way you are holding your camera, view of direction, portrait, landscape etz. based on your phone's compass, accelerometer eventual gyro and such. Gives you information of rotation along x, y, z axcis in radians, as a quarternion or as used in this project as a matrix. Also gives info on acceleration which are not taken into consideration in this project.

Motion motion;
motion = new Motion();
motion.TimeBetweenUpdates = timer.UpdateInterval;
motion.Start();

Gives you access to this:

motion.CurrentValue.Attitude.RotationMatrix;

Just instanciate a motion from "Microsoft.Devices.Sensors.Motion", start it and you are ready to go by using "Attitude" from your Motion's "CurrentValue". Some people prefer to hook up the CurrentValueChanged event but I did not really see the point in doing so during this simple example.

3D

Initial:

TeapotPrimitive teapot;
Matrix teapotWorld;
Matrix projection;
Matrix view;
teapot = new TeapotPrimitive(
       SharedGraphicsDeviceManager.Current.GraphicsDevice);

Create teapot from your Primitives3D TeaPotPrimitive class. Be sure to state that you are "using Primitives3D" in your GamePage.xaml.cs file. For those of you who did not catch it. The use of a teapot as an example is ofcourse an intentional punch to all those java lovers out there. " src="http://www.codeproject.com/script/Forums/Images/smiley_smile.gif" />

Update or Draw section:

view = Matrix.CreateRotationX(MathHelper.PiOver2);
view *= motion.CurrentValue.Attitude.RotationMatrix;
view *= Matrix.CreateLookAt(Vector3.Zero, new Vector3(0.0f, 0.0f, -0.001f), Vector3.Up);
projection = Matrix.CreatePerspectiveFieldOfView(
  1, SharedGraphicsDeviceManager.Current.GraphicsDevice.Viewport.AspectRatio, 0.1f, 100f);

Firstly, you will want to orientate your xna world with real world view. This is done by rotating your Xna-world view 90 degrees around the x-axcis. Both real-world: y-axcis (Altitude) and Xna-world: y-axcis is now pointing towards the sky.

Here comes the Motion API in handy. The projection matrix creation should realy be moved to the OnNavigatedTo event so as to save execution time in your gameloop. (Does not need to be constantly updated.)

Teapot:

teapotWorld = Matrix.CreateScale(3.0f)
        * Matrix.CreateFromYawPitchRoll(
                 MathHelper.ToRadians(0), MathHelper.ToRadians(0), MathHelper.ToRadians(0))
        * Matrix.CreateTranslation(new Vector3(
                (float)PositionChangeX + 0.0f, 0.0f, (float)(PositionChangeZ - 5.0f)));
teapot.Draw(teapotWorld, view, projection, Color.Red);

Maybe a handful to swallov, but take it slow an easy and you should have no problems coping.

The Matrix teapotWorld is used to manipulate how this Model is positioned in your total Xna world.

You can scale, rotate and translate (position) your teapot as you wish. Remember to do this only in the scale, rotate, translate sequence. Rotating, translating and then scaling might give you unexpected results. Matrix multiplication is not like ordinary multiplication in this respect.

Never use the CreateScale Matrix with 0 as argument and expect to see anything in your view.

1 is to be used if you like to maintain the current Model size. (Or you can leave out this matrix all together in that case)

In the Matrix.CreateFromYawPitchRoll (decides which side of the teapot that are directed against you) I like to utilize the MathHelper.ToRadians since I always struggle to think in Radians. I find it much easier to think in the term of degrees of rotation.

The CreateTranslation part is using variables PositionChangeX and PositionChangeZ which I will come back to in the "GeoCoordinateWatcher" -> "Change In Position" section.

GeoCoordinateWatcher

Initial:

const double MeterInLatitude = 111290.91975341858;
const double MeterInLongitude = 55729.5173743959;
double PositionChangeX = 0;
double PositionChangeZ = 0;
GeoCoordinateWatcher geoWatcher;
GeoPositionStatus currentState = GeoPositionStatus.Initializing;
public GeoPosition<GeoCoordinate> InitialGeoCoordinate { get; set; }
public GeoPosition<GeoCoordinate> CurrentGeoCoordinate { get; set; }

We will come back to the "MeterInLatitude" and "MeterInLongitude" in the "Points Of Interest" section.

Instanciation:

geoWatcher = new GeoCoordinateWatcher(GeoPositionAccuracy.High);
geoWatcher.StatusChanged += new EventHandler<GeoPositionStatusChangedEventArgs>
(geoWatcher_StatusChanged);
geoWatcher.PositionChanged += new EventHandler<GeoPositionChangedEventArgs<GeoCoordinate>>
(geoWatcher_PositionChanged);
geoWatcher.MovementThreshold = 0.0;
geoWatcher.Start();

Two events are hooked up and the thingie is started...

Status:

void geoWatcher_StatusChanged(object sender, GeoPositionStatusChangedEventArgs e)
{
    currentState = e.Status;
    switch (e.Status)
    {
        case GeoPositionStatus.Disabled:
            if (geoWatcher.Permission == GeoPositionPermission.Denied)
            {
                string[] strings = { "ok" };
                MessageBox.Show("Please turn on geo-location service in the settings tab.");
            }
            else
                if (geoWatcher.Permission == GeoPositionPermission.Granted)
                {
                    string[] strings = { "ok" };
                    MessageBox.Show("Your device doesn't support geo-location service.");
                }
            break;
        case GeoPositionStatus.Ready:
            InitialGeoCoordinate = geoWatcher.Position;
            break;
    }
}

Keep track of where you are in the world when the adventure begin...

Change in position:

void geoWatcher_PositionChanged(object sender, GeoPositionChangedEventArgs<GeoCoordinate> e)
{
    if (currentState == GeoPositionStatus.Ready)
    {
        CurrentGeoCoordinate = e.Position;
        PositionChangeZ =
           (CurrentGeoCoordinate.Location.Latitude - InitialGeoCoordinate.Location.Latitude)
           * MeterInLatitude;
        PositionChangeX =
           (InitialGeoCoordinate.Location.Longitude - CurrentGeoCoordinate.Location.Longitude)
           * MeterInLongitude;
    }
}

Every time the PositionChanged event fires, we estimate how many meters you have moved in both the Longitude (X) and the Latitude (Z) position. The Altitude (Y) position is not taken into consideration here yet. Even though the "Location" has an "Altitude" attribute, your hardware will, as of current, hardly be able to utilize it.

Points of Interest

Constants:

GeoCoordinate GeoCoordinate1 = new  GeoCoordinate(59.949702262878418, 10.763088226318359);
double meterInLatitude = GeoCoordinate1.GetDistanceTo(
                         new  GeoCoordinate(60.949702262878418, 10.763088226318359));
// 1 Latitude degree = 111290.91975341858 meter
double meterInLongitude = GeoCoordinate1.GetDistanceTo(
                          new  GeoCoordinate(59.949702262878418, 11.763088226318359));
// 1 Longitude degree = 55729.5173743959 meter

I estimated the value of the constants "MeterInLongitude" and "MeterInLatitude" by using the "GeoCoordinate"s "GetDistanceTo" function. Just checking the distance between two GeoCoordintes with 1 Longitude in difference on the exact same Latitude and vice versa.

Animation:

It is not to difficult to have your teapots spin around by updating the Yaw (around y-axcis) part of the CreateFromYawPitchRoll in the estimation of the teapotWorld by using the GameTimerEventArgs Totaltime multiplied by some factor. In fact an interesting task might be to animate a model of the Japanese "bullet" train Shinkansen in real time as it moves from Tokyo to Nagasaki. (Just kidding).

Fixed position in the world:

If you have a model, like the Eiffel tower, which you always would want to be located in Paris, even though you yourself are situated in Uruguay?, you should calculate your own spesific "EiffelTowerPositionChangeX" and "EiffelTowerPositionChangeZ". This then based on the actual GeoCoordinates of the tower (in sted of using "InitialGeoCoordinate") and use these new variables in your EiffelTowerWorld. Of course, then again if you are french and lives in Paris you would not have your silly little model to obscure the real view of this "magnifique" landmark.

Word of warning:

When you fly around outdoors (the GeoCoordinateWatcher works best in the open) looking for flying teapots, please beware of the trafic around you. It's not all "augmented" you know.

Also running around, wavering your telephone in the air, trying to see things that are not really there, might have you're colleagues view you as kind of a nuttcase...

Comments on how to improve functionality is highly appreciated...

Good luck to you and have fun decorating the world with flying teapots...

History

Created August 2012
Attached picture and zip file download...