Lego Pan Tilt Camera and Objects Tracking

• Download demo project - 85.1 KB
• Download source - 94.5 KB

Introduction

If you work in the computer vision area, then you know for sure -a camera is nice, but a pan-tilt camera is much better! But, what if you don't have a special pan-tilt camera, but just a regular USB web camera? Well, that is not a big issue if you like building things on your own. Doing a quick search on the Internet, you may find different recipes of building pan-tilt cameras. Some of these recipes use quite simple electronics and stepper motors, but other recipes use different special pan-tilt equipment, which may be used for turning any camera into a pan-tilt camera.

In this article, we are going to discuss one more recipe of building a home-made pan tilt camera, which utilizes a regular USB web camera and the Lego Mindstorms NXT robotics kit. We've already discussed communication with Lego robotics kits using AForge.NET framework in another article, so now, we'll take that as a base and concentrate on building our pan tilt device and manipulating it.

After we build our pan tilt camera, we are going to make use of it - we'll try to locate some objects, and track them utilizing image processing routines from the above mentioned AForge.NET framework. This part will be quite generic, and may be applied to any camera with a pan-tilt device.

So, let's start building...

Changing the power supply of Lego Mindstorm

When you build a robot which moves around certain areas, the mobility of the robot is quite important, and any wire may break it easily. So, mobile robots should be wireless, carrying a battery with them and doing wireless communication with the host if required. However, pan tilt cameras don't need to move around, but they just need to stay in a place, turning their camera only for a certain degree. This means that instead of wireless mobility, we may better want constant power supply, which makes the device to work as long as required.

So, the first step we are going to do is to get rid of Lego Minstorm's batteries and power it with a regular power adapter. The NXT brick requires a 9 volts power supply, which was originally achieved utilizing 6 batteries - each battery provides 1.5 volts. What we need to do is to find a power adaptor which provides something similar. I did not have an exact 9 volts adapter, but it looks like 7.5 volts also works fine. So, taking some wires and soldering tools, we can make our NXT brick powered by the power adapter ...

Plugging the power adapter to the upgraded NXT brings it to life! We may use a simple NXT test application from the AForge.NET framework to check the connectivity and see the brick's power - it shows 7671 millivolts for our 7.5 volts adapter.

Building a pan tilt camera

Well, it is a bit hard to provide step by step instructions on how to build a pan and tilt modules from Lego NXT pieces. But actually, it is not required, since it may kill creativity and imagination. The pan module is quite easy - the simplest way is to use the one provided by the set rotating the platform piece. The tilt module may require a bit of work. I've tried several variants, and came to the decision that the block based one with a thread manipulated platform works best for me.

Joining both parts together and attaching a Logitech camera resulted in a quite interesting construction, which gave the camera two degrees of freedom. Yes, the device does not look small, but was built using generic parts from the Lego constructor. And actually, it is enough for our further experiments.

Camera view and its manipulation

Before we start with the camera manipulation, we need to get a view from it first. That is quite simple to do, and the VideoSourcePlayer control will do all the work for us, since it is responsible for video displaying. All we need to do is to put the control on our form, create the required video source provider, and give it to the control. In our case, we are going to use the VideoCaptureDevice class, which works with capture devices through the DirectShow interface:

VideoCaptureDevice videoSouce = new VideoCaptureDevice( deviceMonikerString );

videoSourcePlayer.VideoSource = videoSouce;
videoSourcePlayer.Start( );

Now, when we have the view, we need to manipulate it. To do this, we'll use two special controls which allow controlling the camera - one controls the pan device, and the second controls the tilt device. These are simple controls which have a manipulator in the center, and dragging this manipulator away from the center results in camera movement in one or another direction. The screenshot below may clarify the idea ...

We are not going to discuss the code of the manipulation control here (it may be found in the attachment to the article), but mention that it generates an event on the manipulator's position change, which provides its current position in the [-1, 1] range (0 means that the manipulator is in the center). So, when receiving the event, we just need to convert the manipulator's position to the motor's power and then turn it on (we are using the NXTBrick class from the AForge.NET framework, which was discussed previously).

// convert power to the |[55, 60]| range
int power = (int) ( 5 * panMotorPower + 55 * Math.Sign( panMotorPower ) );

NXTBrick.MotorState motorsState = new NXTBrick.MotorState( );
// check if we need to stop
if ( power == 0 )
{
    motorsState.Mode      = NXTBrick.MotorMode.None;
    motorsState.RunState  = NXTBrick.MotorRunState.Idle;
}
else
{
    motorsState.Mode       = NXTBrick.MotorMode.On;
    motorsState.RunState   = NXTBrick.MotorRunState.Running;
    motorsState.TachoLimit = 0;
    motorsState.Power      = power;
    motorsState.TurnRatio  = 80;
}

nxt.SetMotorState( NXTBrick.Motor.A, motorsState );

Note: The power conversion equation may look different - it all depends on how fast you want your motor moving.

So now, we have a pan tilt camera built from the Lego Mindstorms NXT Robotics kit and the Logitech camera, which can be manipulated manually with a regular mouse and two controls. The next step is to make it moving on its own ...

Object tracking

Pan tilt cameras are much more interesting than static cameras, because they open quite interesting possibilities like object tracking, for example. Since we've just built a pan tilt camera, let's try to accomplish the task of tracking some simple objects. For example, let's track a ball of a certain color.

Before we start tracking something, the first task is to detect an object and get its position on the image acquired from the camera. Since we are going to track simple objects of solid colors, the object detection may be done quite easily utilizing image processing routines provided by the AForge.NET framework. The first step to do is to perform color filtering, keeping only objects of the colors we are interested in, and removing all the rest, which may be achieved using the ColorFiltering class for example:

// create color filter
ColorFiltering colorFilter = new ColorFiltering( );
// configure the filter to keep red object only
colorFilter.Red   = new IntRange( 110, 255 );
colorFilter.Green = new IntRange(   0,  60 );
colorFilter.Blue  = new IntRange(   0,  60 );
// filter image
colorFilter.ApplyInPlace( image );

Now, when we have an image containing the interesting object only, we need to get its coordinates. This may be achieved by utilizing the BlobCounter class, which is able to locate stand alone objects. However, this class works only with grayscale and binary images, so we need to use the GrayscaleBT709 filter before, for example:

// create grayscale filter
GrayscaleBT709 grayscaleFilter = new GrayscaleBT709( );
// apply it to color filtered image
Bitmap grayImage = grayscaleFilter.Apply( image );

// create blob counter
BlobCounter blobCounter = new BlobCounter( );
// configure it to filter out small objects
blobCounter.MinWidth  = 25;
blobCounter.MinHeight = 25;
blobCounter.FilterBlobs = true;
// set ordering - bigger objects go first
blobCounter.ObjectsOrder = ObjectsOrder.Size;

// locate blobs
blobCounter.ProcessImage( grayImage );
Rectangle[] rects = blobCounter.GetObjectRectangles( );

// get the biggest object
if ( rects.Length > 0 )
{
    Rectangle objectRect = rects[0];

    ...
}

As we can see from the code above, we take the first object only, which is the largest, since we don't want to handle any small objects which may have a similar color or just noisy pixels.

Now, the last step is to get the coordinates of the object's center and move the camera towards it ...

int objectX = objectRect.X + objectRect.Width / 2 - image.Width / 2;
int objectY = image.Height / 2 - ( objectRect.Y + objectRect.Height / 2 );

Finally, let's see this in action ...

Conclusion

Well, finally, we've built a pan tilt camera, which is even able to track some objects. Not a superior camera, but something made from generic Lego pieces.

I've discovered some underwater stones with my Lego NXT motors, which don't allow performing precise movements, what makes it hard to implement a smooth movement of the camera. According to Lego documents a motor's power is in the [0, 100] range ([-100, 0] for backward movement). However, setting the power to 50 gives nothing except nasty noise, but setting it to 60 may result in too fast a movement, which is not acceptable for precise manipulations.

Regarding color filtering, which we've used to locate the object of our interest, there are different approaches which may give even better results. The ColorFiltering filter which we've applied is the simplest one, and performs color filtering in RGB color space, which gives the fastest performance on RGB images. However, filtering in HSL color space, for example, may give better results in terms of locating only colors we want and rejecting the rest (see the HSLFiltering filter).

Although the article is describing object tracking using a manually built pan tilt camera from a Lego NXT constructor's pieces, the tracking code may be easily applied to any other pan tilt camera. All you need is to detect an object, calculate its center's coordinates, and then move the camera towards it.