How to use video and sound information to detect intruder

• Download source - 699.3 KB

Introduction

This article shows you how to use video and sound information to detect an intruder. Image data and sound data are continually collected from the environment respectively through webcam and microphone. Once the conditions of surrounding environment is changing, the corresponding alarm will be raised immediately.

The package contains:

• DirectShowLib: Manipulate the webcam, grab images. I have wrapped functions related to IntruderDetection in class CManipulateWebcam. For more information about DirectShowLib, please see directshow.net.
• ManipulateMicrophone: Manipulate the microphone, grab sound, play sound. Functions related to IntruderDetection are wrapped in class CManipulateMicrophone. For more information about ManipulateMicrophone, please see the part "Using the code" of the article: Sending and playing microphone audio over network.
• WavStream: Read .wav file, save data into .wav file. The class CManipulateMicrophone uses WavStream.dll to save sound data into .wav file. The following code is useful:
  C#

  WaveStreamWriter wavwrite = WaveStreamWriter
  	(FileName, SamplingRate, Channels, BitPerSample);
  // SoundBuffer is an array which contains the sound data you want to save
  wavwrite.Write(SoundBuffer, BufferLength);

• IntruderDetection: The main application needs (DirectShowLib.dll, ManipuateMicrophone.dll, WavStream.dll).
• DirectShowLib, ManipulateMicrophone and WavStream are independent projects. You can use DirectShowLib in your application to interact with webcam. Use ManipulateMicrophone to interact with microphone. Use WavStream to interact with .wav file. Of course, use IntruderDetection to detect intruder.

How Does the IntruderDetection Work

• Video detection: Check the box "Video Detection" to open video detection. If the box "Video Alarm" is checked, the alarm will be activated. Once there is something, abnormal information (0-3 second) will be automatically recorded in the folder: IntruderDetection>>bin>>Debug>>image>>(yyyy-MM-dd-H), filename (mm-ss-ff).jpg
• Sound detection: Check the box "Sound Detection" to open sound detection. If the box "Sound Alarm" is checked, the alarm will be activated. Check the box "Sound Echo" to play the sound from microphone. In the same way, abnormal information (0-3 second) will be automatically recorded in the folder:
  IntruderDetection>>bin>>Debug>>sound>>(yyyy-MM-dd-H), filename (mm-ss-ff).wav

Background knowledge of IntruderDetection

Video detection

For video, define the distance of two image a(i,j) (r_a,g_a,b_a) and  b(i,j)(r_b,g_b,b_b) as

D₁=∑((|r_a-r_b|+|g_a-g_b|+|b_a-b_b|)/3)  i=1,2...; j=1,2,... or 

D₂=∑((|r_a-r_b|²+|g_a-g_b|²+|b_a-b_b|²)/3)  i=1,2...; j=1,2,...  or

D₃=max((|r_a-r_b|+|g_a-g_b|+|b_a-b_b|)/3)  i=1,2...; j=1,2,...

If the D₁,D₂ or D₃ is bigger than a threshold value, we believe something is happening.  For example:

private bool CompareImage(Bitmap a, Bitmap b)
{
    int width = b.Width;
    int height = b.Height;
    BitmapData data_a = a.LockBits(new Rectangle(0, 0, width, height), 
			ImageLockMode.ReadWrite, PixelFormat.Format24bppRgb);
    BitmapData data_b = b.LockBits(new Rectangle(0, 0, width, height), 
			ImageLockMode.ReadWrite, PixelFormat.Format24bppRgb);
    unsafe
    {
        byte* pa = (byte*)data_a.Scan0;
        byte* pb = (byte*)data_b.Scan0;
        int offset = data_b.Stride - width * 3;
        double Ra, Ga, Ba, Rb, Gb, Bb;
        double temp1 = 0, temp2 = 0;
        double max = 0;
        for (int y = 0; y < height; y++)
        {
            for (int x = 0; x < width; x++)
            {
                Ra = (double)pa[2];
                Ga = (double)pa[1];
                Ba = (double)pa[0];
                Rb = (double)pb[2];
                Gb = (double)pb[1];
                Bb = (double)pb[0];
                temp1 = (Math.Abs(Ra - Rb) + Math.Abs(Ga - Gb) + Math.Abs(Ba - Bb)) / 3;
                temp2 += temp1;
                if (temp1 > max)
                {
                   max = temp1;
                }
                pa += 3;
                pb += 3;
            }
                pa += offset;
                pb += offset;
        }
        temp2 = temp2 / (height * width);
        a.UnlockBits(data_a);
        b.UnlockBits(data_b);
        if (max > 200 || temp2 > 5)
        {
            return true;
        }
        else
        {
            return false;
        }
    }
} 

In the default case, pointer is not allowed in C#. You need to open the Properties of IntruderDetection, click Build, check the box "Allow unsafe code". You may adjust the threshold values to adapt your machine.

Sound Detection

Calculate the sound energy every 25 millisecond. If the energy is bigger than a threshold value, something is happening.

Conclusion

This article shows how to use video and sound information to detect intruder (dig into IntruderDetection). In this process, some useful intermediate technology is used, including how to manipulate webcam (dig into DirectShowLib), how to manipulate microphone (dig into ManipulateMicrophone), how to interact with .wav file (dig into WavStream). Have fun!

History

• 14^th September, 2011: Initial version of the article