Face Alignment According to Eye Position in Emgu CV

Introduction

Face recognition is an important task in image processing. But because of various poses of faces, such as left or right rotated face, accuracy of face recognition comes down. Hence, it is very necessary to rotate face and make it frontal for better face recognition.

Background

I searched lot of web sites for face recognition and hence for face rotations. I got some code and hints but they did not work for me at all. Hence, on implementing some of my ideas, I wrote down this code for face alignment according to eye positions. And, it is working for me with more than 97% accuracy.

Using the Code

First, detect the right and left eye using haar cascade.

HaarCascade haar_righteye = new HaarCascade(path + "haarcascade_mcs_righteye.xml");
HaarCascade haar_lefteye = new HaarCascade(path + "haarcascade_mcs_lefteye.xml");      

//following variables are used to detect right eye and 
//left eye for fixing position and hence used for face alignment
MCvAvgComp[][] Right_Eye = upper_face.DetectHaarCascade(haar_righteye, 
1.4, 4, Emgu.CV.CvEnum.HAAR_DETECTION_TYPE.DO_CANNY_PRUNING, new Size(4, 4));
MCvAvgComp[][] Left_Eye = upper_face.DetectHaarCascade(haar_lefteye, 
1.4, 4, Emgu.CV.CvEnum.HAAR_DETECTION_TYPE.DO_CANNY_PRUNING, new Size(4, 4));

Here, upper_face is the upper part of face from mid of nose. Simply, I will take sub rectangle of detected face image with height set to half of original.

int height = Face.Height / 2;
Rectangle rect = new Rectangle(0, 0, Face.Width, height);
Image<Gray, byte> upper_face = Face.GetSubRect(rect);

I will find out the left eye and right eye according to eye position (X-value) in an image. The said haar cascade does not always directly give left and right eyes. Hence, it is important to check for them.

To actual rotate the face, I will find out angle according to eye positions. Then, using this angle in calculation with 180 degree, the final angle to rotate face image is found out.

var deltaY = (L_eye.rect.Y + L_eye.rect.Height / 2) - (R_eye.rect.Y + R_eye.rect.Height / 2);
             //using horizontal position and width attribute find out the variable deltaX
             var deltaX = (L_eye.rect.X + L_eye.rect.Width / 2) - (R_eye.rect.X + R_eye.rect.Width / 2);
             double degrees = (Math.Atan2(deltaY, deltaX) * 180) / Math.PI;//find out the angle 
                                                                           //as per position of eyes
             degrees = 180 - degrees;
             Face = Face.Rotate(degrees, new Gray(220),true);

Find the complete code below:

public Image<Gray, byte> AlignFace(Image<Gray, byte> Face)
    {
        try
        {
            int height = Face.Height / 2;
            Rectangle rect = new Rectangle(0, 0, Face.Width, height);
            Image<Gray, byte> upper_face = Face.GetSubRect(rect);

            //following variables are used to detect right eye and 
            //left eye for fixing position and hence used for face alignment
            MCvAvgComp[][] Right_Eye = upper_face.DetectHaarCascade
            (haar_righteye, 1.4, 4, Emgu.CV.CvEnum.HAAR_DETECTION_TYPE.DO_CANNY_PRUNING, new Size(4, 4));
            MCvAvgComp[][] Left_Eye = upper_face.DetectHaarCascade
            (haar_lefteye, 1.4, 4, Emgu.CV.CvEnum.HAAR_DETECTION_TYPE.DO_CANNY_PRUNING, new Size(4, 4));

            bool FLAG = false;
            foreach (MCvAvgComp R_eye in Right_Eye[0])
            {                          
                foreach (MCvAvgComp L_eye in Left_Eye[0])
                {
                    if (R_eye.rect.X > (L_eye.rect.X+L_eye.rect.Width))
                    {
                        //upper_face.Draw(R_eye.rect, new Gray(200), 2);
                        //upper_face.Draw(L_eye.rect, new Gray(200), 2);
                        var deltaY = (L_eye.rect.Y + L_eye.rect.Height / 2) - 
                                     (R_eye.rect.Y + R_eye.rect.Height / 2);
                        //using horizontal position and width attribute find out the variable deltaX
                        var deltaX = (L_eye.rect.X + L_eye.rect.Width / 2) - 
                                     (R_eye.rect.X + R_eye.rect.Width / 2);
                        double degrees = (Math.Atan2(deltaY, deltaX) * 180) / Math.PI;//find out 
                                                                   //the angle as per position of eyes
                        degrees = 180 - degrees;
                        Face = Face.Rotate(degrees, new Gray(220),true);
                        FLAG = true;
                        break;
                    }                    
                }
                if(FLAG==true)
                {
                    break;
                }
            }          
        }
        catch (Exception d) {
            op += " Align Error: " + d.Message;
        }
        res = op;
        return Face;
    }

Points of Interest

The calculation of calculated degree has angle 180, such that the code line...

degrees = 180 - degrees;

...is one of the interesting points in this code. Because, by adding this one line code, accuracy is improved.

Why will I add this?

Find out....