Detection Implementation
Our first step is detection. We chose Viola-Jones algorithm as our implementation. The implementation required full frontal upright faces, which means the entire face must point to the camera and should not face to other sides. Although it is a restraint, we are only working on the basic facial detection and focusing more on the modification to best fit the face. Therefore, this robust, efficient implementation is our best choice. We decided to first try the provided detection method based on Viola-Jones to test our selected images. Once the outcomes turned out to be good, we will go on implementing a better detection based on current methods. For the better modification, such as the rotation process, we implemented an alternative method to detect left eye and right separately after cutting the face into half.
We are doing this by four steps:
First, we start by detecting the face one by one from the target image and take every single face out.
Second, we cut the face into left half and right half using the four coordinates of the defined area. The reason we do this is because we want to detect left face and right face separately, since the detection of a single eye will not function well over a whole face.
Therefore, the next step, we detect the left eye in the left half face and do the same to the right half face to obtain two boxes of the eyes.
Finally, we compute the distance between two boxes and the angle for the rotation of the sticker, in this example, a pair of glasses. The resize of the source image is no longer based on the detection of a pair of eyes, but based on the difference between the x and y coordinates of the two boxes.
Modification
After the detection of the features, we can now do the modifications on them. First, we searched for the materials we need for our modification. The desired material are in PNG type, which is a common type of image for image processing. In addition, the materials we gained is straightforward to replace over the original image: The margin parts which we are not replacing has all zeros over the pixels. Therefore we can simply detect if the pixel is a non-zero value, and cover the original image with every such pixel. The use of restricted type of source image has several advantages: We do not need to apply blending filters to eliminate the redundant backgrounds, which could also cause blurring or ghost.
We made several tests using different stickers and we decided to work on the glasses because it is related to the position of the eyes. And since we don’t have time to test stickers other than glasses. For this project, we will only implement modification of glasses, based on the detection of human eyes. The detect, resize, rotate and replace process is more or less the same when applying other source image on the target image.
1. Replacement
The implementation of our modification is intended to replace the area we detected with the source image. Therefore, we wrote several methods for the replacement. The first thing to do is the resize of source image. To fit the output image, we resize the source image to fit the detected area by using the imageresize function.
Original source image:
Resized source image:
The replacement starts from the upper left corner of the detected area. We detect every pixel and determine if we need to replace the original image with it based on the color of the source pixel: if it is zero, then we keep the original one. This straightforward method allows us to test a large amount of PNG source image on the faces without modifying other functions. However, we did come across some problem during our testing: When we tried to add a pair of black glasses over the eyes, the result turned out to be nothing added. We immediately figured out that we can’t treat a pure black source image (0,0,0) the same way. Because it is just a pure zero image, so the program won’t do anything after the detection. We solved the problem by changing the color of the source image, so we will not deal with images with pure black color.
2. Rotation
Since we need a better detection method for rotation. We went back to our detection method and improved our detection - the computation of the degree for rotation, as detailed described in the detection section.
After we obtained the degree for rotation, we resized and rotated the source image. We used the lower left corner of the left eye and the upper right corner of the right eye to obtain the length of the height and width of the desired image. Then, we tried to replace the target image with the original method, starting from the lower left corner. One issue we came up with is the scale of the source image. Since the size of the source image will always change after the rotation: it will result in redundant marginal area.
Before rotation:
After rotation:
We are using black as background for easier overlay process. Overall, the results are good and fit the faces. We did several different tests using this method.
Results
Our primary test, to test the rotation of the sticker.
A image with slight degree.
A larger image with slight degree.
Overview of the Face Detection and
Adding Stickers Process
Presentation
