Python and the OpenCV library make it very easy to work with visual input such as images or videos. In this tutorial, I discuss contents on image-blending and present a way of merging pixel information of two images. To accomplish this, I use a third image (the so-called mask), which serves as a template for the blending process. Because this template is a matrix of weighting factors, it is possible to produce results, which cannot be obtained with blending methods one usually finds on the web (or at least I did not find them).
Introduction
Photoshop and other image processing programs as well (e.g., freeware alternatives such as GIMP or Paint.net) offer a wide range of possibilities to edit and manipulate images and photos. However, sometimes, one wishes to perform operations that are not built into standard software or simply look under the hood and get an idea how specific photo-editing procedures might work. Both goals can be achieved by trying to translate photo-editing procedure into programming code. Here, in this piece of text, I will do that and present code for merging parts (or the whole image as well) of an image with another image.
To comprehend the contents of this tutorial, you should have a basic knowledge of Python (Python 3) and the ways in which images are represented on a computer (i.e., RGB or BGR model). However, as the code is not overly complicated, it should be accessible to people with backgrounds in other programming languages as well. If you plan to execute or play around with the code I have loaded up, you need to install OpenCv and NumPy on your machine (use the pip command or the Anaconda Navigator to do that) along with Python, of course. OpenCV is a freely available image processing library (interfaces are available for C++, Python etc.) containing many features that help to process visual input such as videos and images. Here, in this piece of code, I only employ some basic methods. I use commands for loading and displaying images on a screen and set up a while loop to enable some interactions with the program (but this is also very much down to earth). As already mentioned, NumPy is also required. NumPy is Python’s number one library to handle numbers, arrays and matrices and it comes in very handy that OpenCv image data can be manipulated and modified using NumPy methods.
Background
Blending Images
Illustrative sample code on how to use OpenCV to merge information stored in one image with information of a second image is widely available on the web (e.g., see this link). For instance, the cv2.addWeighted(image1, weighting_factor1, image2, weighting_factor2)
OpenCV function is a one-liner which is able to perform image blending in a straightforward way. If one chooses 0.5 for both weighting factors, the function returns an equally weighted (or mean) blend of two images. Choosing 0.7 and 0.3 as input weights, will result in an image consisting of 70 percent pixel color of image number one and 30 percent pixel color of image two. Choosing 1 and 0, to give another example, will return an unchanged version of image one, whereas 0 and 1 will return an unchanged version of image two. I mention this, because it becomes important below, when I go into the details of the code I wrote.
Using a Mask for Image Insertion
The cv2.addWeighted()
function is particularly useful when information of two equally sized images is combined (or more precisely, regions of interest) and the same weighting factors are applied to all pixels. However, sometimes one plans to transfer a specific region from one image to another image. This can be done in different ways, but one common one is using a mask. A mask is usually a binary image (only containing black and white pixels) that serves as a template enclosing the regions for insertion or blending. To clear that up, I give an example. If one wants the sky of an image to be replaced by the background of a second image, the mask (or the binary image) needs to have a black region representing the sky of image one, so that the program “knows” where to put the background of image two. In other words, where the pixel color black (0 in an 8-bit image; [0,0,0] for a 24-bit image) is met in the mask, pixels of the first image are replaced by pixels of the second image.
Using a Matrix of Weighting Factors for Image Overlay
In the sample code I present here, I marry methods of image blending with methods for image insertion based on a mask. In contrast to the “standard” methods, I use a non-binary mask (can hold all values between 0 and 255), in which each pixel serves as an individual weight. Thus, I do not define an overall weighting factor (e.g., 0.7) for image one and an overall weighting factor for image two (e.g., 0.3), but use each pixel of the mask as weighting factors instead (see Figure below). For instance, if a pixel in the mask at position [X, Y] holds the value of 0 (which is black) the pixel of image 1 is placed on that spot. If a pixel of the mask holds the value of 255 (white), the corresponding pixel of image two takes this place. If it is something in between (e.g., 170) one gets a mix of image one and image two. Using the mask as a map consisting of weighting factors offers more flexibility in overlaying two images. It makes it possible to produce transitions or “soft” borders (and even more).
Using the Code
I divided the code sample you can find in the repository into four sections. Below, I only present code from the two inner sections, as they cover the main topic of this tutorial. For the sake of completeness, I briefly comment on section 1 and 4 in the following and then present essential information on section 2 and 3 in the form of bullet points.
In section 1, you find OpenCV code for loading the images (i.e., cv2.imread(“X.jpg”,cv2.IMREAD_UNCHANGED)
) needed. There are three images: image 1
is the surface on which the contents of image 2
are pasted, and image 3
is the mask or template defining the conditions for the blending operation. There are some additional features as well, but I leave it to you to find out what they are for (there are informative comments in the sample code I uploaded).
In section 4, you find a “while
loop” that allows a kind of rudimentary interaction with the program. The loop is waiting for key events. The program is quitted when pressing “q
”, while pressing “s
” saves the results of the blending operations and then quits the program.
Sections 2 and 3 contain code for blending information of two images based on the weighting factors provided by the mask.
- The function
mix_pixel(pix_1, pix_2, perc)
processes input in a similar manner as cv2.addWeighted()
does. It takes pixel information from two images and blends it according to the weights given in the third parameter (i.e., perc
) of the function (e.g., when perc
is in the middle of 0 and 255, it gives a fifty-fifty mix of pixel 1 and pixel 2; see also in section blending images). When the parameters are NumPy
arrays (Note: All arrays need to have the same dimension) the operations are applied to all corresponding pixels of the images. The third parameter is an array (the mask) containing an individual weighting factor for each pixel. - The
blend_images_using_mask(img_orig, img_for_overlay,img_mask)
function is quite simple. It takes the three images, checks whether the mask is a 3 channel greyscale image (otherwise, it would not have the same dimension as the other images), calls the mix_pixel()
function and then returns the result as a 3 channel 8 bit unsigned integer (otherwise OpenCV commands for displaying images would fail). - The final commands you find in the sample below are for displaying the different types of images, in order to make the outcome of the procedures visible and accessible for evaluation. There are also some lines of code that are used for making the images smaller, as the original size of a photo is usually is too large for a standard screen.
import numpy as np
import cv2
def mix_pixel(pix_1, pix_2, perc):
return (perc/255 * pix_1) + ((255 - perc)/255 * pix_2)
def blend_images_using_mask(img_orig, img_for_overlay, img_mask):
if len(img_mask.shape) != 3:
img_mask = cv2.cvtColor(img_mask, cv2.COLOR_GRAY2BGR)
img_res = mix_pixel(img_orig, img_for_overlay, img_mask)
return img_res.astype(np.uint8)
img_blended = blend_images_using_mask(img, img_insert, img_insert_mask)
rf = 0.4
wi = img.shape[1]
hi = img.shape[0]
img_sm = cv2.resize(img, (int(wi * rf), int(hi * rf)),
interpolation=cv2.INTER_CUBIC)
img_insert_sm = cv2.resize(
img_insert, (int(wi * rf), int(hi * rf)), interpolation=cv2.INTER_CUBIC)
img_blended_sm = cv2.resize(
img_blended, (int(wi * rf), int(hi * rf)), interpolation=cv2.INTER_CUBIC)
cv2.imshow("Original Image", img_sm)
cv2.imshow("Insert This Image", img_insert_sm)
cv2.imshow("Blended Images", img_blended_sm)
Points of Interest
There is no guarantee that the code is free of bugs and works on each platform. Be careful when running the code and ending it not properly (by not pressing 'q' or 's'). You should restart the Kernel then (is possible when you use Jupyter Notebook) or remove the thread from the system (TaskManager in Windows). Also, all files have to be in the same folder for the program to be executed properly. Feel free to make suggestions for further improvement.
History
- 23rd December, 2020: Initial version