Introduction
OpenCV is a popular open source image processing library that can easily run on Android. Although you will need to use the Android Native Development Kit (NDK) and may need C++, it's quick and easy to get started with OpenCV on Android. This article shows you how to run OpenCV on a standard Android Studio AVD. You do not need any special hardware or software, just Android Studio. You do not even need an Android phone as the AVD will work fine.
To help you understand how OpenCV image processing can be applied to 360 images, the AVD simulates the RICOH THETA, a consumer 360 camera with an embedded Android OS.
The same technique can be used on any Android mobile phone or any camera that runs the Android OS inside the camera.
If you're interested in 360 images, a companion article, VR Image Processing with Android, introduces the basic concepts of 360 images used in VR headsets and Google StreetView. It also covers basic image processing using the standard Android BitmapFactory and Bitmap classes.
Don't be intimidated by OpenCV. It's easier than you think.
Background
OpenCV is a flexible library for computer vision and image processing. The open source project, HDR2EXR, built by developer Kasper Oelemans, combines 33 still 360 degree images into a single EXR file. That file is then used to provide lighting for 3D assets. In the Blender modelling example above by Alexandre Dizeux, the light and reflection are generated from a file produced by OpenCV running inside a consumer camera, the RICOH THETA.
OpenCV can also be used with a video stream. In the open source project theta plugin opencv detection sample, developer roohii_3, shows how to trigger the camera to take a picture when motion is detected.
To keep things simple, I will show you how to apply OpenCV on a single still image. This will get you started on the basics of running OpenCV on an Android device.
Using the Code
Fork this project from GitHub.
https://github.com/codetricity/no-camera-opencv
Open the project in Android Studio.
Build
- JRE 1.8.0
- Android Studio 3.3.2
- Gradle 4.10.1
- OpenCV 3.4.5
- In
gradle.build (app), specify sourceCompatibility and targetCompatibility as 1.8 due to use of lambda
In Android.mk, set your include path.
To see the Android.mk file, go to the "Project Files" view of Android Studio. The default view is "Android". You need to access Project Files from the drop-down menu.
Edit the path in the screenshot below to match the path of your local OpenCV source code (which is part of this repo).
When you run the project, it will start a simulator.
Test out the processing with the included sample images.
This demo uses the built-in OpenCV sample code to swap the color space.
Look at the short C++ example here.
The section RGBA2BGRA will swap the color space. You can now edit that line to process the still image in other ways.
This code will flip the image vertically.
cv::flip(m_src, m_dst, 0);
Limitations
Not all of the OpenCV library is working in this 360 camera simulation. This simulation also uses OpenCV3, not the more recent OpenCV4.
Points of Interest
Although I've focused on running OpenCV inside of a camera, you can run OpenCV directly on your phone using the OpenCV for Android SDK. I like the unique perspective of a full 360 degree view, but I understand that specialized hardware is expensive. Through learning about OpenCV, I also learned about EXR files and how they are used in the special effects industry to provide global lighting in tools like Blender (open source), Autodesk MAYA, Nuke from Foundry, and Davinci Resolve Fusion.
To test the global illumination, someone even built a 3D asset of the camera many of us use.
Next Steps
Although OpenCV, the Android NDK, and C++ might be new concepts, it's possible to get started with minor code changes. If you build something with OpenCV and 360 images or video, let us know about it in the theta360.guide dev community.
History
- 28th June, 2019: Initial version
