Introduction
Hello all! My name is Diego Catalano, and I am a Java programmer. And this for a short while had been a problem, because I am a huge fan of the AForge.NET and Accord.NET Frameworks for image processing and scientific computing in .NET. Unfortunately, this meant I couldn't use either framework in one of my favorite environments: Android.
In this article, I am going to present an ongoing effort to bring the best features from both frameworks to Java; and, in turn, give a hint on what we can do once we get all this power into mobile computing. I am going to present the Catalano Framework, a framework for scientific computing written in Java with maximum portability in mind. Here I am getting some help from Accord.NET's author, César Souza, to not only bring the newest additions into my own framework, but to also contribute my own implementations and discoveries into Accord.NET.
Also, let me get this straight. Due to my knowledge on using both AForge.NET and Accord.NET I have been rewarded with a job. My desire, today, is thus to return the favor and reward the community with my own contributions. A tool to bring scientific computing features to desktop, server and mobile applications, all in a single shell.
Introduction
The main idea was to extend, improve and port most classes from both frameworks to Java. Thus said, I've established the same well known architectural division found on the aforementioned libraries, using distinct namespaces for distinct needs.
If you are already familiar with any of the two projects, you should feel at home. The framework is divided into the following namespaces:
- Catalano.Image
- Catalano.Fuzzy
- Catalano.MachineLearning
- Catalano.Math
- Catalano.Neuro
- Catalano.Statistics
In order to use the framework, just add library references as you would do in AForge.NET or Accord.NET. In NetBeans, you can add references to the .jar files in the same way you would load a .dll library in Visual Studio or SharpDevelop.
If you are using Eclipse to develop an Android application (i.e. using the Android SDK), there is only one slight difference you will need to do: Instead of referencing Catalano.Image.jar, add a reference to Catalano.Android.Image.jar instead.
This is all you need to do to be up and running with the library. Now let's see what we will be able to use it for!
Imaging Processing Library
Image processing is one of the core AForge.NET Features. Besides providing all implementations found on the aforementioned framework, I've added several new image processing algorithms, specially suited for a variety of applications. One of the applications I would like to show is the automatic binarization of image documents.
Let the two images below demonstrate how we can recover a document using Bradley Local Thresholding:
Original Image: Processed Image:
Here is the code I've used to make this work:
FastBitmap fb = new FastBitmap("bradley.png");
Grayscale g = new Grayscale();
g.applyInPlace(fb);
BradleyLocalThreshold bradley = new BradleyLocalThreshold();
bradley.applyInPlace(fb);
JOptionPane.showMessageDialog(null, fb.toIcon(), "Result", JOptionPane.PLAIN_MESSAGE);
As it might be seen, here we are using the almost exact interfaces given by the AForge.NET Framework. This should make it easy to port knowledge from a framework to the other with minimum work. Besides the Bradley Local Threshold, in the Catalano Framework you will find several other algorithms, like
- Bernsen Threshold
- Niblack Threshold
- Sauvola Threshold
- Wolf Jolion Threshold
If you are not into image binarization, let's find another example: Let's find some interest points in a scene.
Corner detection algorithms have several uses, including some uses already shown here in other CodeProject articles. Among other things, they can be used to match points between images, to perform feature extraction for posterior image classification, scene understanding, image registration, among several others.
In the Catalano Framework, corners detection follows almost exactly the interfaces found in Accord and AForge.NET. The example below shows how to apply Susan's corners detection to identify interest points in a sample image:
FastBitmap fb = new FastBitmap("blocksTest.gif");
Grayscale g = new Grayscale(Grayscale.Algorithm.Average);
g.applyInPlace(fb);
SusanCornersDetector susan = new SusanCornersDetector();
ArrayList<IntPoint> lst = susan.ProcessImage(fb);
fb.toRGB();
for (IntPoint p : lst) {
fb.setRGB(p, 255, 0, 0);
}
Susan Corners Detector Harris Corners Detector
Imaging Processing Library (Android)
Now, here comes the real power of this framework. Let's see how we can achieve the same things we've done before, but this time from within an Android device. The following code reproduces the first example given in this article, binarizing a sample image using Bradley's Local Thresholding algorithm, but directly from an Android device.
ImageView image = (ImageView)findViewById(R.id.imageView1);
Bitmap bm = BitmapFactory.decodeResource(getResources(), R.drawable.sample20);
FastBitmap fb = new FastBitmap(bm);
Grayscale g = new Grayscale();
g.applyInPlace(fb);
BradleyLocalThreshold bradley = new BradleyLocalThreshold();
bradley.applyInPlace(fb);
image.setImageBitmap(fb.toBitmap());
Several other image processing algorithms are also available under this same shell. Other examples include edge detection algorithms, such as the Sobel operator. Being an edge detection algorithm means this algorithm can highlight any borders or boundaries presented in an image. Its usage is completely similar in both Android and standard Java applications.
The following example demonstrates how to run the Sobel Edge Detector in Android.
Grayscale g = new Grayscale();
g.applyInPlace(fb);
SobelEdgeDetector sobel = new SobelEdgeDetector ();
sobel.applyInPlace(fb);
I hope this can get most of you interested - but note we have just started scratching the surface of what is available in the framework!
Improvements
One of the oldest features available on AForge.NET is the ability to perform a Fast Fourier Transform (FFT) in an image. This feature transposes an image from the space domain to the frequency domain, allowing the use of several interesting image processing filters such as frequency-pass filters or even helping improving convolution through the Convolution-in-space/Multiplication-in-frequency theorem.
However, there is a catch present on most FFT implementations, including the one available in AForge.NET. The problem is that the FFT, by definition, can only be performed on images which have both width and height as powers of two. This means it is fine to compute a FFT for 64x64, 512,512, 1024x1024 images, but if we attempt to compute the transform on a 592x634 image, it will not work.
Please note that this is not a limitation of the implementations per se, but rather a consequence of one of the basic assumptions that make the Fast Fourier Transform, well, Fast in the first place. In order to circumvent this, one can use a feature known as zero-padding.
Zero-padding simply adds extra zero-valued pixels to an image until its size forms a power of two. After that, we compute the FFT as normal, and then we remove the extra elements we added in the first step. As such, please consider the following example using AForge.NET in C#:
Bitmap image = new Bitmap("test.bmp");
image = new GrayscaleBT709().Apply(image);
ComplexImage ci = ComplexImage.FromBitmap(image);
ci.ForwardFourierTransform();
Here, if test.bmp is not test.bmp does not have a size which is a power of two, we will likely encounter an AForge.Imaging.InvalidImagePropertiesException.
Now, in the following example, we will not only perform a Fourier Transform, but also run a frequency filter to extract low frequencies from an arbitrarily sized image:
FastBitmap fb = new FastBitmap("c:\\files\\test.bmp");
fb.toGrayscale();
JOptionPane.showMessageDialog(null, fb.toIcon(), "Image", JOptionPane.PLAIN_MESSAGE);
FourierTransform ft = new FourierTransform(fb);
ft.Forward();
fb = ft.toFastBitmap();
JOptionPane.showMessageDialog(null, fb.toIcon(), "Fourier Transform", JOptionPane.PLAIN_MESSAGE);
FrequencyFilter ff = new FrequencyFilter(0, 60);
ff.ApplyInPlace(ft);
fb = ft.toFastBitmap();
JOptionPane.showMessageDialog(null, fb.toIcon(), "Frequency Filter", JOptionPane.PLAIN_MESSAGE);
ft.Backward();
fb = ft.toFastBitmap();
JOptionPane.showMessageDialog(null, fb.toIcon(), "Result", JOptionPane.PLAIN_MESSAGE);
The results are shown below. We can see how the non-square image was transformed into the frequency domain, filtered, and then converted back to the space domain in the same, previous original size.
| |
|
|
| Fourier Transform (512,512)
|
| | |
| Frequency Filter (512,512)
|
|
Benchmark
I've decided to dedicate a section to present a benchmark between the Catalano framework and the standard features available in AForge.NET and Accord.NET. Here, the goal is not to prove one is better than the other, but instead is to provide some baseline comparisons to indicate what can be expected from the current implementations.
All tests have been repeated 100 times. Initial executions (before those 100 times) have been discarded to let any possible compiler/JIT optimizations quick in before the actual measurements. All tests have been made outside Visual Studio, i.e. by going to the binary folders and running the executables directly. All binaries have been compiled in Release mode, for Any CPU (meaning they were ran in 64-bits mode). All tests have been ran using the same machine with the following specifications:
Processor: Intel® i7-2670QM CPU @ 2.20Hz, 4 cores (8 in HyperThread), 6 MB cache
RAM: 8 GB, DDR3 1600MHz
The first test started with a simple Median filter. In this test, we processed a 1024 x 770 RGB image using a 5x5 kernel window.
In this particular test, the difference between the Catalano Framework (non concurrent) and AForge.NET is roughly 10s. If we consider a parallel implementation, which should make use of all 4 cores in the test machine, the difference grows even more, to a whooping 71 seconds.
The second test started with a simple Sobel Edge Detector. In this test, we processed a 1024 x 770 Grayscale image using both parallel, direct, and AForge.NET's default implementation.
As we can see here, the difference between the Catalano Framework (non concurrent) and AForge.NET is of just 467 ms. The concurrent version improves further on this mark, being roughly twice as fast as the original AForge.NET implementation. As the test machine had 8 virtual cores, this means the algorithm is not straightly easily parallelizable.
Now, on with the third test. This test involved measuring the time it takes to process a Gaussian Blur over a 1024 x 770 RGB image. Results are shown below.
In this particular test, we can see that AForge.NET outperforms Catalano with a margin of at least 4 seconds. Again we can see both implementations are comparable, while AForge.NET scores a little better at this test.
The fourth test was done against a Harris Corners Detector from Accord.NET. In this test, we had processed a 1024 x 770 Grayscale image, with results shown below.
As we can see, Accord.NET scores slightly better at this test, but the difference is very comparable and equivalent among the two implementations. This is an indication both C# and Java can also shown very similar performance on such applications.
Finally, the last test was done by comparing the FAST Corners Detector in in both Catalano and Accord.NET. This test involved the processing of a 1024 x 770 Grayscale image using FAST-9, which is a particular version of the FAST detection. Results are shown below.
As it can be seen, this time the difference between Catalano Framework and Accord.NET was actually very noticeable, being around 23649 ms. This test clearly reveals that Accord.NET's implementation outperforms the Catalano Framework implementation. As we can see, there is much room for improvement and work to be done - but mostly, the overall intent, which was to bring image processing into Android, has mostly been achieved!
Conclusion
In this article, I've attempted to present the Catalano framework to the world. Here we have seen how this relatively new framework complements the open-source ecosystem by offering a new way to create image processing and scientific computing applications in Java by unifying the best features from the Accord and AForge.NET Frameworks. Development is on full steam and is being ran at Github.
During the time I've started building the framework until now, I acquired a lot of knowledge in several different areas, starting from the scientific method up to software engineering and systems design. For that, I am quite grateful for all the good people who I found during the way, including the fellow researches who agreed to share their code under LGPL-compatible licenses so they could be integrated in the framework.
I hope you will enjoy the framework the same amount I've enjoyed working on it!
Future works
From now, there are several areas I intend to improve. Almost all of my contributions are being gradually shared with César (from the Accord.NET Framework) to be integrated into his framework; and other features are also being ported from .NET to Java with some help.
I also intend to create a blog explaining each filter in Catalano Framework, and present some actual sample applications demonstrating its features. There is also a plan to move the repository to GitHub to make code contributions easier.
There are also some key points I would like to continue working on:
- Port to Google Glass® (I would gladly accept one for testing purposes :-P ).
- Convert all filters to use multi-threaded, parallel processing.
- Add a library for Biometrics related algorithms.
- Continue working, extending and improving the whole framework.
History
- 01 November 2013 - First version published.
