Introduction
In image processing, it is often useful to display overlay graphics on images, e.g. to display a segmented region, but this is not provided for in .NET. I decided to create a custom control that can easily be included into other projects and that provides some basic functionality for images, like zooming, panning and the display and creation of annotations. Together with a library for image processing which I will present later, this should provide the basic building blocks for any application that tackles image processing.
The annotation classes
My first try at this problem as published here was a bit haphazard, and left me not very satisfied. Indeed, the object hierarchy wasn't logical, and the collections weren't type safe. Also, animation wasn't flicker-free. I decided to rework the whole project using a clean object oriented approach, and came up with the following:
-
Class AnnotatedImage
Control that holds the bitmap on which the annotations are drawn, and the properties and methods for drawing that bitmap. This is the control that is to be included in other projects.
ImageSize
DisplaySize
DisplayOffset
DisplayZoom
Bitmap
Annotations
MaximumSize
PixelAveragingWindowSize
DrawErasableRectangle
DrawErasableLine
SaveImage
LoadImage
-
Class Annotations
Class that contains all code for the annotations, and all its collections.
-
Class Container
Class used to contain annotations.
-
Class ContainerCollection
Class defining a strongly typed collection of Containers.
-
Class Annotation
Abstract class used to derive more specific annotation classes.
Key
RegionAddMode
Color
Visible
Text
Key
Points
Region
BorderRegion (Readonly)
AddPoint
RemovePoint
ClearPoints
-
Class AnnotationCollection
Class defining a strongly typed collection of annotations
-
Class AnnotationRegion
Class for an annotation consisting of points that form a region, as such.
-
Class PointBasedAnnotation
Abstract class used to derive more specific point-based annotation classes
Annotation properties and methods
LinesVisible
PointsVisible
MaximumNumberOfPoints
-
Class AnnotationLoose
Class for an annotation consisting of loose points, i.e. points which do not define a region
-
Class AnnotationBorder
Class for an annotation consisting of points that form a closed contour and thus define a region by forming a border.
PointBasedAnnotation properties and methods
AddRectangle
The classes AnnotatedImage and Annotations are meant to work together, and they hold references to each other. By instancing an AnnotatedImage, you get the following hierarchy:
AnnotatedImage
Annotations
Containers (Type safe collection)
Container
Annotations (Type safe collection)
All the code for drawing the annotations is inside the Annotations class, except for 2 routines for XOR drawing that are stored inside the AnnotatedImage. The Annotations class intercepts the Paint event of the AnnotatedImage control in order to redraw every annotation Container object, which in turn calls the 'Draw' method on every individual Annotation object. The AnnotatedImage control implements double buffering to ensure smooth drawing.
All events pertaining to annotations are handled in the Annotations class, e.g. interactive drawing, and all those concerned with the image in AnnotatedImage, e.g. zooming. This means a Windows event like a mouse click, will trigger several event handlers, and these will decide if they should do something with it.
Using the code
The code will compile to a DLL that can be added to the Visual Studio toolbox, and can then be included as a control in any application. Of all the events exposed, Resize is the only one that must be handled, as it is through this event that the parent control can resize itself in order to accommodate changes in the size of the custom control, e.g. after zooming. The custom control usually has the size of the displayed image, except when it exceeds MaximumSize for very large images or a large displayZoom (do not forget to set this property!). In that case, only a portion of the image is displayed, and panning can be activated by dragging the image with the middle mouse button. On top of that, the control responds to the wheel mouse and z and shift-z for zooming, to w and shift-w for setting the averaging window size that can be used to provide feedback to the user about the color in the area around the cursor. This averaging window is also drawn on the control.
Explanations and points of interest
At first, I tried to derive my custom control from the PictureBox class, but I failed to draw any graphics over the bitmap that can be assigned to the PictureBox.Bitmap by overriding the Paint method. Indeed, it seemed that the bitmap was drawn after any custom drawing I performed on the graphics created from the PictureBox (this was puzzling). Drawing on the bitmap itself was, of course, no option as it corrupts the bitmap. Consequently, I opted to derive from the control class (the scrollablecontainer didn't work well either). I tried to include a horizontal and vertical scrollbar, but these wouldn't render on their own, and I found no easy way to paint them manually on the control. So, instead I implemented panning by dragging the middle mouse button, which becomes active when the image cannot fit into the custom control anymore.
Annotations are implemented using 3 extra classes: an Annotations top-level object, an annotation Container and an Annotation object. This last object is actually an abstract class, with the really instantiable classes being AnnotationLoose, AnnotationBorder and AnnotationRegion. I stayed away from the GraphicsPath object because it didn't fit my purpose very well, although it is used sometimes internally when computing regions. Each AnnotatedImage image control has 1 Annotations object, that can hold several Containers, which each can hold several Annotation objects. These can be of different subtypes. Note that these objects are not to be confused with vector graphic objects that are drawn on a background bitmap, rather they are conceptual and only represent certain divisions or regions in images. It is for this reason that these annotations have no intrinsic line thickness and are represented by a line of thickness of 1 pixel, regardless of the zoom factor of the image (I therefore did not use the coordinate transformations included in .NET, as they also transform line thicknesses and fonts. This is also one of the reasons why GraphicsPath was no good)! The reason for using 3 objects instead of 2 is that in doing so, annotations can be grouped together logically. Moreover, it is possible to combine the annotations in a Container to create a region, based on the RegionAddMode property of the individual annotations. This makes it easy to create several complex regions, and to use these, e.g. for further image processing. To give an example, I use this at work to compute color differences for clinical images of the skin, with a 'normal skin' and a 'lesion' container. With skin lesions sometimes having a mottled appearance, i.e. with patches of normal skin inside of them, this would not have been possible with a 2 object approach.
An annotation can be added to an image by adding it to the annotation Container, after this Container has been added to the top-level object Annotations.
Dim objAnnSet As Annotations.ContainerCollection = _
AnnotatedImage.Annotations.Containers
Dim objAnnC1 As New Annotations.Container("Normal")
objAnnSet.Add(objAnnC1)
Dim objAnn As New Annotations.AnnotationBorder("Annotation 1", objColor)
objAnnC1.Annotations.Add(objAnn)
objAnn.PointsVisible = False
Dim objrect as new rectangle
...
objAnn.AddRectangle(objrect)
AnnotatedImage.Refresh()
In contrast to the first version, all collections are now type safe, and for those who are interested in how to do this, check out the following snippet:
Public Class ContainerCollection
Inherits NameObjectCollectionBase
Public Sub New()
End Sub
Default Public ReadOnly Property Item(ByVal key As String) As Container
Get
Return CType(Me.BaseGet(key), Container)
End Get
End Property
Default Public ReadOnly Property Item(ByVal index As Integer) As Container
Get
Return CType(Me.BaseGet(index), Container)
End Get
End Property
Public Sub Add(ByVal value As Container)
If Me.KeyExists(value.Key) Then Me.BaseRemove(value.Key)
Me.BaseAdd(value.Key, value)
End Sub
Public Function KeyExists(ByVal key As String) As Boolean
If Me.Item(key) Is Nothing Then
Return False
Else
Return True
End If
End Function
Public Overloads Sub Remove(ByVal key As String)
Me.BaseRemove(key)
End Sub
Public Overloads Sub Remove(ByVal index As Integer)
Me.BaseRemoveAt(index)
End Sub
Public Sub Clear()
Me.BaseClear()
End Sub
Public Shadows Function GetEnumerator() As ContainerCollectionEnumerator
Return New ContainerCollectionEnumerator(Me)
End Function
Public Class ContainerCollectionEnumerator
Implements IEnumerator
Private objContainerCollection As ContainerCollection
Private index As Integer = -1
Public Sub New(ByVal objContainerCollection As ContainerCollection)
Me.objContainerCollection = objContainerCollection
End Sub
Public ReadOnly Property Current() As Object Implements IEnumerator.Current
Get
Return CType(Me.objContainerCollection.Item(index), Container)
End Get
End Property
Public Function MoveNext() As Boolean Implements IEnumerator.MoveNext
If index < objContainerCollection.Count - 1 Then
index += 1
Return True
Else
Return False
End If
End Function
Public Sub Reset() Implements IEnumerator.Reset
index = -1
End Sub
End Class
End Class
In order to provide smooth redrawing, double buffering was activated on the AnnotatedImage control by settings some of its style properties:
setstyle(ControlStyles.UserPaint, True),
setstyle(ControlStyles.AllPaintingInWmPaint,
True),setstyle(ControlStyles.DoubleBuffer, True)
The important point here is that you must use the graphics object provided by the paint event of the AnnotatedImage in the drawing routines of the individual annotations. Indeed, with double buffering, this object is actually pointing to an in-memory buffer which is copied selectively to the client area of the control when drawing is finished. This also means it is not appropriate to try to invalidate limited areas of the control when redrawing annotations to speed up redrawing, because we would need to know the bounding box of the annotations to be drawn, before we draw them (and add them to the bounding box of the already drawn annotations). This is tedious and would mean a lot of extra code, with little or no gain as the double buffering already does part of this for us and is pretty fast.
Comments, to do, bugs
This component has been used in a user control specialized in sRGB image viewing, which in its turn is used in an application for measuring skin lesions using calibrated sRGB images. It can be found here. So far this application has been remarkably stable compared to a previous version using VB6 and leadtools 11.
To-do's include mainly serialization of annotations to file, probably using XML, and maybe the ability to lock annotations with passwords (important in medical environments).
