Click here to Skip to main content
65,938 articles
CodeProject is changing. Read more.
Articles
(untagged)

Selection Overlay DLL

0.00/5 (No votes)
18 Jun 2012 1  
API which shows a Selection Overlay and notifies the caller when it's resizing and notifies the final rectangle.

Introduction

This is an API which shows a Selection Overlay and notifies the caller when it's resizing and notifies the final rectangle. 

Background 

I'm a programmer for the Royal Dutch Navies. A few weeks back a colleague asked me if I knew a way to show a Selection Overlay (such as a transparent square as Windows Explorer uses in its listview when you select a file). The source provided with this example is the result of the above need. 

Using the code 

To be able to use the code, you need to include SelectionOverlay.h and link to SelectionOverlay.lib or late load the calls. Besides this u'll need to include windows.h before u include the h.

In VB or C#, you can just use the wrappers. (For convenience purposes, I've left a compiled compressed version of the DLL in the VB Test directory.) 

You fill the structure to call with with the necessary information, like this example taken from test.cpp, where filling of the Alpha Bitmap has been overriden (optional): 

...
OurInstance = new CallBackTest();
... 
CallBackTest::CallBackTest()
{
  BorderWidth = 2;
  CheckWidth = 1 + BorderWidth;
  ColorBorder = CreateAlphaColor(RGB(0, 0, 200), 200);
  ColorBackground = CreateAlphaColor(RGB(75, 128, 255), 50);
  ColorBackground2 = CreateAlphaColor(RGB(100, 128, 255), 75);
  ColorBackground3 = CreateAlphaColor(RGB(0, 64, 128), 63);
}

bool CallBackTest::OnFillAlpha(void* Tag, HBITMAP BitmapDIBHandle)
{
  bool Result = false;
  BITMAP Bitmap = {0};
  GetObject(BitmapDIBHandle, sizeof(BITMAP), &Bitmap);                          

  if ((Bitmap.bmWidthBytes * Bitmap.bmHeight) >= sizeof(long))                  
  {
    long* Pixels = (long*) Bitmap.bmBits;                                       

    if (Pixels)                                                                 
    {
      Result = true;                                                            

      long BorderRight = Bitmap.bmWidth - CheckWidth;                           
      long BorderBottom = Bitmap.bmHeight - CheckWidth;                         

      for (long Y = 0; Y < Bitmap.bmHeight; Y++)                                
      {
        for (long X = 0; X < Bitmap.bmWidth; X++)                               
        {
          long Index = ((Y * Bitmap.bmWidth) + X);                              

          if ((X < BorderWidth) || (X > BorderRight) ||
            (Y < BorderWidth) || (Y > BorderBottom))                            
          {
            Pixels[Index] = ColorBorder;                                        
          }
          else if (((Y % 8) == 1) || ((X % 8) == 1))                            
          {
            Pixels[Index] = ColorBackground3;                                   
          }
          else if ((X & 2 & Y) != 2)                                            
          {
            Pixels[Index] = ColorBackground;                                    
          }
          else
          {
            Pixels[Index] = ColorBackground2;                                   
          }
        }
      }
    }
  }

  return Result;
}

bool CallBackTest::OnFillRGB(void* Tag, HDC hDC, int Width, int Height)
{
//no need to be implemented, but this way u can place a breakpoint
  return false;
}

void CallBackTest::OnMove(void* Tag, RECT Overlay)
{
//no need to be implemented, but this way u can place a breakpoint
  int c =2;
}

void CallBackTest::OnEnd(void* Tag, RECT Overlay)
{
//just a breakpoint moment here atm, but implementation required
  int c =2;
} 

//... Start of the code on a mouse down
SELECTIONOVERLAYDATA_CLASS Temp = { 0 };
//... Needed even if u implement FillAlpha, in case user fails we fill it
Temp.AlphaLevel = 50;
Temp.BorderAlphaLevel = 200;

Temp.BorderWidth = 1;
//... Needed even if u implement FillAlpha, in case user fails we fill it
Temp.OverlayColor = RGB(75, 128, 255);
Temp.BorderColor = RGB(0, 0, 200);

Temp.Parent = hWnd;
Temp.Destination = (ISelectionOverlay*)OurInstance;

switch (message)
{
  case WM_LBUTTONDOWN:
    Temp.ButtonLead = LBUTTON;
    break;
  case WM_MBUTTONDOWN:
    Temp.ButtonLead = MBUTTON;
    break;
  case WM_RBUTTONDOWN:
    Temp.ButtonLead = RBUTTON;
    break;
  case WM_XBUTTONDOWN:

    switch ((wParam  & 0x0000FFFF))
    {
      case MK_XBUTTON1:
        Temp.ButtonLead = XBUTTON1;
        break;
      case MK_XBUTTON2:
        Temp.ButtonLead = XBUTTON2;
        break;
    }
    break;
}

SelectionOverlay(Temp); 

After this, the overlay will ask the caller if it wants to draw the bitmap itself (or on failure of drawing will ask the caller if it wants to draw the dc) then the overlay will notify the caller with Move data (created when the overlay has the need to resize) and at final the overlay will call the caller with End data (created when the mouse button goes up and the final rectangle is known). 

I will skip the SelectionOverlay.h in this document. If you want to see how the six different structures look, you can read this in the h. The difference in the structures is present to supply the widest range I could.

I will explain each parameter present in the structures to explain their use:

  • Parent - The parent of the overlay window (notice it's our parent, with creation but the overlay is no WS_CHILD), Mandatory.
  • OverlayColor - Color of the overlay (RGB), Mandatory.
  • BorderColor - Color of the border of the overlay (RGB), Mandatory.
  • BorderWidth - Width of the border in pixels, Mandatory.
  • AlphaLevel  - Alphalevel to blend the overlay with, Mandatory.
  • BorderAlphaLevel  - Alphalevel to blend the border of the overlay with, Mandatory.
  • Tag - Value which can be passed on in our most basic structures, value will be returned on the notifications to the listening window.
  • ButtonLead - Tells which button we monitor: LBUTTON, MBUTTON, RBUTTON, XBUTTON1, or XBUTTON2, if not supplied LBUTTON will be defaulted.
  • Listener - The window addressed to listen to our notifications of WM_ONMOVE and WM_ONEND, if not supplied Parent will be used instead.
  • ClientRect   - Supplies the screen rectangle the overlay must stay between (Used in the 3 RECT structures), Mandatory.
  • Destination   - Interface to tall back to (used in the CLASS structure), Mandatory.
  • Dummy - What it says, unused, used to align it with out base structure SELECTIONOVERLAYDATA (Used in the CB structures).
  • Move   - Callback OnMove (Used in the CB structures).
  • EndCall - Callback OnEnd (Used in the CB structures), Mandatory. 
  • FillAlpha - Callback OnFillAlpha (Used in the CB structures). 
  • FillRgb - Callback OnFillRgb (Used in the CB structures). 

Concluding:

The RECT structure with their corresponding calls, are ment for the cases u want to use other screen coords for your rectangle then the clientrect of the Parent. The CB structures with their corresponding calls supply the possibility to be called back on a standard call method. The CLASS structures with their corresponding calls supply the possibility to be called back on a class interface (where OnMove, OnFillAlpha and OnFillRgb has an implementation and OnEnd has not (is Pure)) 

Rules: 

Not much actually, though the API demands that the supplied mouse button is in down state on calling. And each mandatory value need to be supplied.

Points of Interest

Wrappers

I've supplied two wrappers, one for Microsoft Visual Basic 6.0 and one for Microsoft C# (v4.0). They both have support for normal and ClientRect and overriding the Filling functions.

Why VB? Pretty convenient reason actually, VB tends to crash on events which are not generated by it's own threads. This aided in getting the call close to threadsafe. If VB would stay alive, likely I would supply a bigger purpose then C/C++ alone. As result as long as the trigger for calling the api is coming forth out of the same thread as the window comes which we call back to, we would come back in the correct thread. Which means that we would not have a reason to Invoke in C#, or any corresponding technique in an other language. Last of course only valid if the trigger and it's result are only used upon windows in the same thread. Thou be warned, the filling of a bitmap/dc comes forth out of it's own thread, though VB6 as the VB test example shows, can handle it in this way (thanx to cSuperclass) 

For the window procedure hook in VB6 I used: 

Fastest, safest subclasser, no module! 

Size: 

These days most things aren't about size anymore, though i prefer still to keep things as small as I could make them. To accomplish a smaller size i used a library coming out of an old MSDN
(Mine version comes from MSDN October 2001 (The last MSDN with full information for Microsoft Visual Studio 6.0)) called LIBCTINY.LIB, it's technique is very well explained in it's document:
MSDN October 2001, Under The Hood: Reduce EXE and DLL Size with LIBCTINY.LIB. Further more i've compressed the release version of the DLL even more with help of a splendid compressor tool, called UPX

32Bit:

As I've no 64 bit machine nor has access to one, I've only designed this API to work with 32 bit. LIBCTINY itself, I expect not to work with 64 bit, but also calls as SetWindowLong should be replaced in use with a 64 bit machine, if anyone is willing to convert this API to 64 bit, I would love to add it to this project.

Late loading:

I wanted to narrow down the dependencies of this API, to get the widest support i could reach in 32 bit Windows. All calls to the kernel are linked, all calls to User32 and GDI32 are late loaded (making use of loadlibrary and getprocadress). As result the DLL has four DLLs it depends on. Kernel, NTDLL, User32 and GDI32 (also one of the results of using LIBCTINY).

How it works::

Threads, keeping them alive and cross bridges:

Because I wanted the call to be asynchronous (and with that not blocking the caller after calling us), I needed a second thread. A thread where I could receive data from the low level mouse hook. Problem with a low level mouse hook is that you've only a certain amount of time to process, if u take to long, you'll be kicked out of the hook chain. To make sure i keep the overlay alive i needed to move all drawing to another thread.

So in short we've three threads: the caller, the mousehook, our overlay. The caller is kept alive on it's own, so we had no need to keep the thing alive, though our own 2 threads we did had the need for. I used a technique i hadn't used in ages, coming forth out of the beginning of programming windows in plain C. 

To keep the thread alive i used GetMessage. As most will see, I only check it's result on 0 and don't translate or dispatch. Why I don't use it is simple, i don't want to process errors, just go on and pray
was my thought. I don't translate or dispatch cause my input is a mousehook and my own generated messages.

Now i still needed to make a bridge between the threads. The solution for this also came back to old times, SendNotifyMessage, Which comes back with result from the call if it's in your own thread, but comes back without waiting for a result if it adds an message to a window procedure in another thread. In order to be able to receive I needed windows to notify to.

I used the HWND_MESSAGE on parent to get message windows.

CreateWindowEx(WS_EX_NOPARENTNOTIFY, "STATIC", NULL, 0, -1, -1, 0, 0, HWND_MESSAGE, NULL, 0, 0); 
Which are light weighted in comparison with normal windows. I didn't needed to show them anyways.

Colors:

I wanted the user to be able to supply RGB and the alphalevel for that parts of the overlay, in order to allow that, i needed to convert the RGB to alphacolor. After reading Vorlath - Project V: Advanced Alpha Blending 

I knew what the alphacolor actually was, and what i needed to do to create a source alpha. I took the 100% proven method which did not use dividing, LIBCTINY does not supply it, and with this solution I could keep using LIBCTINY. 

The overlay:

The overlay is a plain old static window with these specifics: WS_EX_NOPARENTNOTIFY, WS_EX_LAYERED, WS_EX_TRANSPARENT, WS_POPUP, and WS_VISIBLE.

  • WS_EX_NOPARENTNOTIFY: we don't want our parent to get notifications coming forth from our window.
  • WS_EX_LAYERED: We need this to be able to alphablend.
  • WS_EX_TRANSPARENT: We don't want to obscure the windows we're above. With this flag we're assured all beneath us has done it's drawing when we receive WM_PAINT.
  • WS_POPUP: our window acts as popup.
  • WS_VISIBLE: we want to be seen.

The alpha blending: 

We create a 32 bit bitmap which we fill with our alphablended colors. If we cannot create a 32 bit bitmap, we fill the dc of the screen with 2 fillrect commands. After this Updatelayeredwindow is called with correct flags. After being done all created will be destroyed again.

Caching:

To make sure i would get as quick as I could think off, I decided to cache as much as I could, and to use casting for data which aligned correctly, but actually request another structure;

(POINT*)&crPos, (SIZE*)&Info.bmiHeader.biWidth

are both examples of this. 

In Short: 

We create necessary colors and brushes. We create a thread to listen to the low level mouse hook. We create a thread to update a layered window, which also notifies the caller with Move and End information. 

To keep our own threads alive we use a window and GetMessage.

Beneath u'll see how the examples from the source (for C# and VB) use the dll, both override FillAlpha, done this so everyone can read how to coop with the pointer in each language, thou ofcourse the drawing of the dll is still present so u don't need to override the filling, it's optional. 

C# Example:   

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.Runtime.InteropServices;
using SelectionOverlayWrap;

namespace SelectionOverlayTest
{
    /// <summary>
    /// We want to get OnMove and OnEnd notifications, so implement both
    /// </summary>
    public partial class Form1: Form, ISelectionOverlayOnEnd, ISelectionOverlayOnMove, ISelectionOverlayOnFillAlpha, ISelectionOverlayOnFillRGB
    {
        [StructLayout(LayoutKind.Sequential)]
        private struct BITMAP
        {
            internal int bmType;
            internal int bmWidth;
            internal int bmHeight;
            internal int bmWidthBytes;
            internal Int16 bmPlanes;
            internal Int16 bmBitsPixel;
            internal IntPtr bmBits;
        }

        [DllImport("gdi32.dll", EntryPoint = "GetObjectW")]
        private static extern int GetObject(IntPtr hObject, int nCount, ref BITMAP lpObject);

        private static int BorderWidth = 2;
        private static int CheckWidth = 1 + BorderWidth;
        private static int ColorBorder = SelectionOverlay.CreateAlphaColor(SelectionOverlay.RGB(0, 0, 200), 200);
        private static int ColorBackground = SelectionOverlay.CreateAlphaColor(SelectionOverlay.RGB(75, 128, 255), 50);
        private static int ColorBackground2 = SelectionOverlay.CreateAlphaColor(SelectionOverlay.RGB(100, 128, 255), 75);
        private static int ColorBackground3 = SelectionOverlay.CreateAlphaColor(SelectionOverlay.RGB(0, 64, 128), 63);

        /// <summary>
        /// Will be called on OnMove notification
        /// </summary>
        /// <param name="Tag">ignored</param>
        /// <param name="Rectangle">The square of the overlay</param>
        public void OnMove(object Tag, Rectangle Rectangle) 
        {
            button1.Text = Rectangle.Left.ToString() + ", " +
                           Rectangle.Top.ToString() + ", " +
                           Rectangle.Right.ToString() + ", " +
                           Rectangle.Bottom.ToString();
        }

        /// <summary>
        /// Will be called on OnEnd notification
        /// </summary>
        /// <param name="Tag">ignored</param>
        /// <param name="Rectangle">The square of the overlay</param>
        public void OnEnd(object Tag, Rectangle Rectangle)
        {
            button2.Text = Rectangle.Left.ToString() + ", " +
                           Rectangle.Top.ToString() + ", " +
                           Rectangle.Right.ToString() + ", " +
                           Rectangle.Bottom.ToString();
        }

        public bool OnFillRGB(object Tag, IntPtr hDC, int Width, int Height) 
        {
            return false;
        }

        public bool OnFillAlpha(object Tag, IntPtr BitmapDIBHandle) 
        {
            bool Result = false;

            if (BitmapDIBHandle != IntPtr.Zero) 
            {
                BITMAP Retriever = new BITMAP();
                GetObject(BitmapDIBHandle, Marshal.SizeOf(Retriever), ref Retriever);

                int lSize = ((Retriever.bmWidthBytes * Retriever.bmHeight)/4);

                if ((lSize >= 4) && (Retriever.bmBits != IntPtr.Zero))
                {
                    int[] lPixels = new int[lSize];
                    try
                    {
                        Marshal.Copy(Retriever.bmBits, lPixels, 0, lSize);
                        Result = true;
                    }
                    catch { }

                    if (Result)
                    {
                        long BorderRight = Retriever.bmWidth - CheckWidth;
                        long BorderBottom = Retriever.bmHeight - CheckWidth;

                        for (long Y = 0; Y < Retriever.bmHeight; Y++)
                        {
                            for (long X = 0; X < Retriever.bmWidth; X++)
                            {
                                long Index = ((Y * Retriever.bmWidth) + X);

                                if ((X < BorderWidth) || (X > BorderRight) ||
                                    (Y < BorderWidth) || (Y > BorderBottom))
                                {
                                    lPixels[Index] = ColorBorder;
                                }
                                else if (((Y % 8) == 1) || ((X % 8) == 1))
                                {
                                    lPixels[Index] = ColorBackground3;
                                }
                                else if ((X & 2 & Y) != 2)
                                {
                                    lPixels[Index] = ColorBackground;
                                }
                                else
                                {
                                    lPixels[Index] = ColorBackground2;
                                }
                            }
                        }

                        Result = false;
                        try
                        {
                            Marshal.Copy(lPixels, 0, Retriever.bmBits, lSize);
                            Result = true;
                        }
                        catch { }
                    }
                }
            }

            return Result;
        }

        /// <summary>
        /// Starts us
        /// </summary>
        public Form1()
        {
            InitializeComponent();
        }

        /// <summary>
        /// On mousedown checks which button was fired and if it's an supported one
        /// it opens a selection overlay for that button
        /// </summary>
        /// <param name="sender">Whisperer</param>
        /// <param name="e">Mouse data</param>
        private void Form1_MouseDown(object sender, MouseEventArgs e)
        {
            bool NoGo = false;
            VKKeys ToUse = VKKeys.LBUTTON;

            if ((e.Button & System.Windows.Forms.MouseButtons.Left) == System.Windows.Forms.MouseButtons.Left)
            {
                ToUse = VKKeys.LBUTTON;
            }
            else if ((e.Button & System.Windows.Forms.MouseButtons.Right) == System.Windows.Forms.MouseButtons.Right)
            {
                ToUse = VKKeys.RBUTTON;
            }
            else if ((e.Button & System.Windows.Forms.MouseButtons.Middle) == System.Windows.Forms.MouseButtons.Middle)
            {
                ToUse = VKKeys.MBUTTON;
            }
            else if ((e.Button & System.Windows.Forms.MouseButtons.XButton1) == System.Windows.Forms.MouseButtons.XButton1)
            {
                ToUse = VKKeys.XBUTTON1;
            }
            else if ((e.Button & System.Windows.Forms.MouseButtons.XButton2) == System.Windows.Forms.MouseButtons.XButton2)
            {
                ToUse = VKKeys.XBUTTON2;
            }
            else 
            {
                NoGo = true;
            }

            if (!NoGo)
            {
                SelectionOverlay.Show(Color.FromArgb(50, 75, 128, 255),
                                      Color.FromArgb(200, 0, 0, 200),
                                      1,
                                      this,
                                      this.Handle,
                                      ToUse);
            }
        }
    }
} 

VB Example:  

Option Explicit

'Only need to implement the functions u want to use (IOnEnd is mandatory though)
Implements IOnEnd
Implements IOnMove
Implements IOnFillAlpha
Implements IOnFillRGB

Dim blaat   As Double
Dim Client  As New VBRect

Private Type BITMAP
    bmType As Long
    bmWidth As Long
    bmHeight As Long
    bmWidthBytes As Long
    bmPlanes As Integer
    bmBitsPixel As Integer
    bmBits As Long
End Type

Dim BorderWidth         As Long
Dim CheckWidth          As Long
Dim ColorBorder         As Long
Dim ColorBackground     As Long
Dim ColorBackground2    As Long
Dim ColorBackground3    As Long

Private Declare Function GetObject Lib "gdi32.dll" Alias "GetObjectA" (ByVal hObject As Long, ByVal nCount As Long, ByVal lpObject As Long) As Long
Private Declare Sub RtlMoveMemory Lib "kernel32.dll" (ByRef Destination As Any, ByRef Source As Any, ByVal Length As Long)

Private Sub Form_Load()
    BorderWidth = 2
    CheckWidth = 1 + BorderWidth
    ColorBorder = CreateAlphaColor(RGB(0, 0, 200), 200)
    ColorBackground = CreateAlphaColor(RGB(75, 128, 255), 50)
    ColorBackground2 = CreateAlphaColor(RGB(100, 128, 255), 75)
    ColorBackground3 = CreateAlphaColor(RGB(0, 64, 128), 63)
End Sub

Private Sub Form_MouseDown(Button As Integer, Shift As Integer, X As Single, Y As Single)
   ShowOverlay Me.hWnd, 50, 200, 1, RGB(75, 128, 255), RGB(0, 0, 200), Me
'   Client.Left = 25
'   Client.Top = 25
'   Client.Right = 400
'   Client.Bottom = 300
'   ShowOverlay Me.hWnd, 50, 200, 1, RGB(75, 128, 255), RGB(0, 0, 200), Me, , Client
End Sub

Private Sub IOnEnd_OnEnd(ByVal Tag As Object, ByVal Left As Long, ByVal Top As Long, ByVal Right As Long, ByVal Bottom As Long)
    Print Left & ", " & Top & ", " & Right & ", " & Bottom
End Sub

Private Function IOnFillAlpha_OnFillAlpha(ByVal Tag As Object, ByVal hBitmapHandle As Long) As Boolean
    Dim bResult As Boolean
    Dim oBitmap As BITMAP
    Dim lSize As Long
    
    bResult = False
    
    Call GetObject(hBitmapHandle, Len(oBitmap), VarPtr(oBitmap))
    
    lSize = oBitmap.bmWidth * oBitmap.bmHeight
    
    If ((lSize >= 4) And (oBitmap.bmBits <> 0)) Then
        bResult = True
        Dim lPixels()       As Long
        Dim BorderRight     As Long
        Dim BorderBottom    As Long
        Dim X               As Long
        Dim Y               As Long
        Dim LoopHeight      As Long
        Dim LoopWidth       As Long
        Dim Index           As Long
        
        LoopHeight = oBitmap.bmHeight - 1
        LoopWidth = oBitmap.bmWidth - 1
        
        BorderRight = oBitmap.bmWidth - CheckWidth                                         
        BorderBottom = oBitmap.bmHeight - CheckWidth                                       
        
        ReDim lPixels(lSize)
        RtlMoveMemory lPixels(0), ByVal oBitmap.bmBits, lSize * 4
        
        For Y = 0 To LoopHeight
        For X = 0 To LoopWidth
            Index = (Y * oBitmap.bmWidth) + X
            
            If ((X < BorderWidth) Or (X > BorderRight) Or _
                (Y < BorderWidth) Or (Y > BorderBottom)) Then                               
                lPixels(Index) = ColorBorder                                                
            ElseIf (((Y Mod 8) = 1) Or ((X Mod 8) = 1)) Then
                lPixels(Index) = ColorBackground3                                           
            ElseIf ((X And 2 And Y) <> 2) Then                                              
                lPixels(Index) = ColorBackground                                            
            Else
                lPixels(Index) = ColorBackground2                                           
            End If
        Next X, Y
        RtlMoveMemory ByVal oBitmap.bmBits, lPixels(0), lSize * 4
    End If
    
    IOnFillAlpha_OnFillAlpha = bResult
End Function

Public Function IOnFillRGB_OnFillRGB(ByVal Tag As Object, ByVal hDC As Long, ByVal Width As Long, ByVal Height As Long) As Boolean
    IOnFillRGB_OnFillRGB = False
End Function

Private Sub IOnMove_OnMove(ByVal Tag As Object, ByVal Left As Long, ByVal Top As Long, ByVal Right As Long, ByVal Bottom As Long)
    Command1.Caption = Left & ", " & Top & ", " & Right & ", " & Bottom
End Sub

History   

rev 9: Expanded the DLL with callback to overwrite filling of the bitmap/dc, updated the C# and VB wrapper, adjusted test.cpp from the C++ test.exe, adjusted this page. Added CreateAlphaColor to the DLL. Restyled VB Wrapper to have only 1 SelectionOverlay function, rect has become an optional.  

rev 7: Bugfix in the cpp. _stdcall was missing on the two callbacks used in the class variant of the method. Somehow this bug slipped through, cause for a reason i don't understand, calling a _cdecl from out debug with an _stdcall prototype did not crash, while release ofcourse does. A well found, solved. 

rev 5: Adjusted some grammar. And a missing hint about windows.h  

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here