Custom Context Menu

Introduction

The following tip shows how to create a window that will behave like a context menu. The goal is to create such custom context menus that can hold any kind of child controls to enhance UI in general.

It is the result of some brainstorming that happened in a question I asked. I would like to thank all the contributors that helped in finding the solution.

Link to the CP question

Background

This tip requires that you are familiar with pure Win32 API, window hooks and WNDPROCs in general.

Using the Code

I am not going to explain in detail how to register and create the custom menu window, it conforms to basic window class registration.

It's worth noting here that the created window is not visible and you need to pay attention to the styles used (although they can be changed later through SetWindowLong).

More information about these styles is available on MSDN.

Getting Started

Create the custom menu like displayed below:

HWND hMenuWnd = CreateWindowEx(WS_EX_TOPMOST | WS_EX_NOACTIVATE, 
"MyCustomMenu", NULL, WS_POPUP | WS_BORDER, 0, 0, 1, 1, hMainWindow, 0, hInstance, 0);

Notice the WS_EX_NOACTIVATE style which instructs Windows to skip window activation that normally happens when a new top level window is shown. Later on, the menu window will be positioned and shown by its owner as a result from clicking a button. When calling SetWindowPos to set the menu in the correct location, make sure you are adding SWP_NOACTIVATE flag, otherwise this function will remove WS_EX_NOACTIVATE from the window.

This is a simple usage example:

case IDC_MYMENUBUTTON:
   SetWindowPos(hMenuWnd, 0, x, y, wdith, height, SWP_NOZORDER | SWP_NOACTIVATE);
   ShowWindow(hMenuWnd, SW_SHOWNOACTIVATE);

   break;

Again, to keep the menu window from being activated, you need to handle WM_MOUSEACTIVATE message in the menu window procedure and return MA_NOACTIVATE.

case WM_MOUSEACTIVATE:
    return MA_NOACTIVATE;

Now that the window is ready to become visible, you will need to monitor mouse clicks especially those that go outside its area to cause the menu to hide. This will be achieved by setting a local mouse hook (will only watch the current thread message queue) that will catch mouse clicks and decide if it's time to hide the menu. Also, you will want this hook to be active only when the menu is visible (no need to filter messages when no action is required) so for this, you will handle WM_WINDOWPOSCHANGED message like so:

case WM_WINDOWPOSCHANGED:
    WINDOWPOS* wp = (WINDOWPOS*) lParam;
    
    // if the window is shown
    if (wp->flags & SWP_SHOWWINDOW)
    {
        // set our hook and store the handle in the global variable
        g_hMouseHook = SetWindowsHookEx(WH_MOUSE, MyMenuMouseHook, 0, GetCurrentThreadId());
        
        if (!g_hMouseHook)
        {
            // handle your error!
        }
     } else if (wp->flags & SWP_HIDEWINDOW) // the menu has been hidden
     {
        UnhookWindowsHookEx(g_hMouseHook); // unhook
     }

     // pass on the message to DefWindowProc
     return DefWindowProc(hWnd, Msg, wParam, lParam); 

Now the hook procedure that intercepts all mouse button actions (down, up, etc.) and decides when the menu should be hidden.

LRESULT CALLBACK MyMenuMouseHook(int Code, WPARAM wParam, LPARAM lParam)
{
    // messages are defined in a linear way the first being WM_LBUTTONUP up to WM_MBUTTONDBLCLK
    // this subset does not include WM_MOUSEMOVE, WM_MOUSEWHEEL and a few others
    if (wParam >= WM_LBUTTONUP && wParam <= WM_MBUTTONDBLCLK)
    {
        // it's a fair assumption to say that you should have only one menu displayed 
        // at a certain point in the owner window
        HWND hMenuWnd = FindWindow(L"MyCustomMenu", 0);


        if (hMenuWnd)
        {
            POINT pt; RECT rcWindow;        
            GetCursorPos(&pt);
            GetWindowRect(hMenuWnd, &rcWindow);
            
            // if the mouse action is outside the menu, hide it. the window procedure 
            // will also unset this hook 
            if (!PtInRect(&rcWindow, pt))
                ShowWindow(hMenuWnd, SW_HIDE);
        }   
    }

    return CallNextHookEx(NULL, Code, wParam, lParam);
}

There are mouse actions that the hook will not intercept such as clicks outside the owner window as they are not posted on the hook thread queue. Handle WM_KILLFOCUS message in the owner procedure and check if the menu is visible, then hide it.

case WM_KILLFOCUS:
    if (IsWindowVisible(hMenuWnd))
        ShowWindow(hMenuWnd, SW_HIDE);
    
    break;

You will also have to hide the menu yourself when one of its internal controls are clicked.

I am open to suggestions if you have an alternative solutions/discovered any problems/questions, please use the comments section to express yourself.

Points of Interest

• MSDN reference for Window Styles
• MSDN reference for Extended Windows Styles
• MSDN Windows Hooks Overview

History

• 29^th March, 2014 - 1^st version of the document posted