First I would like to thank everybody that has contributed with ideas to this question, here's the solution I found to fit best my needs.
For those jumping straight to the solution, I am displaying a custom context menu that is hosted by a popup window. The point is to have the menu window not interfere with activation and focus of the owner window (short, make it behave like a context menu).
The window is created like so:
HWND hMenuWnd = CreateWindowEx(WS_EX_TOPMOST | WS_EX_NOACTIVATE, "MyCustomMenu", NULL, WS_POPUP | WS_BORDER, 0, 0, 1, 1, hMainWindow, 0, hInstance, 0);
Notice the
WS_EX_NOACTIVATE style which instructs Windows to skip window activation that normally happens when a new top level window is shown.
Later on the menu window will be positioned and shown by it's owner as a result from clicking a button. When calling
SetWindowPos to set the menu in the correct location make sure you are adding
SWP_NOACTIVATE flag otherwise this function will remove
WS_EX_NOACTIVATE from the window.
This is a simple usage example:
case IDC_MYMENUBUTTON:
SetWindowPos(hMenuWnd, 0, x, y, wdith, height, SWP_NOZORDER | SWP_NOACTIVATE);
ShowWindow(hMenuWnd, SW_SHOWNOACTIVATE);
break;
Again, to keep the menu window from being activated you need to handle
WM_MOUSEACTIVATE message in the menu window procedure and return
MA_NOACTIVATE.
case WM_MOUSEACTIVATE:
return MA_NOACTIVATE;
Now that the window is ready to become visible, you will need to monitor mouse clicks especially those that go outside it's area to cause the menu to hide. This will be achieved by setting a local mouse hook (will only watch the current thread message queue) that will catch mouse clicks and decide if it's time to hide the menu.
Also, you will want this hook to be active only when the menu is visible (no need to filter messages when no action is required) so for this you will handle
WM_WINDOWPOSCHANGED message like so:
case WM_WINDOWPOSCHANGED:
WINDOWPOS* wp = (WINDOWPOS*) lParam;
if (wp->flags & SWP_SHOWWINDOW)
{
g_hMouseHook = SetWindowsHookEx(WH_MOUSE, MyMenuMouseHook, 0, GetCurrentThreadId());
if (!g_hMouseHook)
{
}
} else if (wp->flags & SWP_HIDEWINDOW) {
UnhookWindowsHookEx(g_hMouseHook); }
return DefWindowProc(hWnd, Msg, wParam, lParam);
Now the hook procedure that intercepts all mouse button actions (down, up, etc) and decides when the menu should be hidden.
LRESULT CALLBACK MyMenuMouseHook(int Code, WPARAM wParam, LPARAM lParam)
{
if (wParam >= WM_LBUTTONUP && wParam <= WM_MBUTTONDBLCLK)
{
HWND hMenuWnd = FindWindow(L"MyCustomMenu", 0);
if (hMenuWnd)
{
POINT pt; RECT rcWindow;
GetCursorPos(&pt);
GetWindowRect(hMenuWnd, &rcWindow);
if (!PtInRect(&rcWindow, pt))
ShowWindow(hMenuWnd, SW_HIDE);
}
}
return CallNextHookEx(NULL, Code, wParam, lParam);
}
Later Edit: There are mouse action that the hook will not intercept such as clicks outside the owner window as they are not posted on the hook thread queue. Handle
WM_KILLFOCUS message in the owner procedure and check if the menu is visible then hide it.
case WM_KILLFOCUS:
if (IsWindowVisible(hMenuWnd))
ShowWindow(hMenuWnd, SW_HIDE);
break;
You will also have to hide the menu yourself when one of it's internal controls are clicked.
-Alex