It's not possible with an unaltered OpenFileDialog box - it expects an extension, and windows removes the "." from files end with it, so "*." will never work.
You could probably do it, if you can hook the standard dialog (see here:
http://msdn.microsoft.com/en-us/library/ms646960(VS.85).aspx[
^], about 1/3 the way down, under filters) but that's a lot of work in C# for a small feature!
Is there any good reason why your Hex files have no extension? It would be a lot easier to give them a name like *.MyAppHex" or similar then filter for them.
If you really do have to do this, I would be tempted to not use the standard box at all, but re-create it's features myself - it's probably a lot easier than implementing the hooking on a standard dialog!