Check out the Contact Sheet II.jsx script that comes with PS. It has key filtering stuff in it. Ignore the stuff for handling the Enter key: it's there to validate the form before processing.
X is an extremely talented Photoshop scripter where I'm just a hacker that just hacks small scripts to do what I want done. I created a small script a while back to tile images onto a canvas. Data required is numeric. I don't know Photoshop ScriptUI and those that do are annoyed at Adobe for not fixing bugs they report. So I just steal their dialogs and smash them into what I want.and hopefully avoid the bugs. When I create the dialog it would accept any values. I wanted to allow only numeric data so after doing some searching I think if i remember correctly I found some code I believe Michael Hale posted to handle the keyboard. I added that code to my small script. You may have a easier time find our how to do what you want to do looking at a small script. Contact Sheet II.jsx is huge. How things fit may be harder to see in there. http://www.mouseprints.net/old/dpr/PasteImageRoll.jsx If the uses enter anything other the a number or . and some edit keys like backspace in a numeric files they receive a beep and the key is not accepted.