I've bought the glyphish pack. I've opened the AI files in AI (CS5) and resized some of the icons down to 16px. They look fine in AI but by the time I save for web as PNG, I view them in the web app, and they look quite soft. Particularly compared to some of the more slick looking web apps out there.
How do I go about keep them as sharp as possible?
Check your document resolution. It needs to be 72 dpi. Also enable the pixel grid.
Of course they look fine in Illustrator, because on screen they will always be rendered sharply according to your zoom level (unless you use pixel preview as Mylenium already noted). If you want them to be sharp, you need to design them according to the pixel grid and maybe optimize them using a raster image editor.