Is it normal for the pen tool to produce pixelated jagged edges?
When ever I use the pen too and create curves the edges look very pixelated and cheep. I tried illustrator and that seemed to produce nice smooth edges like i'd expect.
I always though fireworks was a vector based tool and that vectors = smooth.
Is there a setting or something that I can change to make the drawings smooth?
I'm using Fireworks CS5 on windows 7.
First, check your Zoom level and set it to 100%. For whatever reason, Fireworks often opens a document at less than 100% zoom, and this creates very bad looking results on the screen. You can find the Zoom level in the lower right corner of the canvas or go to View > Magnification. You can also use the shortcut Command-1 to set it to 100%, or Command-minus or Command-plus to decrease or increase zoom.
Second, realize that Fireworks is basically always in Pixel Preview mode. It's committed to showing you the pixels that will be rendered upon export as a bitmap. This is different from Illustrator, which was designed for print, first and foremost; Illustrator is not committed to a single output format and instead tries to render vectors as smoothly as possible on screen, at all zoom levels.
Third, check your Stroke settings in the Properties panel. This is not likely the problem, as the default is 1-Pixel Soft, but it's worth checking. If you're working with strokes larger than 2px, you can adjust their softness somewhat using the Edge slider. You can try different Stroke presets or experiment with their parameters using the Advanced options (available under Stroke Options, at the bottom of the Stroke presets dropdown menu).
Hope this helps! I do think it's a shame that Fireworks doesn't offer better rendering at zoom levels other than 100%. I've written about this before—it basically uses something like a Nearest Neighbor interpolation algorithm to render zoom instead of a smoother-looking Bicubic algorithm—and submitted a feature request to Adobe regarding the issue, but haven't heard anything.
Unfortunately it is at 100%, 1px soft.
This could be a stupid question, but my trial of Illustrator has run out so I cant test this. If I did create a vector like the ones above in illustrator and export them to a gif would they be just as jagged?
No, vectors exported from Illustrator should not be jagged. You can also preview their appearance using View > Pixel Preview.
But vectors in Fireworks are not typically as jagged as in your example, either. Looking at your sample more closely, the vector on the left clearly has a hard, non-anti-aliased stroke applied. (It appears to have been drawn using a stroke preset such as Pencil > 1-Pixel Hard, set to a 2-pixel stroke weight.) The vector on the right seems to have some anti-aliasing applied, but also has an unusually jagged quality; I'm not sure what's going on there—that's what made me think it was being viewed at a smaller zoom percentage.
Are you getting the same rough-looking vector strokes when using the Shape or Line tools?