Both eps and jpg doers not support transparency so they are out. If this will be for the web, you are going to have to use png.
As for removing the white, there are several ways to do so.
First you need to unlock the background layer by clicking on the padlock or dragging the padlock to the trash can.
Next you need to select the white using one of the selection tools such as the magic wand or quick selection tool.
With the white selected you can 1)hit delete key to delete the white. 2)create a mask to hide the white 3) invert the selection and move the selected object(s) to a new layer.
The mask is the only option that is non-destructive. In that you can at any time in the future change your mind.
Now if you use layers, masks, transparencies, etc. you need to use a format that supports them. Png only supports transparency, where as psd and tiff support all or majority of the features of photoshop. Psd is the default format as it does support all features.