Encore creates a bluray image with an iso extension. I wonder if you have windows explorer set to show no extension.
You may not have "iso" associated with a program on your computer.
Also, you cannot burn the "file" to a BD and have it play. You need a program to take the image and burn it in the correct bluray format onto the disk.
Most of us love imgburn: