The total file size is not really relvant.
The important part is the per frame memory usage.
Which OS are you using? RAM starvation can be worse in Win XP 32, since the total available RAM for all applications combined is 2.7 GB.
Try this - render the Comp to a lossless format (such as Quicktime Animation, huge file but you can delete it afterwards), and then close AE and launch Adobe Media Encoder in standalone mode and try to encode using your Blu-Ray preset from AME.
Im using windows 32 bit and yeah the media encoder thing worked now its finally able to render in AAE thanks so much!
Try to loose Win XP 32 whenever it's possible. Since the AME encoding processes are independent from AE, it seems your system can't provide enough RAM for it all. Either because of lack of physical RAM, or because of the XP 32 limitations.
im getting a new pc soon with vista 64 bit so i should be good then