The one thing that raises a red flag for me is the .264 extension.
.264 files work well in En if they adhere to the BD spec. I've used .264 files from both MeGUI and x264Pro with no issues whatsoever.
I don't think it's the file extension.
Thanks to reply.
Whitout Target bitrate (from BD Preset bitrate 20Mb/s to custom bitrate, 4.5Mb/s) , I didn't change.
Encoder setting are below.
Bitrate buffer: 3MB
VBR buffer initial: 10%
VBR buffer target: 100%
Max I-frame, B-frame, P-frame, Br-frame size: Auto
I-frame interval = 30
Min. I-frame interval = 1
Mark every 1 I-frame as IDR
Fixed I-frame distance: disable
Adaptive B-flame placement: enable
Subpixel search depth: 1/4
Reference frames: 4
Serch range: 99
Weighted prediction for p-frames: enable
Fast intra decision and inter decision: enable
Simplified RD optimaization: enable
Deblocking filter: CABAC
I think this setting is much to Blu-ray standerd.