Does the printer also have a choice of "tabloid" in addition to 11 x 17? I'vegot an older Xerox printer and it's pretty strange what happens when you try certain paper sizes.
That said, have you tried setting the size inthe driver, rather than inthe print dialog, then setting the dialog to "Defined by Driver"? Sometimes that helps. This kind of problem seems prevalent on Macs with 10.6. up to 5. The 10.6.6 update is supposed to have fixed a lot of printing issues, if that applies to your situation.
Another work around that you may want to try is to export to PDF, then print from Acrobat or Reader.
I thought I had this problem fixed as of August 2010, but in the last couple of weeks/months I have been proven wrong. The Xerox Phaser 7760 in our office continues to print incorrect orientations (never consistent among users), will not select proper page size (whether "Choose paper size by PDF page size" is selected or not), and will not scale pages properly.
Does not make a difference whether a user is Acrobat 9 Pro, 8, etc.
Systems/Users: Windows XP SP3 (multiple), Windows 7 Pro 32bit (multiple) on a domain.
Documents: Multiple, 8.5x11 pages mixed with 11x17's at the end of the document which do not orient or switch to 11x17 automatically - which at one point these types of compiled PDFs DID print correctly. Still searching....
Windows XP (Pro 32bit SP3) fix that I'm testing:
Printers and Faxes -->
Phaser 7760 --> Right click, Properties -->
Advanced tab -- Print Processor.
I changed mine from "Winprint (Default Data Type RAW)" to "XeroxV5Print (Default Data Type RAW)".
This was a guess. The result allowed me to print pages that were landscape orientation in a PDF without having to manually rotate them before sending to print. I tested this with 11x17 pages as well, yet it did not work. After wasting a few pages, thinking I'd get something correct, I'm not testing any more until I try the first fix with 8.5x11's on a Windows 7 machine.
I tested a PDF (8.5x11) by rotating some pages in the PDF itself 90 degrees, printing double sided, and telling Adobe to "auto rotate and center". Please note the 'auto rotate and center' was part of the problem originally concerning my scaling/rotating issue also reported by the OP with this printer.
Will update when I have tested it on Windows 7 (Pro 32bit, most of them SP1)....