you can use the navigation box in the lower left corner of the document window instead, which allows you to pick a number...
At least then the pages panel matches the page showing on the pasteboard, but I really wanted to use the pages panel for navigation amoung pages.
Any more ideas?
Learn to double-click.
Page that you're currently on is highlighted in reverse (black background white text)
If you're select a page in the pages panel it highlights blue to indicate it's selected. This allows you to move the page if you need to, or select over dozen other options for that page, numbering, sections, override, move, duplicate, apply master page to, delete, rotate view, colour label, page transiitions, hide master items and spread flattening!
You double click a page to move to that page. If it was only a single click then it would be quite frustrating to be hopping all over the document when all you want to do is renumber/resection/dupe or over a dozen different options mentioned above.
CTRL/CMD J will bring up the Go To Page dialog box.
Navigate with the page numbers as peter suggested.
There's a reason it works like that - because it works best.