If you're reading a book and you see the arrows then yes, there should be a gray thumb with I think three horizontal white bars in there somewhere.
Now, if you're in 'fit single page' and you can see the whole book you'll see the arrows, but the thumb then is a small gray bar underneath the 'up' arrow. It's a bit hard to see. The cursor should turn to a hand when you're over the thumb.