Lightroom 3 and Lightroom 5 use different catalogs. You are using the wrong desktop shortcut to start Lightroom 5.
First - Jim... Mac does not have desktop shortcuts, that is a Windows thing.
OP - your Dock icon will still be pointing to Lr3 so you need to go to the Applications folder and drag the Lightroom 5 app icon into the Dock. It is up to you whether you keep the Lr3 one in the dock or not. To remove an icon from the Dock, just drag it out.
You can always open Lr5 by double clicking the app in the Applications Folder.
You can create a LR5 desktop shortcut (Alias) on Mac.
But launch LR5 from Finder >> Applications
Or directly from Launch Pad
Then click the top menu in the Library
File >> Open Catalog
Navigate to your named catalog file.