You will upgrade your LR3 catalog to LR5. This should happen automatically the first time you run Lightroom 5.
But you won't be able to reopen the catalog in LR3, if that matters.
To maybe slightly clarify (and hopefully NOT confuse) what DJ and Michael have correctly stated, when you open your LR 3 catalog in LR5, it will let you know it's upgrading the catalog and give it a default new name which you can change if you wish. This new upgraded catalog will not be openable by LR3 but your existing LR3 catalog still can be opened by LR3.
If unsure about LR5 then download the demo first. Any Catalogs created by the demo will be usable by the paid for version (the demo version is the same, just time limited).