Upgrading to Snow leopard won't help with this particular download, but it WILL allow you to run the latest version of Flash Player, which will allow you access to much more Flash content on the web today.
Apple sells Snow Leopard (OS 10.6) for $19.99 and they ship it free.
As to your current install issue:
Download the latest working version for OS 10.5.8 (Intel) here: Flash Player 10.3.183.90
Unzip it. Inside thereshould be a few folders. One is named "10_3_r183_90".
Inside that folder is a flashplayer_10_3r183_90_mac_intel.dmg
Double click that and it will mount on your desktop, and open a window with the installer.
Right click the installer and choose "Show Package Contents"
Go to: Contents/Resources and double click the Adobe Flash Player.pkg file
The installer will open, ask for your Admin password, and it will install.
Do that and order a copy of 10.6. and install it. then you'll need to update it to 10.6.8, and you can run all sorts of stuff that 10.5 won't handle.
The install was successful but when I test on youtube I says I need to download flash player...
That'll happen with OS 10.5.8.
The Flash Player you have is the newest for your OS - it ISN'T the newest available, and many websites require a minimum of version 11 to access their content. That's coded into the videos or games in the page. There's no way around it, other than ordering a copy of OS 10.6 from Apple, installing it and running the Apple updater to get it to OS 10.6.8. THEN you can install Flash Player 11.9.900.170.