Did you make the backup from the Organizer:
File >> Backup (Ctrl+B)
Are you using an external hard drive that is compatible with both Windows and Mac? Most are compatible for reading and transferring files but won’t let you write from the Mac to the external hard drive.
If you have made a correct backup you should be able to launch Organizer on the Mac and use:
File >> Restore Catalog
Then navigate to the TLY file. If iPhoto opens automatically press Cmd+Q to quit that application to avoid duplication.