I don't know if this will work but it might:
Using Windows Explorer find My Digital Editions in your My Documents folder.
Find the book you returned and delete it. Apparently ADE leaves books in My Digital Editions after they are returned but flags them so they won't open.
I have to wait for the book loan to expire. BTW, I don't use Windows.
I had not noticed that effect on Open Library, I have 'returned' books early and they then became available for others to check out but I never tried to re-download inside the loan period. Re not using Windows: I expect there will still be an equivalent folder to "My Digital Editions" into which ADE saves each epub. It still might be worth searching for the book and deleting it from that folder then trying to re-download the book.