I don't know if I can help.
I also have a NOOK and am having a problem with a Library book, but I hasten to add that it is a different problem,... trouble in downloading.
However, how are you trying to "return" the book ?
When I return them early I do so from ADE, clicking on "Item Options" then "Return borrowed Item". This immediately "returns" it and it is removed from my Library account.
How are you doing this via your NOOK ?
...by the way just noticed this post by Zschau13;
"ADE Cannot Connect to Fullfillment Server"
Is that the same problem as you ?
I think I've found the problem. I am using a wireless modem and there seems to be an issue with Firefox. So, I switched to Internet Explorer. I was able to return and download books without problems. The steps are a little different but it worked just fine.