Please go to the folder and delete the files. Library will be updated automatically.
On importing items to the library, they will be copied to a temp location(If your composition is not saved yet) or to your project location where you have saved your animation. Deleting the items from that location, removes them from the library as well.
Also, pl note this does not mean the original files that are imported will be deleted. Removing from library only removes the duplicate copy created when importing to the same.