Anyone coming across this problem: check VMs listed under 'Datastore' -> 'Virtual Machines' view has CD drives connected to a path on the datastore you are unable to remove.
This was the issue for me in vSphere 4.1.
I had the same problem here with vSphere 5, but the above point resolved the issue, worth taking 1 minute to check!
Thanks