The reason could be the different vmfs version 5 and 3 ?
I agree, but upgrading a shared datastore to VMFS-5 would certainly be a self-introduced issue 
I checked to documentation regarding compatibility real quick and what I found is:
http://www.vmware.com/support/vsphere5/doc/vsphere-esx-vcenter-server-50-release-notes.html
vCenter Server 5.0 can manage ESXi 5.0 hosts in the same cluster with ESX/ESXi 4.x and ESX/ESXi 3.5 hosts. It can also manage ESX/ESXi 3.5 hosts in the same cluster with ESX/ESXi 4.x hosts. vCenter Server 5.0 cannot manage ESX 2.x or 3.0.x hosts.
André