Reporting back for posterity: indeed, the problem was was the huge (fake) file size of the -rdmp files. Turns out that in the previous experiment I placed the datastore on NFS (just for the experiment, not as a permanent solution) and apparently something in the NFS layer rejected a file with such a reported size (4TB). Once I moved the datastore to local storage, vmkfstool completed just fine and I (seem to) have raw disk VMDK files.
Hope this will help someone someday.
Thanks to everyone who replied!
Not that my issues with ESXi are over: I now find myself not being able to launch the console. I get "VMRC console has disconnected - attempting to reconnect..." and no console. But this is for a separate thread.
It's like pulling teeth.