Well, I decided I'd wipe ESXi and recreate everything from scratch. So new install of ESXi(updated to the latest public build, 1157734), I did reuse the vmdk files but the remainder of the VMs was recreated from scratch. Worked fine for 3 whole days...
Well, this evening I was replacing a disk in my zpool(FreeNAS VM) and wouldn't you know, the darn thing disconnected. This time I didn't do any kind of experimenting. Very basic setup of the VMs. And for whatever reason this evening, poof. Checked the syslog and lots of "Unknown: out of memory [6955]" errors. Just like above, but 1 extra line. This is getting dangerous because I can't even rebuild my pool to restore redundancy without having problems.
From the first post, right after the "Request Header Id" error I have a new entry:
sfcb-VMware_base[5830]: Dropped response operation details -- namespace: root/cimv2, className: OMC_RawIpmiSensor, Type: 0
Because of the pool resilvering the M1015(using VT-d to pass through to the FreeNAS machine) was heavily loaded during this time. Any chance this could be a trigger? Does VT-d have a "cache" that may need to be bigger to store commands for the PCIe devices that are passed through? CPU usage was very high during the resilvering.
Edit: It turns out that my desktop began an automated backup to the FreeNAS VM, so the high traffic across the network port is probably responsible for the problem. Is there some kind of network "cache" for ESXi that I may need to make bigger?
The VMs are clearly still running since the resilvering is still running. But last time I looked it had over 40 hours remaining, and I can't just wait until the hdd led's go idle to roll the dice on this thing.