SOLVED
Suspend still writes to disk - where else could it store the suspended state if not on disk ?
I was thinking memory somehow because there is no way it could write that much, that fast to disk in the time it takes to suspend. Reads are massively faster than writes. Before starting my own IT company, I had been a CTO of two large automotive suppliers for many years. ALL of the userland slowdowns were write-related. Except on boot or launch of a new program, only 4% of the reads come from the disk. The rest are from cache and read-aheads. Writes are incredibly expensive. For every write, the OS needs to check the cache and determine which sections to mark as dirty, then there is read after write to make sure it actually was written, and 100% of the writes need to make it to disk within a short period of time. If it is a large write, it becomes more efficient for the OS to suspend reads, do one large write that flushes the write buffers, then mark all the dirty cache, and then go back to reading and writing. This also prevents what could turn into an endless loop. That's also why caching controllers flunk the benchmark throughput tests, but are astounding at improving userland experience on busy systems. 75-90% of their cache is used for write. Even though there are far more reads, they are also very cheap. Write is everything.
What I discovered:
I understand why VMWare changed it, why it makes a ton of sense, and why it is confusing people.
Earlier I mentioned that I had forced the power off during Windows shutdown because I knew it was not possible to suspend/write 4 gigs that quickly. I proved that by waiting the normal 5 seconds plus another 5 seconds, for shutdown, and then powered off the PC. It resulted in a corrupted suspend. This was to prove that suspend never happened. This time I monitored disk I/O when I did the mirculous suspend, and this is what I discovered. The suspend APPEARS to happen in a few seconds, but it doesn't. It takes place in the background over time. If you wait for it to finish, you can resume in a few seconds. If you immediately try to resume, you will be waiting a long time.
What this means:
If you commonly work in a VM throughout the day, and suspend it just prior to logging off or shutting down, the time and disk I/O to shut down the VM gets added to the already intense write traffic present at log off and shut down resulting in a long delay. If you wait until the VM is actually suspended, log off and shutdown happen in a few seconds. That also explains the variability in log off times, and why the more of a hurry you are to log off or shut down, the slower it goes. I haven't timed it, but it seems significantly faster to wait until the VM has suspended, and then log off or shut down.
Edit: 3/14/2012
It bugged me that Workstation 7.x on a slower machine was much faster at suspending. Yesterday I got a popup in Workstation 8 that read there was an update available. With low expectations I installed the new version, 8.0.2 build-591240. It FIXED THE PROBLEM ENTIRELY! It suspends like 7.x now. Yes, disk writing happens in the background as before, but the background suspending ONLY REQUIRES 5 TO 15 SECONDS NOW! That's SIXTY TIMES FASTER!
