Some links and graphs on ESXi capping/throtteling disk speeds

As promised in “Solution” on ESXi 6.7 smartinfo throwing error Cannot open device, here are a few links in capping throttling disk speeds by ESXi followed by a few graphs of my own:

• GUI operations might be capped, try dd: [Wayback] datastore – How can I speed up file tranfers on local storage in vSphere 5? – Server Fault
  

  I never got any official confirmation for this, but I believe the I/O is capped (or at least de-prioritized) for datastore copy/move operations from the GUI as I have seen rather similar behaviour in different ESXi environments on from version 3.5.

• [Wayback] Solved: ESXi disk performance, possibly capped? – VMware Technology Network VMTN
  

  Question:

  When I copy a 3GB ISO file from the raid set to the sata disk, the average disk transfer is 50MB/sec. This puzzles me, because the SATA disk can do 100MB/sec easy. When I start a second copy, the average disk transfer doubles to 100MB/sec.

  …

  Answer:

  File transfer speeds are capped in the console so you will not be able to get maximum speeds

My own observations on ESXi 6.7 update 3:

1. One rsync operation:
   

   1 rsync from 860 EVO SSD to 960 PRO NVMe

2. Two rsync operations:
   

   2 rsync from 860 EVO SSD to 960 PRO NVM

3. Resume actions were about 10 times faster than the single rsync read speeds:
   

   Resume action

4. Suspend actions were between 4 and 6 times faster than rsync write speeds:
   

   Start of suspend action

    

   Finish of suspend action

For each rsync operation, I had a separate SSH session going, and the speed doubled.

The resume action of all Virtual Machines was almost a flat speed curve.

The suspend action of all Virtual Machines started fast (when all machines were suspending) and finished slower (when only the largest virtual machines were still suspending)

–jeroen