Photo by Denys Nevozhai on Unsplash

One of my Networking & Security TAM customers recently asked me a question.  “How do we test HCX performance?”

When performing Bulk Migrations with HCX, the migrations were taking longer than they expected.

HCX Perftool to the rescue!  For a quick overview, see this VMware KB article Steps to Run Perftest in HCX (56211)

If you are new to HCX, check out this overview and demo from Tech Field Day 2019

Another great resource for all things HCX, is https://hcx.design/, where you can find VMware HCX Design, Operational and Reference Topics

OK, so know you are familiar with HCX let’s dive into the HCX perftool.

I walked the customer through this tool, and we made changes then validated the changes had a positive impact on performance.

We determined that the HCX Interconnect (HCX-WAN-IX) was hosted on poor performing storage. We migrated the HCX-WAN-IX to a different storage provider and the throughput almost doubled!

DISCLAIMER:  The screen shots below are from a lab, NOT my customers environment.  As this is a lab, the test results are less than you would expect to achieve  in a production environment.

1. Connect to the HCX Manager(Connector) console via SSH
2. Login as admin
3. Type ccli to enter the HCX Central CLI
4. Now type list to display the deployed appliances.

   

5. Note the Id column next to each appliance.  In this example I have the Hybrid Interconnect & Network Extensions deployed
6. Type go 0 to switch to the IX appliance

   

7. Enter perftest help to display the various tests available

   

8. First up I run pertest reachability to confirm connectivity between the components of the HCX Service Mesh. Truncated output as follows

   

9. Next I run pertest uplink . This will run a test to baseline the WAN connection between the HCX sites.  Let me explain the results

   

   1. Round Trip Time
   2. Uplink and test direction.  e.g. Legacy DC HCX IX IP to Hybrid Cloud HCX IX IP
   3. Duration, how long did each test run for? e.g. 60 secs
   4. Transfer, how much data was transferred for the test? e.g. 2.87 GBytes
   5. Bandwidth, what was the transfer rate? e.g. 411 Mbits/sec
   6. Retransmit, how many packets were retransmitted? e.g. 693 High numbers may indicate  network congestion, packet drops etc
   7. Test and results are then shown for the return path.  e.g. Hybrid Cloud HCX IX IP to Legacy DC  HCX IX IP
10. I’m going to run perftest all, then y to start the tests.  Note that you can change the individual test duration and tcp port used for the tests if desired, and also the estimated test duration

    

11. At the end of the test we will be presented with a summary of the results

    

    1. Note that the results for the WANOPT test are low compared to the SITE test and the uplink test we performed earlier.  This may indicate an issue with the storage where the IX appliance is hosted, ESX host contention etc. Slow storage  was causing latency/low throughput in in my real world customer example

After migrating the HCX-WAN-IX to a different storage provider, we re ran the performance tests to check and validate if the changes had a positive impact.

In Summary I hope you know have an understanding of the HCX perftest tool and can leverage this in your own environment

• NSX
• VMC