Insane Mode Encryption Performance

This document discusses Aviatrix Insane Mode (or InsaneMode) performance test benchmarks, parameters that affect performance and how you can tune your environment for best performance.

For more information on Aviatrix Insane Mode, check out this link.

Insane Mode Performance Test Setup

The iperf3 test is performed between a Spoke VPC instance and on-prem VM. The test setup is shown in the diagram below. The encryption is end to end: between on-prem CloudN and Transit GW and between Spoke gateway and Transit GW.

insane_perf_setup

Key variables that affect performance are:

  • MTU size of all devices in the data path.
  • Latency between on-prem and the Transit VPC.
  • Client TCP window size.
  • The number of TCP streams.

AWS Performance Test Results

The performance test is conducted between a c5n.4xlarge instance in the Spoke VPC and on-prem host machine over a 10Gbps Direct Connect between the Transit VPC and an on-prem datacenter (Equinix co-lo). The physical latency is 5ms. Additional latency is injected into the data path to simulate the latency impact to the end to end throughput.

1. MTU = 9000 Bytes, AWS C5n.4xlarge

c5n_throughput_9000B

2. MTU = 1500 Bytes, AWS C5n.4xlarge, on-prem to spoke instance through Transit instance

The test result below is for the end to end encryption performance. Note the Transit gateway does encryption and decryption twice, hence the performance is around 1Gbps.

c5n_throughput_1500B

3. MTU = 1500 Bytes, AWS C5n.4xlarge, instance to instance directly

The test result below is for the instance to instance over AWS Peering.

throughput_1500B_peering

Azure Performance Test Results

Gateway VM Type Throughput with MTU 1500B Throughput with MTU 4000B
Standard_F32s_v2 8.9Gbps 13.3Gbps
Standard_F48s_v2 10.9Gbps 17.4Gbps
Standard_D64_v3 8.2Gbps 12.1Gbps
Standard_D32_v3 7.1Gbps 10.9Gbps
Standard_D5_v2 6.6Gbps 10.1Gbps

How to Tune Performance

1. Check MTU size

Use Trace Path. Go to Troubleshoot -> Diagnostics -> Network. Select a gateway and destination IP address, click Trace Path. It should display MTU of the devices along the path.

2. Tune TCP window size

For Linux machine, follow the instructions here to tune TCP window size.