Cisco Nexus 1000V Switch for VMware vSphere

Introduction

This document describes a problem encountered with the number of maximum ports the VMware Distributed Virtual Switch (DVS) available in its interactions with Cisco Nexus 1000V Series switch (N1KV), and offers a solution to the problem.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• N1KV
• VMware DVS

Components Used

The information in this document is based on N1KV and VMware DVS.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Background Information

One of the most common issues with the N1KV and its interaction with the VMware DVS infrastructure is the limit on the maximum number of ports for the DVS. It is important to be able to create a port profile on the N1KV with enough available ports in order to fulfill your present and future needs. Therefore, most people increase the maximum ports number in the virtual Ethernet (vEth) port profile to a larger number.

From the N1KV side, this increase is fine; from the VMware DVS side, the increase can be a problem. When you create a vEth port profile and enable it, the ports are immediately reserved on the VMware DVS. This can be a problem in the cloud and some enterprise environments. This document examines some limits with the maximum number of ports in order to understand the problem.

Maximum Number of Ports

Here is a list of the maximum number of ports (max ports) per device:

• VMware vCenter Version 4.0 max ports per DVS = 8,192
• VMware vCenter Version 4.1 max ports per DVS = 20,000
• VMware vCenter Version 5.0 max ports per DVS = 30,000
• VMware vCenter Version 5.5 max ports per DVS = 60,000
• N1KV maximum vEth ports = 2,048

Port Consumers

Here are some important points about port consumers:

• Every VMware ESX/ESXi host that is added to the N1KV always consumes 32 ports. There is no way to change port consumption for the vEth type of port profiles.
• Every vEth port profile consumes its max ports setting when it is enabled.
• Some ports are internally-consumed by the DVS and N1KV.

Problem

You encounter a problem when you configure a N1KV with eight vEth port profiles and 32 ESX hosts. Here are the details:

• N1KV Version SV(1.4) with vSphere 4.0 has a DVS maximum of 8,192 ports.
• A setup with 32 ESX hosts requires 1,024 DVS ports (32 ports x 32 hosts).
• With 8,192 max ports, minus the 1,024-port requirement, only 7,168 max ports are available.
• If there are eight vEth port profiles, each with the max ports set to 1,024, then they use 8,192 ports. But, there are only 7,168 max ports left. Therefore, the N1KV is in a deficit. (8,192 - 7,168 = 1,024)
• Another 1,024 available ports are needed on the DVS in order to make the configuration work.

Until Version SV(1.4a), the solution was to set the max ports number lower, upgrade to a newer version of vSphere, or set the DVS available ports higher with the help of VMware or the Increasing the maximum number of vNetwork Distributed Switch (vDS) ports in vSphere 4.x (1038193) Knowledge Base (KB) article. All of the previous options were compromises or difficult to configure, as in the case of the KB article.

N1KV Version SV(1.4a) and Later Solutions

Two new options were introduced in Version 1.4a that improve interactions with the DVS available ports:

• SVS (Software Virtual Switch) connection max port
• vEth port-binding auto

SVS Connection Max Port

This is a new option under the SVS connection where you can actually increase the number of available DVS ports through the SVS connection between the N1KV and the vCenter Server. You set it with this command:

n1kv-14a(config)# svs connection VC
n1kv-14a(config-svs-conn)# max-ports ?
<0-50000>  Number of max-ports for this connection
n1kv-14a(config-svs-conn)# max-ports 9000

In this command, the available ports on a vCenter 4.0 device were changed to 9,000. The downside is that vCenter does not show the extra ports in the vSphere client. However, the next image shows that DVS ports past 8,192 were created:

You must work with your VMware administrator when you make this change. VMware publishes maximum settings for DVS ports for a reason, so you should contact the administrator before you make any drastic changes to this setting.

vEth Port-Binding Auto

The vEth port-binding auto is a great feature. It allocates ports as it needs them, instead of all at once. The auto feature works with static and dynamic port-bindings. Both static and dynamic port-bindings preallocate DVS ports, while the ephemeral does not. In order to turn it on, use this command:

n1kv-14a(config)# port-profile type vethernet profile2
n1kv-14a(config-port-prof)# vmware port-group
n1kv-14a(config-port-prof)# switchport mode access
n1kv-14a(config-port-prof)# switchport access vlan 158
n1kv-14a(config-port-prof)# no shutdown
n1kv-14a(config-port-prof)# max-ports 1024
n1kv-14a(config-port-prof)# port-binding static auto
n1kv-14a(config-port-prof)# state enabled

Note: If you modify a current vEth port profile for auto, you must change the port-binding to no state enable, and then state enable. This is a disruptive change on a current port profile.

The port profile is created with 17 ports allocated from the DVS. You can allocate in chunks of 16 up to the max ports for the port profile. When you get within three ports of the current allocation, you add another 16 ports. Here is a screen shot of the previous port profile in vCenter 4.0:

Here is a screen shot after the addition of the 14th Virtual Machine (VM) to port profile profile2:

If you want to preallocate more than 17 ports to the port profile, then use the new min-ports command under port profile. Min-ports preallocates more ports on the DVS so that you have more overhead if you must create large numbers of VMs on the port profile.

n1kv-14a(config)# port-profile profile8
n1kv-14a(config-port-prof)# min-ports 40

Note: Remember that these features only work with Version 1.4a.