
Document ID: 116248
Updated: Jul 01, 2013
Contributed by Jim Hicks and Shelley Bhalla, Cisco TAC Engineers.
Introduction
This document describes how to configure a Multi-Hop Fibre Channel over Ethernet (FCoE) between the Cisco Unified Computing System (UCS) and a Nexus 5000 switch.
Prerequisites
Requirements
The information in this document is based on these software and hardware versions:
- UCS 5108
- UCS 61xx/62xx
- Nexus 5548UPor 5596UP
Components Used
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Configure
Network Diagram
Figure 1. Topology Used in the Configuration
Configure the UCS
There are two procedures outlined here to help you configure the UCS:
- Create FCoE virtual storage area networks (VSANs).
- Configure the UCS FCoE Port Channel.
Create FCoE VSANs
- You need to create VSANs and FCoE VLANs in UCS in order to carry the FCoE traffic. A best practice, but not a requirement, is to align the VSAN ID with the FCoE VLAN ID in order to help track and troubleshoot.
- In UCSM, navigate to the SAN tab in the navigation pane, and choose SAN Cloud.
- In the main window, click the green + button on the right in order to display the Create VSAN window.
- Complete the fields as follows:
Name: Enter a name for the VSAN
FC Zoning: Disabled ? This is only used when you implement local zoning features
Fabric Radio Buttons: Choose Fabric A
VSAN ID: Enter the VSAN ID for the VSAN that is created on Fabric A ? This should
correspond with the VSAN ID that will be created on the Nexus 5000-A
FCoE VLAN: Enter the FCoE VLAN ID for the VLAN which maps to this VSAN ? This should correspond with the VLAN ID which will be created on the Nexus 5000-A - Click OK in order to display the Create VSAN success window. Click OK.
- Repeat these steps for Fabric B. In the main window, click the green + button on the right in order to display the Create VSAN window.
- Complete the fields as follows:
Name: Enter a name for the VSAN
FC Zoning: Disabled ? This is only used when you implement local zoning features
Fabric Radio Buttons: Choose Fabric B
VSAN ID: Enter the VSAN ID for the VSAN that is created on Fabric B ? This should
correspond with the VSAN ID that will be created on the Nexus 5000-B
FCoE VLAN: Enter the FCoE VLAN ID for the VLAN which maps to this VSAN ? This should correspond with the VLAN ID which will be created on the Nexus 5000-B - Click OK in order to display the Create VSAN success window. Click OK.
- In the main window, verify that the VSANs have been created in their correct fabrics:
Configure UCS FCoE Port Channel
- Create and configure InterSwitch Link / Port Channels. Create an FCoE Uplink or a FCoE Port Channel between the Nexus 5000-A and FI-A.
- When the Cisco Unified Computing System Manager (UCSM) Set Port Channel Name window displays, enter the Port Channel ID (in this example, the Port Channel ID is 2) and Name.
- When you are finished, click Next.
- When the UCSM Add Ports window displays, choose the desired ports in the Port column and click the right arrow between the columns in order to add these ports to the FCoE Port Channel.
- When the FCoE Uplink Ports have been moved to the Port Channel column, click Finish.
- When the Create FCoE Port Channel Success window displays, click OK.
- Review the FCoE Port Channel information in the UCSM main window in order to confirm that the FCoE Port Channel has been created with the correct ports.
- In the Navigation Pane, click the Port Channel that was just created. In the General tab of the UCSM main window, click the VSAN drop-down list and choose VSAN FCoE-VSAN100 (100) in order to assign this Port Channel to VSAN100.
- Click Save Changes. When the Cisco UCSM Success window displays, click OK.
- Repeat Steps 1-9 for FI-B, but substitute Fabric B, FCoE-PortChannel ID 3, and VSAN101 where appropriate.
Configure the Nexus 5000 for FCoE Multi-Hop
There are several procedures that are required in order to configure the Nexus 5000 for FCoE Multi-Hop:
- Enable Feature FCoE
- Enable N Port Identifier Virtualization (NPIV) on the Nexus 5000
- Enable the Link Aggregation Control Protocol (LACP) on the Nexus 5000
- Enable Nexus 5000 Quality of Service (QoS) for FCoE
- Configure VLAN
- Configure VSAN
- Map the VLAN to VSAN
Enable Feature FCoE
- On the Nexus 5000-A switch, enter:
Nexus 5000-A# configure terminal
Nexus 5000-A(config)# feature fcoe
FC license checked out successfully
fc_plugin extracted successfully
FC plugin loaded successfully
FCoE manager enabled successfully
FC enabled on all modules successfully
Enabled FCoE QoS policies successfully - In order to confirm that the feature is now active, enter:
Nexus 5000-A# show feature | include fcoe
Feature Name Instance State
-------------------- -------- --------
fcoe 1 enabled - Repeat Steps 1-2 on the Nexus 5000-B.
Enable NPIV on the Nexus 5000
- In order to enable feature NPIV, enter:
Nexus 5000-A# configuration terminal
Nexus 5000-A(config)# feature npiv - In order to confirm that feature NPIV is now active, enter:
Nexus 5000-A# show npiv status
NPIV is enabled - Repeat Steps 1-2 on the Nexus 5000-B.
Enable LACP on the Nexus 5000
- In order to enable feature LACP, enter:
Nexus 5000-A# configure terminal
Nexus 5000-A(config)# feature lacp - In order to confirm that LACP has been enabled, enter:
Nexus 5000-A#show feature | include LACP
Feature Name Instance State
-------------------- -------- --------
lacp 1 enabled - Repeat Steps 1-2 on the Nexus 5000-B.
Configure Nexus 5000 FCoE QoS Parameters
The configurable Cisco NX-OS QoS features allow you to classify the network traffic, prioritize the traffic flow, and provide congestion avoidance. The default QoS configuration on the Nexus 5000 provides lossless service for FC and FCoE traffic and best-effort service for Ethernet traffic. The Nexus 5000 FCoE QoS Parameters should already be configured by default.
- In order to verify the FCoE QoS parameters, enter:
Nexus 5000-A# show startup-config ipqos
- Within the output, locate these items:
. .
system qos
service-policy type queuing input fcoe-default-in-policy
service-policy type queuing output fcoe-default-out-policy
service-policy type qos input fcoe-default-in-policy
service-policy type network-qos fcoe-default-nq-policy - Skip this step if the FCoE QoS configurations from Step 2 are present. If they are not, enter this configuration:
Nexus 5000-A# configure terminal
Nexus 5000-A(config)# system qos
Nexus 5000-A(config-sys-qos)# service-policy type qos input
fcoe-default-in-policy
Nexus 5000-A(config-sys-qos)# service-policy type queuing input
fcoe-default-in-policy
Nexus 5000-A(config-sys-qos)# service-policy type queuing output
fcoe-default-out-policy
Nexus 5000-A(config-sys-qos)# service-policy type network-qos
fcoe-default-nq-policy - In order to confirm that the FCoE QoS parameters are now configured, enter:
Nexus 5000-A# show startup-config ipqos
- Within the output, locate these items:
..
system qos
service-policy type queuing input fcoe-default-in-policy
service-policy type queuing output fcoe-default-out-policy
service-policy type qos input fcoe-default-in-policy
service-policy type network-qos fcoe-default-nq-policy - Repeat Steps 1-3 on the Nexus 5000-B.
Create Nexus 5000 VLANs
Create two VLANs for FCoE use. The even should correlate with FI-A and the odd should correlate with FI-B. Use the FCoE VLAN IDs specified in the UCS FCoE Multi-Hop section.
- On the Nexus 5000-A that is connected to FI-A, enter:
Nexus 5000-A# configure terminal
Nexus 5000-A(config)# vlan 100
Nexus 5000-A(config-vlan)# name FCoE-VLAN100 - In order to confirm that VLAN has been created on the Nexus 5000-A, enter:
Nexus 5000-A# show vlan brief
VLAN Name Status Ports
---- ----------------------------- --------- -------------------------------
1 default active Eth1/1, Eth1/2, Eth1/3, Eth1/4
Eth1/5, Eth1/6, Eth1/7, Eth1/8
.
100 FCoE-VLAN100 active - On the Nexus 5000-B that is connected to FI-B, enter:
Nexus 5000-B# configure terminal
Nexus 5000-B(config)# vlan 101
Nexus 5000-B(config-vlan)# name FCoE-VLAN101 - In order to confirm that VLAN has been created on the Nexus 5000-B, enter:
Nexus 5000-B# show vlan brief
VLAN Name Status Ports
---- ----------------------------- --------- -------------------------------
1 default active Eth1/1, Eth1/2, Eth1/3, Eth1/4
Eth1/5, Eth1/6, Eth1/7, Eth1/8
.
100 FCoE-VLAN101 active
Create Nexus 5000 VSANs
Create two VSANs. The even should correlate with FI-A and the odd should correlate with FI-B. Use the VSAN IDs specified in the UCS FCoE Multi-Hop section.
- On the Nexus 5000-A that is connected to FI-A, enter:
Nexus 5000-A# configure terminal
Nexus 5000-A(config)#vsan database
Nexus 5000-A(config-vsan)# vsan 100
Nexus 5000-A(config-vsan)# vsan 100 name FCoE-VSAN100 - In order to confirm that VSAN has been created on the Nexus 5000-A, enter:
Nexus 5000-A#show vsan 100
vsan 100 information
name:FCoE-VSAN100 state:active
interoperability mode:default
loadbalancing:src-id/dst-id/oxid
operational state:down - On the Nexus 5000-B that is connected to FI-B, enter:
Nexus 5000-B# configure terminal
Nexus 5000-B(config)#vsan database
Nexus 5000-B(config-vsan)# vsan 101
Nexus 5000-B(config-vsan)# vsan 101 name FCoE-VSAN101 - In order to confirm that VSAN has been created on the Nexus 5000-B, enter:
Nexus 5000-A# show vsan 101
vsan 101 information
name:FCoE-VSAN101 state:active
interoperability mode:default
loadbalancing:src-id/dst-id/oxid
operational state:down
Map the Vlan to VSAN
- In order to map the Nexus 5000-A VLANs to their VSANs (to FI-A), enter:
Nexus 5000-A#configure terminal
Nexus 5000-A(config)# vlan 100
Nexus 5000-A(config-vlan)#fcoe vsan 100 - In order to confirm VLAN-to-VSAN mapping on the Nexus 5000-A, enter:
Nexus 5000-A# show vlan fcoe
Original VLAN ID Translated VLAN ID Association State
---------------- ------------------ -----------------
10 100 Operational - In order to map the Nexus 5000-B VLANs to their VSANs (to FI-B), enter:
Nexus 5000-B# configure terminal
Nexus 5000-B(config)# vlan 101
Nexus 5000-B(config-vlan)# fcoe vsan 101 - In order to confirm VLAN-to-VSAN mapping on the Nexus 5000-B, enter:
Nexus 5000-B# show vlan fcoe
Original VLAN ID Translated VLAN ID Association State
---------------- ------------------ -----------------
10 101 Operational
Configure Nexus 5000 FCoE Uplink Port and Port Channel
- In order to configure and add to the Port Channel the 10GbE ports which have been cabled to the FI-A 10GbE ports on the Nexus 5000-A switch, enter:
Nexus 5000-A#configure terminal
Nexus 5000-A(config)# interface ethernet 1/5-6
Nexus 5000-A(config-if-range)# switchport mode trunk
Nexus 5000-A(config-if-range)# switchport trunk allowed vlan 100
Nexus 5000-A(config-if-range)# channel-group 2 mode active
Nexus 5000-A(config-if-range)#no shutdown
Nexus 5000-A(config-if-range)# exit - In order to create Port Channels, enter:
Nexus 5000-A# configure terminal
Nexus 5000-A(config)# interface port-channel 2
Nexus 5000-A(config-if)# no shutdown - In order to confirm that the Port Channel between the Nexus 5000-A and FI-A is now active, enter:
Nexus 5000-A# show port-channel database
port-channel2
Last membership update is successful
2 ports in total, 2 ports up
First operational port is Ethernet1/5
Age of the port-channel is 0d:00h:09m:19s
Time since last bundle is 0d:00h:05m:27s
Last bundled member is Ethernet1/6
Time since last unbundle is 0d:00h:05m:38s
Last unbundled member is Ethernet1/6
Ports: Ethernet1/5 [active ] [up] *
Ethernet1/6 [active ] [up] - In order to configure and add to the Port Channel the 10GbE ports which have been cabled to the FI-B 10GbE ports on the Nexus 5000-B switch, enter:
Nexus 5000-B# configure terminal
Nexus 5000-B(config)# interface ethernet 1/5-6
Nexus 5000-B(config-if-range)# switchport mode trunk
Nexus 5000-B(config-if-range)# switchport trunk allowed vlan 101
Nexus 5000-B(config-if-range)# channel-group 3 mode active
Nexus 5000-B(config-if-range)# no shutdown
Nexus 5000-B(config-if-range)# exit - In order to create the Port Channel, enter:
Nexus 5000-B# configure terminal
Nexus 5000-B)config)# interface port-channel 3
Nexus 5000-B(config-if)# no shutdown - In order to confirm that the Port Channel between the Nexus 5000-B and FI-B is now active, enter:
Nexus 5000-B# show port-channel database
port-channel3
Last membership update is successful
2 ports in total, 2 ports up
First operational port is Ethernet1/5
Age of the port-channel is 0d:00h:00m:29s
Time since last bundle is 0d:00h:00m:29s
Last bundled member is Ethernet1/6
Ports: Ethernet1/5 [active ] [up] *
Ethernet1/6 [active ] [up]
Create Nexus 5000 Virtual Fibre Channel (VFC)
VFCs need to be created and bound to the Ethernet interfaces (Port Channels) that carry the FCoE traffic.
- In order to create and bind VFC 2 to Port-Channel 2 on the Nexus 5000-A, enter:
Nexus 5000-A# configure terminal
Nexus 5000-A(config)# interface vfc 2
Nexus 5000-A(config-if)# bind interface port-channel 2
Nexus 5000-A(config-if)# switchport trunk allowed vsan 100
Nexus 5000-A(config-if)# switchport mode F
Nexus 5000-A(config-if)# no shutdown - In order to confirm creation and binding of VFC 2 to Port-Channel 2, enter:
Nexus 5000-A# show interface vfc 2
vfc2 is trunking
Bound interface is port-channel2
Hardware is Ethernet
Port WWN is 20:01:54:7f:ee:8b:a6:ff
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is TF
Port vsan is 1
Trunk vsans (admin allowed and active) (100)
Trunk vsans (up) (100)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1020 frames input, 102728 bytes
0 discards, 0 errors
1154 frames output, 478968 bytes
0 discards, 0 errors
last clearing of "show interface" counters never
Interface last changed at Tue Aug 21 06:32:40 2012 - In order to create and bind VFC 3 to Port-Channel 3 on the Nexus 5000-B, enter:
Nexus 5000-B# configure terminal
Nexus 5000-B(config)# interface vfc 3
Nexus 5000-B(config-if)# bind interface port-channel 3
Nexus 5000-B(config-if)# switchport trunk allowed vsan 101
Nexus 5000-A(config-if)# switchport mode F
Nexus 5000-B(config-if)# no shutdown - In order to confirm creation and binding of VFC 3 to Port-Channel 3, enter:
Nexus 5000-B# show interface vfc 3
vfc3 is trunking
Bound interface is port-channel3
Hardware is Ethernet
Port WWN is 20:02:54:7f:ee:8c:e2:7f
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is TF
Port vsan is 1
Trunk vsans (admin allowed and active) (101)
Trunk vsans (up) (101)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
738 frames input, 69404 bytes
0 discards, 0 errors
763 frames output, 427488 bytes
0 discards, 0 errors
last clearing of "show interface" counters never
Interface last changed at Tue Aug 21 06:47:03 2012
Configure Storage Ports on the Nexus 5000
Connect the storage ports from the FCoE-enabled storage system to their designated ports on the Nexus 5000s. VFCs need to be created for these storage ports and the associated interfaces need to be configured.
- In order to configure the Nexus 5000-A Storage Port Ethernet Interface, enter:
Nexus 5000-A# configure terminal
Nexus 5000-A(config)# interface Ethernet 1/32
Nexus 5000-A(config-if)# switchport mode trunk
Nexus 5000-A(config-if)# switchport trunk allowed vlan 100
Nexus 5000-A(config-if)# spanning-tree port type edge trunk
Warning: Edge port type (portfast) should only be enabled on ports connected to a single host.
Connecting hubs, concentrators, switches, bridges, etc... to this interface when edge port type
(portfast) is enabled, can cause temporary bridging loops.
Use with CAUTION
Nexus 5000-A(config-if)# no shut
Nexus 5000-A(config-if)# exit - In order to create VFC and bind the associated Storage Port Ethernet Interface to it on the Nexus 5000-A, enter:
Nexus 5000-A(config)# interface vfc 132
Nexus 5000-A(config-if)# switchport trunk allowed vsan 100
Nexus 5000-A(config-if)# bind interface Ethernet 1/32
Nexus 5000-A(config-if)# no shutdown
Nexus 5000-A(config-if)# exit - In order to map the Storage Port VFC to the FCoE VSAN previously configured on the Nexus 5000-A, enter:
Nexus 5000-A(config)# vsan database
Nexus 5000-A(config-vsan-db)# vsan 100 interface vfc 132
Nexus 5000-A(config-vsan-db)# exit - In order to confirm that VFC has been created and bound to the correct Ethernet Interface and VSAN on the Nexus 5000-A, enter:
Nexus 5000-A(config)# show interface vfc 132
vfc132 is trunking
Bound interface is Ethernet1/32
Hardware is Ethernet
Port WWN is 20:83:54:7f:ee:8b:a6:ff
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is TF
Port vsan is 100
Trunk vsans (admin allowed and active) (100)
Trunk vsans (up) (100)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
10 frames input, 1060 bytes
0 discards, 0 errors
11 frames output, 1404 bytes
0 discards, 0 errors
last clearing of "show interface" counters never
Interface last changed at Tue Aug 21 06:37:14 2012 - In order to configure the Nexus 5000-B Storage Port Ethernet Interface, enter:
Nexus 5000-B# configure terminal
Nexus 5000-B(config)# interface Ethernet 1/32
Nexus 5000-B(config-if)# switchport mode trunk
Nexus 5000-B(config-if)# switchport trunk allowed vlan 101
Nexus 5000-B(config-if)# spanning-tree port type edge trunk
Warning: Edge port type (portfast) should only be enabled on ports connected
to a single host. Connecting hubs, concentrators, switches, bridges, etc...
to this interface when edge port type (portfast) is enabled, can cause temporary
bridging loops.
Use with CAUTION
Nexus 5000-B(config-if)# no shut
Nexus 5000-B(config-if)# exit - In order to create VFC and bind the associated Storage Port Ethernet Interface to it on the Nexus 5000-B, enter:
Nexus 5000-B(config)# interface vfc 132
Nexus 5000-B(config-if)# switchport trunk allowed vsan 101
Nexus 5000-B(config-if)# bind interface Ethernet 1/32
Nexus 5000-B(config-if)# no shutdown
Nexus 5000-B(config-if)# exit - In order to map the Storage Port VFC to the FCoE VSAN that was previously configured on theNexus 5000-B, enter:
Nexus 5000-B(config)# vsan database
Nexus 5000-B(config-vsan-db)# vsan 101 interface vfc 132
Nexus 5000-B(config-vsan-db)# exit - In order to confirm that VFC has been created and bound to the correct Ethernet Interface and VSAN on the Nexus 5000-B, enter:
Nexus 5000-B(config)# show interface vfc 132
vfc132 is trunking
Bound interface is Ethernet1/32
Hardware is Ethernet
Port WWN is 20:83:54:7f:ee:8c:e2:7f
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is TF
Port vsan is 101
Trunk vsans (admin allowed and active) (101)
Trunk vsans (up) (101)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
1 minute input rate 136 bits/sec, 17 bytes/sec, 0 frames/sec
1 minute output rate 200 bits/sec, 25 bytes/sec, 0 frames/sec
10 frames input, 1060 bytes
0 discards, 0 errors
11 frames output, 1404 bytes
0 discards, 0 errors
last clearing of "show interface" counters never
Interface last changed at Tue Aug 21 06:50:31 2012
Verify
Zoning Example (on the Nexus 5500s)
Fabric A
Configuration
zone name ESXI01_1 vsan 100
member pwwn 10:00:e4:11:5b:65:c1:11
member pwwn 50:01:43:80:11:36:37:5c
zone name ESXI02_1 vsan 100
member pwwn 10:00:e4:11:5b:65:c0:c9
member pwwn 50:01:43:80:11:36:37:5c
zone name ESXI03_1 vsan 100
member pwwn 10:00:e4:11:5b:65:c2:01
member pwwn 50:01:43:80:11:36:37:5c
zoneset name Fabric-A vsan 100
member ESXI01_1
member ESXI02_1
member ESXI03_1
zoneset activate name Fabric-A vsan 100
Verification
Nexus 5000-A# show flogi database
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
vfc2 100 0xab0000 10:00:e4:11:5b:65:c1:11 20:00:e4:11:5b:65:c1:11
vfc2 100 0xab0040 10:00:e4:11:5b:65:c0:c9 20:00:e4:11:5b:65:c0:c9
vfc2 100 0xab0020 10:00:e4:11:5b:65:c2:01 20:00:e4:11:5b:65:c2:01
vfc132 100 0xab00c0 50:01:43:80:11:36:37:5c 50:01:43:80:11:36:37:50
This flogi table shows three hosts (pwwns starting with 10:) and one storage pwwn
(50:01:43:80:11:36:37:5c).
Fabric B
Configuration
zone name EVA_1-ESXI01_2 vsan 101
member pwwn 10:00:e4:11:5b:67:c1:12
member pwwn 50:01:43:80:11:36:37:5d
zone name EVA_1-ESXI02_2 vsan 101
member pwwn 10:00:e4:11:5b:67:c0:c8
member pwwn 50:01:43:80:11:36:37:5d
zone name EVA_1-ESXI03_2 vsan 101
member pwwn 10:00:e4:11:5b:67:c2:04
member pwwn 50:01:43:80:11:36:37:5d
zoneset name Fabric-B vsan 101
member EVA_1-ESXI01_2
member EVA_1-ESXI02_2
member EVA_1-ESXI03_2
zoneset activate name Fabric-B vsan 101
Verification
Nexus 5000-B# show flogi database
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
vfc3 101 0xab0000 10:00:e4:11:5b:67:c1:12 20:00:e4:11:5b:65:c1:12
vfc3 101 0xab0040 10:00:e4:11:5b:67:c0:c8 20:00:e4:11:5b:65:c0:c8
vfc3 101 0xab0020 10:00:e4:11:5b:67:c2:04 20:00:e4:11:5b:65:c2:04
vfc132 101 0xab00c0 50:01:43:80:11:36:37:5d 50:01:43:80:11:36:37:50
This flogi table shows three hosts (pwwns starting with 10:) and one storage pwwn
(50:01:43:80:11:36:37:5d).
Troubleshoot
There is currently no specific troubleshooting information available for this configuration.
Open a Support Case (Requires a Cisco Service Contract.)
Related Cisco Support Community Discussions
The Cisco Support Community is a forum for you to ask and answer questions, share suggestions, and collaborate with your peers.
Refer to Cisco Technical Tips Conventions for information on conventions used in this document.