Tag Archives: FEX configuration

Fabric Extender

What is Nexus 2000 Fabric Extender and how to configure them

Hello everyone ! Did not get a chance to write anything for few weeks .

I would like to talk about Nexus 2000 Fabric Extender and it’s relationship with Nexus 5000 and how to configure them.

Nexus 2000 connects directly to Nexus 5010 or 5020 using upto 4 10G ethernet uplink ports. All configuration are done on nexus 5K.  2000 series switch acts as a remote I/O module or virtual line card of nexus 5K.

12 to 20 Cisco Nexus 2000s can be attached to each Cisco Nexus 5000 .

Cisco Nexus 2000 – Deployment Benefit:

  • Physically resides on the top of each rack but logically acts like an end-of-row access device. 
  • Having this on top of the rack, it reduces the cable runs to the servers.
  • Nexus 2000 are managed from Nexus 5000 switch, so it reduces the management points.
  • Ensures features consistency across hundreds of servers.
  • Nexus 2000 basically makes Nexus 5000 to become a high density 1G access layer switch.
  • Investment protection
  • Virtual network link capabilities.

 

Cisco Nexus 2148T is the First product of Nexus 2000 series.

  • 1 rack-unit fixed-config form factor- 48 GE server-facing (downlink) ports, and 4×10 GE network-facing (uplink) ports.
  • No local switching, no spanning-tree.
  • Server-to-server traffic goes to Cisco Nexus 5000 and back.
  • Nexus 2000 switch has Front-to-back airflow(same as servers) – This helps Hot-aisle,cold-aisle Datacenter Design.
  • Redundant hot-swappable power supplies
  • Removable fan tray with redundant fans
  • Over subscription : 1.2 : 1 to 4.8:1

2148T

 

 

Cisco Nexus 2000 Deployment Model..

2k deployment model

This kind of model provides a straight forward switching and cabling strategy that simplifies network operations, and prepares for future networking technologies.

Configuring the Cisco Nexus 2000 Fabric Extender: 

To configure the Cisco Nexus 2148,  the cisco Nexus 5000 requires Cisco NX-OS Software Release 4.0(1a)N2(1).

All configuration are done on the Nexus 5K.

First Step is to create the Fabric Extender instance..

 

Tulip-Tech-N5K-Access1(config)# fex 110
Tulip-Tech-N5K-Access1(config-fex)# description FEX0110
Tulip-Tech-N5K-Access1(config-fex)# pinning max-links 4
Chage in Max-links will cause traffic disruption.

Tulip-Tech-N5K-Access1(config-fex)# exit

Pinning max-links command   binds 48 downlinks to  uplinks, here all 48 ports can use all 4 uplinks. 

Second step is to Associate the Fabric Extender to a Nexus 5000 ethernet interface(static pinning)

Tulip-Tech-N5K-Access1(config)# interface ethernet 1/7
Tulip-Tech-N5K-Access1(config-if)# switchport mode fex-fabric
Tulip-Tech-N5K-Access1(config-if)#no shut
Tulip-Tech-N5K-Access1(config-if)#fex associate 110

 

We can also associate the Fabric Extender to a Cisco Nexus 5020 port channel (load balancing). Most of the time we will be doing this kind of configs.. here’s how a full Fex Config looks like on a Port-channel..

Tulip-Tech-N5K-Access1(config)# interface ethernet 1/29
Tulip-Tech-N5K-Access1(config-if)# switchport mode fex-fabric
Tulip-Tech-N5K-Access1(config-if)#no shut
Tulip-Tech-N5K-Access1(config-if)#fex associate 110
Tulip-Tech-N5K-Access1(config-if)#channel-group 110
Tulip-Tech-N5K-Access1(config)# interface ethernet 1/30
Tulip-Tech-N5K-Access1(config-if)# switchport mode fex-fabric
Tulip-Tech-N5K-Access1(config-if)#no shut
Tulip-Tech-N5K-Access1(config-if)#fex associate 110
Tulip-Tech-N5K-Access1(config-if)#channel-group 110
Tulip-Tech-N5K-Access1(config)# interface port-channel110
Tulip-Tech-N5K-Access1(config-if)# switchport mode fex-fabric
Tulip-Tech-N5K-Access1(config-if)#no shut
Tulip-Tech-N5K-Access1(config-if)#fex associate 110

How to verify Fabric Extenders or FEX…

Tulip-Tech-N5K-Access1# sh int port-channel 110 fex-intf  

Fabric FEX
Interface Interfaces
—————————————————
Po110 Eth110/1/48 Eth110/1/47 Eth110/1/46 Eth110/1/45
Eth110/1/44 Eth110/1/43 Eth110/1/42 Eth110/1/41
Eth110/1/40 Eth110/1/39 Eth110/1/38 Eth110/1/37
Eth110/1/36 Eth110/1/35 Eth110/1/34 Eth110/1/33
Eth110/1/32 Eth110/1/31 Eth110/1/30 Eth110/1/29
Eth110/1/28 Eth110/1/27 Eth110/1/26 Eth110/1/25
Eth110/1/24 Eth110/1/23 Eth110/1/22 Eth110/1/21
Eth110/1/20 Eth110/1/19 Eth110/1/17 Eth110/1/16
Eth110/1/13 Eth110/1/9 Eth110/1/14 Eth110/1/15
Eth110/1/10 Eth110/1/12 Eth110/1/11 Eth110/1/18
Eth110/1/7 Eth110/1/8 Eth110/1/6 Eth110/1/5
Eth110/1/3 Eth110/1/4 Eth110/1/1 Eth110/1/2

Tulip-Tech-N5K-Access1#

Tulip-Tech-N5K-Access1# sh fex detail

FEX: 110 Description: Tulip-Tech-N5K-Access1-fex110 state: Online

FEX version: 5.2(1)N1(1) [Switch version: 5.2(1)N1(1)]
FEX Interim version: 5.2(1)N1(1)
Switch Interim version: 5.2(1)N1(1)
Extender Serial: SSI16XXXXX
Extender Model: N2K-C2248TP-1GE, Part No: 73-13232-01
Card Id: 99, Mac Addr: 64:d8:14:ZZ:XX:YY, Num Macs: 64
Module Sw Gen: 12594 [Switch Sw Gen: 21]
post level: complete
pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth1/29
FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state:
Po110 – Interface Up. State: Active
Eth1/29 – Interface Up. State: Active
Eth1/30 – Interface Up. State: Active
Fex Port State Fabric Port
Eth110/1/1 Up Po110
Eth110/1/2 Up Po110
Eth110/1/3 Up Po110
Eth110/1/4 Down Po110
Eth110/1/5 Down Po110
Eth110/1/6 Down Po110
Eth110/1/7 Down Po110
Eth110/1/8 Up Po110
Eth110/1/9 Up Po110
Eth110/1/10 Up Po110
Eth110/1/11 Down Po110
Eth110/1/12 Down Po110
Eth110/1/13 Down Po110
Eth110/1/14 Down Po110
Eth110/1/15 Up Po110
Eth110/1/16 Up Po110
Eth110/1/17 Up Po110
Eth110/1/18 Down Po110
Eth110/1/19 Down Po110
Eth110/1/20 Down Po110
Eth110/1/21 Down Po110
Eth110/1/22 Up Po110
Eth110/1/23 Up Po110
Eth110/1/24 Up Po110
Eth110/1/25 Down Po110
Eth110/1/26 Down Po110
Eth110/1/27 Down Po110
Eth110/1/28 Down Po110
Eth110/1/29 Down Po110
Eth110/1/30 Down Po110
Eth110/1/31 Down Po110
Eth110/1/32 Down Po110
Eth110/1/33 Down Po110
Eth110/1/34 Down Po110
Eth110/1/35 Down Po110
Eth110/1/36 Down Po110
Eth110/1/37 Down Po110
Eth110/1/38 Down Po110
Eth110/1/39 Down Po110
Eth110/1/40 Down Po110
Eth110/1/41 Down Po110
Eth110/1/42 Down Po110
Eth110/1/43 Down Po110
Eth110/1/44 Down Po110
Eth110/1/45 Down Po110
Eth110/1/46 Down Po110
Eth110/1/47 Down Po110
Eth110/1/48 Up Po110
Logs:
05/26/201304:24:06.19842: Module register received
05/26/201304:24:06.21062: Registration response sent
05/26/201304:24:06.199827: Module Online Sequence
05/26/201304:24:14.829097: Module Online

Tulip-Tech-N5K-Access1# sh int fex-fabric

Fabric Fabric Fex FEX
Fex Port Port State Uplink Model Serial
—————————————————————
110 Eth1/29 Active 1 N2K-C2248TP-1GE SSI111111JX
110 Eth1/30 Active 2 N2K-C2248TP-1GE SSI222222JX

I hope this helps..

I have received few emails asking which book covers the most of these topics.

I have enjoyed reading and certainly can recommend
NX-OS and Cisco Nexus Switching: Next-Generation Data Center Architectures (2nd Edition) (Networking Technology) By Ron Fuller.