Skip to main content

Cisco UCS Mini - Add Extender Chassis

If you happen to own a UCS Mini Setup, a 5108 Chassis with two Fi 6324 or similar, and you are looking for documentation on how to add another 5108 Chassis with fabric extenders (2204XP in my case), then Cisco really does not have much out there, nor is there a lot of googlable information either (Everything you find is related to standalone Fabric Interconnects and "standard" UCS). Even after calling TAC, it took a while to get something, and what they told us was not even accurate. So here is how we did it, and it worked, came up without any interruption to current chassis, network, or running profiles.

Equipment

Of course we used our Cisco vendor to spec the equipment, but just for reference here is the list of what we had and what we added:

  • Original Setup
    • 5108 Chassis 
    • Fi 6324 (Qty 2)
    • Ports 1-2 for Fibre Channel, and 3-4 for Ethernet (MMF)
    • Connected to a stack of switches and pair of FC switches/SAN
    • Running UCS version 4.0.1 (Fairly recently upgraded as of May 2019)
  • Added Equipment
    • 5108 Chassis
    • 2204XP (Qty 2) Fabric Extenders
    • QSFP to 4SFP Breakuout cable (Qty 2)
    • Scalability Port Enablement License for existing Fi6324 (Qty 2) (Allows usage of the QSFP)

Procedure

This ended up being quite simple - it was the uncertainty of doing it in a way that would not impact anything that had us fumbling for documentation. In the end it is pretty painless and straight forward:

  1. Obtain the llicense file from Cisco, we had a PAK imported to the Cisco portal, then  generated Lic files with the FI Serial numbers
  2. Apply the licenses in UCS UI. 
    1. Admin, Licensing
    2. Fabric Interconnect A
    3. Click Download License, Choose the Lic file and off you go
    4. Click on Installed Licenses Tab, you should see the "SCAL_ETH_PORT_ACTIVATION_PKG" or similar with expiry Never
    5. Repeat for Fi B
  3. Rack your Chassis, install fabric extenders (And server blades if you have them).
  4. Wire up the extenders
    1. The QSFP (Scalability port 5) in Fabric A on primary chassis, to Port 1/2/3/4 on the extended chassis XP A (Left)
    2. The QSFP (Scalability port 5) in Fabric B on primary chassis, to Port 1/2/3/4 on the extended chassis XP B (Right)
  5. Plug in power to the extender chassis
  6. Bootup will take a while - give it 10 minutes
    1. In UCS UI, go to Equipment, Click on Fabric Interconnect A node
      1. Select the Physical Ports Tab
      2. Expand the Fixed Module and Scalability Port 5
      3. If these ports are showing As up and enabled you can skip the next step
      4. Right click on each of the 4 ports, one at the time, click "Configure as Server Port"
    2. Go to Equipment, Click on Fabric Interconnect B node
      1. Select the Physical Ports Tab
      2. Expand the Fixed Module and Scalability Port 5
      3. If these ports are showing As up and enabled you can skip the next step
      4. Right click on each of the 4 ports, one at the time, click "Configure as Server Port"
    3. If you did not have to manually configure the ports as server ports above, it may be related to the setting in Equipment, Policies, Port Auto-Discovery, Auto Configure Server port being set to enabled. This was not enabled in our system, and we left it as is and just enabled all the ports manually.
    4. As soon as the ports are enabled, Chassis 2 (extended) should start showing up. It may take quite some time before this is all set, as it performs Firmware upgrades on the fabric extenders and takes care of a lot of things, then does discovery - you will see several warnings and issues for a while till it is all done. Let it sit for an hour probably - after that, things should show up and hopefully be all error free.
    5. At the root of Equipment you should now see your second chassis and be good to go
As you can see, it is quite simple - i just posted this to help anyone having a hard time finding information like we did, not sure how many run Mini's, but can't be all that many.
Here is the simple diagram of how the wiring was done with ethe QSFP Breakouts.













Labels:

Comments

Post a Comment

Popular posts from this blog

Campground Networking

I've travelled a couple years full time in an RV, working remotely. This can be a challenger, many campgrounds have poor Wi-Fi setups, and cell service is not always great (Do not plan on doing any work from the Grand Canyon National Park). Calling ahead and asking usually does not reveal accurate information, you are best off using campground reviews, search them for WIFI and read what people say.  The best network I have seen was at Eagles Landing RV Park in Holt, FL (Pan handle) http://eagleslandingrvpark.com/. Still not perfect. At poor sites, more than once did I offer my assistance in trying to configure and improve, but even the places which have no vendor maintain their system do not want any other hands on it. A couple of times I helped out anyway, default passwords on routers, so I upgraded their firmware, disabled 802.11b, and set a password so no-one else would mess with it. My RV network setup is not of a common type, you sort of have to be a bit of a network-guy to us...
1 comment
Read more

New Lines - Windows/Unix/Linux/MacOS - viM

If you deal with scripts and other text files and move between platforms you probably discovered this "issue". Only the founding developers can explain why they chose what they did - googling about will show you a couple of different explanations - whatever the reasons, here are the differences and how to convert. The formats The Characters in use (referenced in OS info below) LF Usually referred to as LF  or Line feed Ascii code decimal 10 Hex: A or 0xA Octal: 12 or O12 Typical Escaped character in many shells and languages: \n CR Usually referred to as CR or Carriage Return Ascii code decimal 13 Hex D or 0xD Typical Escaped character in many shells and languages:  \r Unix, Linux, and Modern MacOS - The POSIX standard Each Line ends with a single character:  LF Most programming languages will understand/interpret this format properly. Simple Windows programs, like the built in Notepad will not show this properly. Windows (and DOS) Each...
Post a Comment
Read more

Linux/Unix - Create a local Certificate Authority (CA)

I get these questions all the time - people know i have some runtime with certificates and such - one question is "Can't i just issue my own certs?" - and the answer of course is yes - but I always make sure to add that it won't be any use on a public web site since no-one will trust it. So setting up your own CA is not "generally useful", it is more if you need some specific things, like issuing certificates with a single signing source for client logins or similar. Most business will have a  couple of Windows Domain controllers, if you need to sign certs for a limited set of users, what you should do is make sure some system in your windows domain runs Certificate Services, then issue certs from there, make sure any non-domain-members has a trust for that CA. If you actually do need to set up you own CA, here is one way to do it Procedure to set up your own local CA The common name for the CA cert must NOT be the same as a domain name or anything e...
Post a Comment
Read more