References

• Cisco MDS 9513 linecard power consumption and BTU generation detail (Cisco.com documentation)
  
  
• Cisco BTU listings confirm 80% AC/DC power supply efficiency.
  - The 48 port Cisco MDS linecard is listed as 185 watts DC and 790 BTU.
  - There are 3.413 BTU per AC watt.
  - 790 BTU / 3.413 = 231.5 watts AC.
  - 185 watts DC / 231.5 watts AC = 0.799 = 79.9% AC/DC conversion efficiency.
  
  
• Cisco MDS 9513 "certified for operation" 6500 ft / 2000 meter altitude limit vs 10,000 ft / 3000 m "designed and tested" altitude.
  
  
• Cisco "Day In The Life" MDS 9513 architecture discussion regarding 48 Gbit/sec of active crossbar switch bandwidth and 48 Gbit/sec of passive crossbar bandwidth per linecard module.
  "All current linecard modules utilize crossbar switch fabric capacity with 1:1 redundancy. That is, there are always one active channel per crossbar and one standby channel. One active channel and one standby channel are used on each of crossbar switch fabrics, resulting in both crossbars being active (1:1 redundancy). In the event of a crossbar failure, the standby channel on the remaining crossbar becomes active, resulting in an identical amount of active crossbar switching capacity regardless of whether the system is operating with one or two crossbar switch fabrics."
  
  
• Cisco MDS 9513 252 port restriction
  "Using any combination of modules that include a Generation 1 module or a Supervisor-1 module limits the port index availability to 252 on all Cisco MDS 9500 Series directors. Generation 1 modules also require contiguous port indexes where the system assigns a block of port index numbers contiguously starting from the first port index reserved for the slot that the module is inserted in. This means that while there may be enough port indexes available for a Generation 1 module, the module may not boot up because the available port indexes are not in a contiguous range or the contiguous block does not start at the first port index for a given slot."
  
  
• Cisco MDS power draw vs Brocade
  384 port MDS 9513 = 3337 watts 384 port Brocade 48000 = 1127 watts 384 port Brocade DCX = 1337 watts
  3337 - 1127 = 2210 = 2.21 kWh 3337 - 1337 = 2000 = 2.0 kWh
  2.21 kWh * 24 hours/day * 365 days/year = 17721 kWh/year = 1477 kWh/month
  National averages are available at http://www.utilipoint.com/issuealert/print.asp?id=1728. Averages vary by region and may seem low, but if you use A/C, or if you donÕt have natural gas for heating, dryer, hot water heater, you will be above average.
  An average home in NY uses about 6600 kWh per year.
  
  
• Report of the Independent Electrical Contractor
  "I have audited the results presented in this report including a careful review of the power measurements. In my opinion, and as a representative of Emerson Electrical Reliability Service, I certify that the results presented are accurate and based on a sound and repeatable test methodology."
  
  Russell Goldbeck, Electrical Contractor
  Emerson Electrical Reliability Service
  6900 Koll Center Pkwy. Suite 415
  Pleasanton, California 94566
  
  
• MDS 9500 shuts down in 250 seconds if fan tray is removed in SAN-OS 3.x
  
  Symptom: If you remove a fan tray module from an MDS 9500 series switch that is running Cisco MDS SAN-OS Release 3.0(1), 3.0(2), 3.0(2a) 3.0(2b), 3.0(3), 3.1(1), 3.1(2) or 3.1(2a), the switch shuts down if you do not replace the fan tray module within 170 seconds. (In all other SAN-OS releases, you have 250 seconds to replace it.)
  
  Workaround: None.
  
  Cisco MDS 9513 side-to-side cooling and single fan FRU and five minutes or less time limit to replace fan tray
  "The Cisco MDS 9000 Family switches have internal temperature sensors that are capable of shutting down the system if the temperature at different points within the chassis exceed certain safety thresholds. To be effective, the temperature sensors require the presence of airflow; therefore, in the event a fan module is removed from the chassis, the Cisco MDS 9000 Family switches will be shut down after five minutes to prevent potentially undetectable overheating. However, the switches will be shut down sooner if the higher-level temperature threshold is exceeded."

Cisco MDS 9513 Five Minute Countdown

"If the OperStatus is absent, the fan module has been removed. As soon as the fan module is removed, Cisco SAN-OS starts a five-minute countdown.

Caution: If the fan module is not reinserted within five minutes, the entire switch is shutdown.

Software reads a byte on the SEEPROM to determine if the fan module is present. If the fan module is partially inserted or if software is unable to access the SEEPROM on the fan module for any other reason, then Cisco SAN-OS cannot distinguish this case from a real fan module removal. The switch will be shut down in five minutes."

• Cisco MDS 9513 minimum clearance required between devices
  "If installed in an open rack (no side panels), the horizontal distance required between the chassis and any devices that exhaust air towards the chassis is a minimum of 12 in. (304 cm), and the distance required between the chassis air vents and any walls is a minimum of 6 in. (15.2 cm)."
  
  
• Cisco MDS 9513 carbon emissions

  CO₂ emmissions are estimated by taking kWh (kilowatt hours) used and multiplying by 0.42. An MDS 9513 with 384 ports (5 48-port linecards and 5 24-port linecards) draws 3337 watts according to Cisco documentation.
  
  3337 watts / 1000 = 3.337 kilowatt-hours
  = 3.337 kWh * 24 hours/day * 365 days/year = 29232 kWh per year
  = 29232 kWh/yr * 0.42 kg CO₂/kWh ) = 12279 kg / 1000 = 12.3 metric tonnes of CO₂ per year.
  
  

• Cisco power efficiency claims:

  In slides 19-20 of the "Cisco "Architecting The Green Data Center" webcast (registration required) and in the accompanying PDF, Cisco compared a 120 port 9513 (2 x 12 port linecards, 2 x 48 port linecards) against a what is clearly a Brocade 48000 (128 ports, 2 x 16 port 4Gb blades and 3 x 32 port 4Gb blades):

  Slide 19 from Cisco "Architecting The Green Data Center

  Slide 20 from Cisco "Architecting The Green Data Center

  While Cisco may try to claim that they meant older Brocade directors, note that a 32 port FC blade was not available from Brocade until the release of the 48000. Only 16 port FC blades were available for the Brocade 12000 or 24000. "CPs" are a Brocade-specific term. The images clearly depict Brocade 48000s and Brocade 7500s.

  Note that Cisco claims:

  • the Brocade CP blade draws 100 watts, when it actually only draws 63 watts DC
  • the Brocade 32 port blade draws 90 watts, when it actually only draws 53 watts DC
  • the Brocade 16 port blade draws 90 watts, when it actually only draws 33 watts DC
  • a Brocade 48000 draws 750 watts AC, when it actually draws about 600 watts AC, overestimating by 25%
  • a Brocade 7500 FC router draws 600 watts when it actually draws 150 watts AC, overestimating by 400%.

  Cisco repeated these claims in the Cisco-funded ESG "Building Power Efficient Solutions With Cisco MDS 9000 Directors" whitepaper However, in a related Cisco Data Center blog entry, ESG noted and Cisco confirmed that ESG did not do actual lab testing on the configurations (in the 2nd and 3rd comments).
  
  

• Cisco MDS oversubscribed 24 and 48 port line cards with only 12 port available for 4Gb: "All ports in a 24-port 4-Gbps switching module can operate in dedicated mode with a 2-Gbps operating speed. However, if you configure one or more ports to operate in 4-Gbps dedicated mode, some of the other ports in the module would have to operate in shared mode."
  
  "All ports in a 48-port 4-Gbps switching module can operate in dedicated mode with a 1-Gbps operating speed. However, if you configure one or more ports to operate in 2-Gbps or 4-Gbps dedicated mode, some of the other ports in the module would have to operate in shared mode."

  "ISL ports cannot operate in shared rate mode."

  Ports needing dedicated bandwidth require other ports to be taken out of service:

  "Symptom
  Cannot configure port in dedicated mode.
  Possible Cause
  Not enough bandwidth is available in the port group.
  Solution
  Step 1: Right-click the module and select Show Port Resources.... to display the Generation 2 module shared resources configuration.
  In this example, there is not enough available shared bandwidth in Port-Group 1 to switch any more ports to 4-Gbps dedicated mode.
  Step 2: Do one of the following to free up bandwidth for the port that you want to place in dedicated mode.
  a. Right-click one or more ports and choose Service > Out to put a port in out-of-service mode to free up more resources.
  b. Right-click a port and select Configure. Lower the port speed."

  Port Group Bandwidth Limit

  "Port Groups: Each module has four groups of one or more ports that have a combined bandwidth of up to 12.8 Gbps.
  
  "Number of Ports per Port Group:
  48-port 4Gbps - 12 Ports per Port Group
  24-port 4Gbps - 6 Ports per Port Group
  12-port 4Gbps - 3 Ports per Port Group"
  

  Shared Mode on 4Gbps modules does not actually reserve 4Gbps per port:

  "Module Type: 24-port 4-Gbps
  Port Speed: Auto / 4 Gbps
  Bandwidth: 1 Gbps
  
  "Module Type: 48-port 4-Gbps
  Port Speed: Auto / 4 Gbps
  Bandwidth: 0.8 Gbps"
  

  Out Of Service Interfaces:

  "You can take interfaces out of service to release shared resources that are needed for dedicated bandwidth. This feature is especially useful for the 48-port 4-Gbps switching modules. When an interface is taken out of service, all shared resources are released and made available to the other interface in the port group or module."
  
  "Caution: If you need to bring an interface back into service, you might disrupt traffic if you need to release shared resources from other interfaces."
  
  

• Cisco MDS 9513 racking restrictions

  "MDS 9513 - Telco (2-Post) - Do not use"*

  "Two-post telco racks are not intended for use with the Cisco MDS 9513 Director."*

  "Cisco MDS 9513 chassis: 24.5 inches (62.2 cm) or 14 RU, height with required rack mount support is 15 RU." *
  
  "The rack-mount support brackets provided with the Cisco MDS 9513 Director require an additional height of 0.75 inches (1.9 cm). They are required during the installation of the Cisco MDS 9513 Director and cannot be removed."*
  
  

• Cisco PortChannels and load balancing

  Two mechanisms support the load balancing functionality:
  Flow based - All frames between source and destination follow the same links for a given flow. That is, whichever link is selected for the first exchange of the flow is used for all subsequent exchanges.
  Exchange based - The first frame in an exchange picks a link and subsequent frames in the exchange follow the same link. However, subsequent exchanges can use a different link. This provides more granular load balancing while preserving the order of frames for each exchange.
  
  

• Host-Optimized Ports on the MDS 9120 and 9140

  "The 32-port 1/2-Gbps switching module contains 8 port groups of 4 ports each. When configuring these modules or the host-optimized ports in the Cisco 9100 Series, the following guidelines apply:
  - Any (or all) full line rate port(s) in the Cisco MDS 9100 Series can be included in a PortChannel.
  - The host-optimized ports in the Cisco MDS 9100 Series are subject to same PortChannel rules as 32-port switching modules--only the first port of each group of 4 ports is included in a PortChannel.
  - You can configure only the first port in each 4- port group (for example, the first port in ports 1-4, the fifth port in ports 5-8 and so on) as an E port. If the first port in the group is configured as a PortChannel, the other three ports in each group (ports 2-4, 6-8 and so on) are not usable and remain in the shutdown state.
  - If any of the other three ports are configured in a no shutdown state, you cannot configure the first port as a PortChannel. The other three ports continue to remain in a no shutdown state.
  
  In the Cisco MDS 9100 Series, the left most groups of ports outlined in white (4 ports in the 9120 switch and 8 ports in the 9140 switch) are full line rate like the 16-port switching module. The other ports (16 ports in the 9120 switch and 32 ports in the 9140 switch) are host-optimized like the 32-port switching module. Each group of 4 host-optimized ports have the same rules as for the 32-port switching module."
  
  

• Cisco OEMs QLogic

  "Working its way downmarket, Cisco Systems Inc has begun OEMing a low-end SAN switch from QLogic Corp. The box is a 20-port 4Gbit Fibre Channel device, which will be branded as the Cisco MDS 9020."
  
  

• Cisco Enterprise Package License, required for Inter-VSAN Routing.
  
  
• MDS 9124 Power Draw

  Power and Cooling
  Power supplies (300W AC) (maximum of two per switch)
  Input: 100 to 240 VAC nominal (+/-10% for full range)
  Input current maximum 20A
  Input current steady state
  4A @ 110 VAC
  2A @ 220 VAC
  50 to 60 Hz nominal (+/-3 Hz for full range)
  Output: 300W @ 100 VAC
  300W @ 220 VAC
  
  

• Rising Electricity Costs

  "During the first 6 months of this year, average U.S. residential electricity prices rose by about 11 percent compared with the same period last year. Electricity prices surged for two reasons. First, the costs of fuels for electricity generation have risen. For example, coal and natural gas, which account for about 68 percent of the electricity generated, increased in price by 11 percent and 12 percent on average, respectively, during the first half of 2006. Second, retail electricity price caps have been recently loosened in some States, particularly in New England and the South Atlantic region, as a result of restructured electricity markets. We still expect prices to increase further during the forecast period, but at a slower rate, as moderation in natural gas and coal fuel costs are passed through to retail customers. In 2005, residential electricity prices rose an estimated 5.1 percent nationally. In 2006, these prices are expected to increase by 8.9 percent and, in 2007, by another 3.6 percent."
  
  

• QLogic 5602 oversubscription calculation
  
  16 * 4 Gbit/sec FC ports = 64 Gbit/sec divided by 10Gbit/sec across an ISL = 6.4:1



• Brocade and Microsoft Simple SAN

  Brocade has two Microsoft Simple SAN bundles with the EMC CLARiiON AX150.

  While Cisco is listed as a general partner, they do not have any Microsoft Simple SAN bundles.
  

• Fibre Channel Frame, Sequence and Exchange
- frame: 2112 bytes
- sequence: up to 65,536 frames
- exchange: up to 1024 sequences