Director Bandwidth

Architecture | Power and Cooling | Local Switching | Trunking | Oversubscription | IP Distance | Power Calculator

Why is director slot bandwidth important?

Modular Fibre Channel products allow high-bandwidth scalability by connecting blades or line cards in slots with central switching components (crossbars or control processors) across a high-bandwidth backplane or midplane. Each vendor has unique terms for the part or parts used for these port and control components:

However, not all products are created equal — higher bandwidth provides more room for growth, something critical when considering the high bandwidth demands of virtualized servers, tape backup and trunks to other switches. As a result, some ports are more capable than others.

How does Brocade compare?

Brocade's DCX Backbone architecture provides more bandwidth than any other product in the industry. The Brocade DCX provides 256 Gbit/sec of usable bandwidth between each port blade and the core switching blades. On each of the eight (8) ports blades, up to thirty-two (32) full-speed 8 Gbit/sec devices and ISLs (or forty-eight (48) 4 Gbit/sec devices) can simultaneously communicate across the backplane to the core switching blades for a total of 2048 Gbit/sec (2Tbits/sec) of backplane bandwidth. (Note that the DCX has two distinct CP blades and two distinct core switching blades.) Additionally, Brocade's exclusive local switching technology provides even more full-speed 8 Gbit/sec bandwidth for neighboring ports on the same blade, up to 384 Gbit/sec per blade for a total of 3 Tbit/sec of usable local switching bandwidth per chassis. In addition to the 3 Tbit/sec of core and local switching bandwidth, the DCX provides an additional 512 Gbit/sec of ICL connectivity to another DCX, the equivalent of 64 × 8 Gbit/sec ISLs. In other words, instead of 128 ports sacrificed for ISLs, 128 more devices can be attached to two DCXs connected by ICLs.

The Brocade 48000 provides 64 Gbit/sec of usable bandwidth between each port blade and the control processors. On each of the eight (8) ports blades, up to sixteen (16) full-speed 4 Gbit/sec devices and ISLs (or 32 2 Gbit/sec devices) can simultaneously communicate across the backplane to the control processors for a total of 512 Gbit/sec of backplane bandwidth. Additionally, Brocade's exclusive local switching technology provides even more full-speed 4 Gbit/sec bandwidth for neighboring ports on the same blade, up to 192 Gbit/sec per blade. Any Brocade 48000 can also take 8 Gbit/sec blades with a non-disruptive firmware upgrade. With 64 Gbits/sec per slot, 8 ports can communicate at full-speed 8 Gbit/sec to blades in other slots while up to 40 ports locally switch at full-speed 8Gbit/sec. A Brocade 48000 with 8 Gbit/sec blades can also build a 64 Gbit/sec-wide trunk to a DCX or another 48000.

You can learn more about Brocade director design by reading the Brocade 48000 Architecture white paper and as can be seen in the online port and power calculator.

The Brocade M6140 provides approximately 8 Gbit/sec of usable bandwidth between port modules and crossbars. Each of the thirty-five (35) port modules has four ports which can support either 2 Gbit/sec or 4 Gbit/sec ports for a total of 280 Gbit/sec of crossbar bandwidth.

The Brocade Mi10000 can provide 64 Gbit/sec of usable port bandwidth between up to eight (8) line modules (LMQs) and up to four (4) switching modules (SWXs) for a total of 512 Gbit/sec of crossbar bandwidth.

The Cisco MDS 9513 has two possible slot speeds -- 48 Gbit/sec with the original active/passive Gen1 fabric crossbars, or 96 Gbit/sec with the Gen2 fabric crossbars (released in Q4 2008). The 8 Gbit/sec-capable 24 and 48-port "performance" linecards require "Cisco MDS NX-OS" (aka SAN-OS) 4.1 as well as the Gen2 fabric crossbars. The 4/44 "host optmized" linecards only require "Cisco MDS NX-OS" 4.1. However, there are significant restrictions on 8 Gbit/sec ports due to the architecture of the linecards and the internal switching architecture.

The original Cisco MDS 9513 with Gen1 fabric crossbars provides 48 Gbit/sec of usable bandwidth between linecards and Gen1 crossbars.

• On each of the 11 user slots, no more than twelve (12) full-speed 4 Gbit/sec devices or trunks can communicate across the crossbar since Cisco 4Gbit/sec modules have only 4 port groups of 12.8 Gbit/sec each.
• 12.8 × 4 = a theoretical 51.2 Gbit/sec per slot.
• As a result, even on the 24- and 48-port MDS linecards, only 12 ports of full-speed 4 Gbit/sec are available — on the 48-port linecard, when 12 ports are locked at 4Gb, the other 36 ports only have access to 3.2 Gbit/sec that the slot can provide.*
• While the 4/44-port "Host Optimized" 8-Gbps linecard can be used in a 9513 with a Gen1 fabric crossbar, only 4 of the 48 ports can be run at 8 Gbit/sec speeds (since there are only 4 × 12.8 port groups on the 4/44 linecard).
• If an 8 Gbit/sec ports is active, only a single 4 Gbit/sec port can be active in that port group (since 12.8 - 8 = 4.8). In other words, an MDS with Gen1 fabric crossbars can only support [44 × 8 Gbit/sec + 44 × 4 Gbit/sec] ports at once.

A Cisco MDS 9513 with Gen2 fabric crossbars provides 96 Gbit/sec of usable bandwidth between linecards and Gen2 crossbars. However, not all of the 96Gb can be used for 8 Gbit/sec ports.

• The 24 and 48 port 8Gb "performance" linecards have 8 ports groups of 12.8Gb each, meaning only a single 8 Gbit/sec port can be in each port group.
• 12.8 × 8 = a theoretical 102.4 Gbit/sec per slot.
• Bandwidth cannot be shared between port groups, and even local 8 Gbit/sec traffic must pass through the fabric crossbars.
• As a result, an MDS 9513 can have no more than 88 ports of 8Gb active at once.
• If an 8 Gbit/sec ports is active, only a single 4 Gbit/sec port can be active in that port group (since 12.8 - 8 = 4.8). In other words, an MDS with Gen2 fabric crossbars can only support [88 × 8 Gbit/sec + 88 × 4 Gbit/sec] ports at once.
• The Gen2 fabric crossbars do allow 24 ports of full-speed 4 Gbit/sec per slot if no ports are set at 8Gbit/sec.

A 9513 with Gen1 fabric crossbars can be upgraded from 48Gbit/sec per slot to 96Gbit/sec by installing Gen2 fabric crossbars and upgrading the firmware to "Cisco MDS NX-OS" 4.1. However, in order to take advantage of the 96Gb per slot (i.e. making the new crossbars active-active), the 9513 requires a disruptive switch reset.

The Cisco director design has more components per linecard and an active midplane, as opposed to Brocade's efficient integrated ASIC architecture and passive backplane design. As a result, Cisco directors require considerably more power and generate much more heat.

Data flow within Brocade and Cisco products differs considerably.

At 64 Gbit/sec, the Brocade 48000 and Mi10000 have 33% more backplane bandwidth per slot than an Gen1 MDS 9513. At 256 Gbit/sec, the Brocade DCX has two and a half (2.5) times the bandwidth per slot than the gen2 MDS 9513, and can support three to four times the number of 8Gbit/sec ports per chassis.

In addition, both the Brocade DCX and 48000 support local switching and is thus able to save that valuable bandwidth for data that must cross the backplane to get to another blade. In contrast, any time data enters a Cisco port, it always crosses the backplane and consumes backplane bandwidth even if traffic is local to the linecard or module.

This architectural difference has significant consequences. Whereas each Brocade ASIC can make forwarding decisions, the Cisco crossbar design requires that the forwarding ASIC confirm with the central arbitor on the supervisor before sending packets through the crossbar to the receiving ASIC even if ports are on the same linecard. Since Cisco MDS linecards are limited to 12.8 port groups, no more than 3 ports of full-speed 4 Gbit/sec can be active at once. If there are more than 3 ports in that port group, they will be fighting for bandwidth. As previously noted, the 12.8Gb limit forces bandwidth contention that becomes more obvious when 8 Gbit/sec is needed.

Note that Cisco will often double bandwidth numbers in a marketing effort to overcome their technical limitations. They will often reference the 96 Gbit/sec per slot in the 9513 as "192Gbit/sec". These "full duplex" or "cross-sectional" bandwidth measurements need be compared to the port speed in order to make equivalent measurements. However, no matter how many times you double the bandwidth number, the MDS 9513 supports just 8 full-speed 8 Gbit/sec ports per slot, and only 88 full-speed 8 Gbit/sec ports per chassis, a fraction of the 48 full-speed 8 Gbit/sec ports per slot and 384 full-speed 8 Gbit/sec ports in each DCX. While the Gen2 9513 can support up to 264 full-speed 4 Gbit/sec ports before it runs out of bandwidth, the DCX can support 256 ports of full-speed 4 Gbit/sec ports plus another 128 ports of 8 Gbit/sec. On top of that, two DCXs can be attached using 512 Gbit/sec worth of dedicated ICLs. Another way to consider 512 Gbit/sec is 64 × 8 Gbit/sec ports or 128 × 4 Gbit/sec ports on each chassis that can be used for hosts or storage rather than ISLs.

Brocade's architecture allows SAN administrators to avoid wasting time managing port bandwidth and focus on more important tasks.

Click here for references.