FCoE Fibre Channel over Ethernet Overview

FCoE Fibre Channel over Ethernet tutorial

In this SAN and NAS Storage tutorial, I cover the SAN protocol Fibre Channel over Ethernet (FCoE). FCoE became possible with the advent of 10Gb Ethernet which has enough bandwidth to support both data and storage traffic on the same adapter. Scroll down for the video and also text tutorial.

Want the complete course? Click here to get the Introduction to SAN and NAS Storage course for free

FCoE Fibre Channel over Ethernet – Video Tutorial

YouTube video

Tejus Netrakar

Tejus Netrakar

It was really good learning for me, everything was very easy to understand and fun as it was direct to the point. Neil is very responsive and replies to your issues as soon as he can. Looking forward for more.

Tejus Netrakar

I recommend you check my Fibre Channel Overview series first to get a good grounding in overall Fibre Channel SAN basics before moving on to FCoE. The series of three videos are available here:

Fibre Channel SAN Part 1 – FCP and WWPN Addressing

What Is Fiber Channel over Ethernet (FCoE)?

 

Fibre Channel over Ethernet, or FCoE, is a merger of ethernet and fibre channel. In the past, companies used ethernet strictly to share information among devices in their networks (LAN) and they mainly relied on fibre channel for data storage (SAN).

 

With FCoE, it became possible for Fibre Channel frames to be encapsulated and transported over Ethernet links. Both of the traffic passes through the same physical cable but they are still able to be separated on the same hardware.

Fibre Channel and FCoE Comparison

 

To understand FCoE, it is essential to understand that FCoE uses FCP, the Fibre Channel Protocol. The exact same Fibre Channel Protocol that is used with native Fibre Channel. FCoE works exactly the same way as native Fibre Channel.

 

We still have WWPNs on the initiator and target, and we still use the FLOGI, PLOGI, and PLRI login process. The difference is that we encapsulate that same FCP traffic inside an Ethernet header so we can run it over an Ethernet network, rather than over a native Fibre Channel network.

 

SAN Protocol Stack Comparison

SAN Protocol Stack Comparison

 

FCoE retains the reliability and performance of native Fibre Channel. Quality of Service (QoS) is used to guarantee the required bandwidth for the storage traffic. It ensures the normal data traffic does not hog too much bandwidth.

What is the Primary Benefit of the FCoE

 

One of the main benefits we get from FCoE is the savings we get in our network infrastructure. When we have redundant native Fibre Channel storage and Ethernet data networks, there are 4 adapters on our hosts and 4 cables that are connected to 4 switches.

 

Native Fibre Channel

Native Fibre Channel

 

 

With FCoE, we run the data and storage traffic through shared switches and shared ports on our hosts. Now we just have 2 adapters, 2 cables, and 2 switches. The required infrastructure is cut in half. We save on the hardware costs because of this and also require less rack space, less power, and less cooling, which gives us more savings.

 

Fibre Channel over Ethernet

Fibre Channel over Ethernet

Advantages of FCoE

 

Aside from saving money from short and long-term perspectives, FCoE also allows organizations to take advantage of all the strong suits of ethernet and fibre channel in a single network.

 

FCoE combines the speed, reliability, and scalability of ethernet with the increased security, efficiency, and faster backup recovery capacity of fibre channel.

 

With FCoE, we are capable to use converged infrastructure which gives us the following benefits:

  • Reduced number of Network Interface Cards (NIC), adapters, cables, and network devices.
  • Unified Input/Output (I/O) ports.
  • Simplified Network Management.
  • Reduced Power and Cooling costs.
  • Increased System Flexibility.

FCoE Networks

 

In FCoE, both the storage and the data traffic uses the same shared physical interface on our hosts – the Converged Network Adapter (CNA). The CNA replaces the traditional Network Interface Card (NIC) in the host. (See the SAN and NAS Adapter Card Types post for a quick review.)

 

The storage traffic uses FCP so it requires a WWPN. The data traffic requires a MAC address. The way that Ethernet data traffic and FCP storage traffic works is totally different so how can we support them both on the same physical interface?

 

The answer is we virtualize the physical interface into two virtual interfaces: a virtual NIC with a MAC address for the data traffic and a virtual HBA with a WWPN for the storage traffic. The storage and the data traffic are split into two different VLANs, a data VLAN and a storage VLAN.

 

In the diagram below we have a single server, Server 1. It’s got two physical interface cards, CNA1 and CNA2. Both CNAs are split into separate virtual adapters for data and storage.

 

For the data traffic, we've got virtual NIC-1 on CNA1, and virtual NIC-2 on CNA2. Those virtual NICs will both have MAC addresses assigned to them. On the switches, we're trunking the data VLAN down to the physical part on the CNA. We cross connect our switches for the data VLAN traffic.

 

FCoE Data VLAN

FCoE Data VLAN

 

We also have virtual HBAs on the CNAs, we've got virtual HBA-1 on CNA1 and virtual HBA-2 on CNA2. We have WWPNs on our virtual HBAs. We're trunking the storage VLAN down from the switches to the virtual HBAs on the converged network adapters.

 

This time we do not cross-connect our switches because we need to comply with SAN best practice of physically separate Fabric A and Fabric B.

 

FCoE Storage VLAN

FCoE Storage VLAN

 

If we put the whole thing together, you can see that we're running both our data and our storage traffic over the same shared infrastructure.

 

We’re trunking both the data and the storage VLANs down to single physical parts on our CNAs, and then that traffic is split out into a virtual NIC for the data traffic with our MAC address on it, and a virtual HBA for the storage traffic with a WWPN on it.

 

FCoE Network

FCoE Network

 

Another thing we need to discuss is lossless FCoE. Fibre Channel is a lossless protocol, meaning it ensures that no frames are lost in transit between the initiator and the target. It uses a buffer to buffer credits to do this.

 

Ethernet is not lossless. TCP uses acknowledgments from the receiver back to the sender to check that traffic reaches its destination. If an acknowledgment is not received then the sender will resend that packet.

 

FCoE uses FCP which assumes a lossless network, so we need a way to ensure our storage packets are not lost while traversing the Ethernet network. Priority Flow Control (PFC) which is an FCoE extension for Ethernet, is used to ensure that lossless delivery.

 

PFC works on a hop by hop rather than end to end basis, so we don't just need to support it on our end hosts. Each NIC and switch in the path between initiator and target must be FCoE capable. You can't use just standard 10Gb Ethernet NICs and switches, you need to use CNAs and the switches have to be FCoE capable.

Additional Resources

 

RFC 6847 - Fibre Channel over Ethernet

Wikipedia - Fibre Channel over Ethernet

Cisco's FCoE Basics

 

Free ‘Introduction to SAN and NAS Storage’ training course

The video is an excerpt from my Introduction to SAN and NAS Storage training. You can get the entire series for free here:

https://www.flackbox.com/introduction-to-san-and-nas-storage

 

When you're ready to take your storage knowledge to the next level, you can get my 'NetApp ONTAP 9 Storage Complete' training course.