In this SAN and NAS Storage tutorial, I cover the SAN protocol Fibre Channel over Ethernet (FCoE). FCoE became possible with the advent of 10Gb Ethernet which has enough bandwidth to support both data and storage traffic on the same adapter. Scroll down for the video and also text tutorial.
Want the complete course? Click here to get the Introduction to SAN and NAS Storage course for free
FCoE Fibre Channel over Ethernet – Video Tutorial
I work in education and we are currently having issues with our SAN cluster. Although I’m an entry level tech, this course has definitely provided me with a wealth of knowledge and resources to understand the technology as well as make suggestions to the storage admins.
I recommend you check my Fibre Channel Overview series first to get a good grounding in overall Fibre Channel SAN basics before moving on to FCoE. The series of three videos are available here:
What Is Fiber Channel over Ethernet (FCoE)?
Fibre Channel over Ethernet, or FCoE, is a merger of ethernet and fibre channel. In the past, companies used ethernet strictly to share information among devices in their networks (LAN) and they mainly relied on fibre channel for data storage (SAN).
With FCoE, it became possible for Fibre Channel frames to be encapsulated and transported over Ethernet links. Both of the traffic passes through the same physical cable but they are still able to be separated on the same hardware.
Fibre Channel and FCoE Comparison
To understand FCoE, it is essential to understand that FCoE uses FCP, the Fibre Channel Protocol. The exact same Fibre Channel Protocol that is used with native Fibre Channel. FCoE works exactly the same way as native Fibre Channel.
We still have WWPNs on the initiator and target, and we still use the FLOGI, PLOGI, and PLRI login process. The difference is that we encapsulate that same FCP traffic inside an Ethernet header so we can run it over an Ethernet network, rather than over a native Fibre Channel network.
FCoE retains the reliability and performance of native Fibre Channel. Quality of Service (QoS) is used to guarantee the required bandwidth for the storage traffic. It ensures the normal data traffic does not hog too much bandwidth.
What is the Primary Benefit of the FCoE
One of the main benefits we get from FCoE is the savings we get in our network infrastructure. When we have redundant native Fibre Channel storage and Ethernet data networks, there are 4 adapters on our hosts and 4 cables that are connected to 4 switches.
With FCoE, we run the data and storage traffic through shared switches and shared ports on our hosts. Now we just have 2 adapters, 2 cables, and 2 switches. The required infrastructure is cut in half. We save on the hardware costs because of this and also require less rack space, less power, and less cooling, which gives us more savings.
Advantages of FCoE
Aside from saving money from short and long-term perspectives, FCoE also allows organizations to take advantage of all the strong suits of ethernet and fibre channel in a single network.
FCoE combines the speed, reliability, and scalability of ethernet with the increased security, efficiency, and faster backup recovery capacity of fibre channel.
With FCoE, we are capable to use converged infrastructure which gives us the following benefits:
- Reduced number of Network Interface Cards (NIC), adapters, cables, and network devices.
- Unified Input/Output (I/O) ports.
- Simplified Network Management.
- Reduced Power and Cooling costs.
- Increased System Flexibility.
In FCoE, both the storage and the data traffic uses the same shared physical interface on our hosts – the Converged Network Adapter (CNA). The CNA replaces the traditional Network Interface Card (NIC) in the host. (See the SAN and NAS Adapter Card Types post for a quick review.)
The storage traffic uses FCP so it requires a WWPN. The data traffic requires a MAC address. The way that Ethernet data traffic and FCP storage traffic works is totally different so how can we support them both on the same physical interface?
The answer is we virtualize the physical interface into two virtual interfaces: a virtual NIC with a MAC address for the data traffic and a virtual HBA with a WWPN for the storage traffic. The storage and the data traffic are split into two different VLANs, a data VLAN and a storage VLAN.
In the diagram below we have a single server, Server 1. It’s got two physical interface cards, CNA1 and CNA2. Both CNAs are split into separate virtual adapters for data and storage.
For the data traffic, we've got virtual NIC-1 on CNA1, and virtual NIC-2 on CNA2. Those virtual NICs will both have MAC addresses assigned to them. On the switches, we're trunking the data VLAN down to the physical part on the CNA. We cross connect our switches for the data VLAN traffic.
We also have virtual HBAs on the CNAs, we've got virtual HBA-1 on CNA1 and virtual HBA-2 on CNA2. We have WWPNs on our virtual HBAs. We're trunking the storage VLAN down from the switches to the virtual HBAs on the converged network adapters.
This time we do not cross-connect our switches because we need to comply with SAN best practice of physically separate Fabric A and Fabric B.
If we put the whole thing together, you can see that we're running both our data and our storage traffic over the same shared infrastructure.
We’re trunking both the data and the storage VLANs down to single physical parts on our CNAs, and then that traffic is split out into a virtual NIC for the data traffic with our MAC address on it, and a virtual HBA for the storage traffic with a WWPN on it.
Another thing we need to discuss is lossless FCoE. Fibre Channel is a lossless protocol, meaning it ensures that no frames are lost in transit between the initiator and the target. It uses a buffer to buffer credits to do this.
Ethernet is not lossless. TCP uses acknowledgments from the receiver back to the sender to check that traffic reaches its destination. If an acknowledgment is not received then the sender will resend that packet.
FCoE uses FCP which assumes a lossless network, so we need a way to ensure our storage packets are not lost while traversing the Ethernet network. Priority Flow Control (PFC) which is an FCoE extension for Ethernet, is used to ensure that lossless delivery.
PFC works on a hop by hop rather than end to end basis, so we don't just need to support it on our end hosts. Each NIC and switch in the path between initiator and target must be FCoE capable. You can't use just standard 10Gb Ethernet NICs and switches, you need to use CNAs and the switches have to be FCoE capable.
Free ‘Introduction to SAN and NAS Storage’ training course
The video is an excerpt from my Introduction to SAN and NAS Storage training. You can get the entire series for free here: