Creating a 40Gb / 10Gb Home Network on a $1000 Budget
Description: While 100Gb networks are becoming more affordable just recently, older 10Gb and 40Gb have been steadily declining in price. Just recently I revamped my home network by jumping knee-deep into 40Gb.
Parts Listed
Diagram:
Explanation of Components
Router:
Supermicro X10SLH-N6-ST031
This router has 3 x520 intel nics on board for a total of 6x 10Gb ethernet ports. They are only 1/10Gb compatible so keep this in mind when purchasing this for the purpose of connecting to a 2.5Gb or 5Gb link. The way to circumvent this issue is to use a NIC with an SFP+ multi-gig adapter that can take a 10Gb signal and downgrade it to 5Gb or 2.5Gb. My OS of choice is Pfsense but Opnsense is also really great, just a personal matter.
Switch:
Cisco 56128P (2u) or 5672UP (1u)
These switches can be used with or without a license. With the routing license installed the switches can be used as a layer 3. I went with the 2u variant, but the 1u is perfectly fine for a home lab. Keep in mind the 2u is 600w normal use.
40Gb NIC
Chelsio T580
This NIC can be had for under $100 and is supported by TrueNAS core (plug-n-play). This NIC is also compatible with Cisco’s DACs.
10Gb NIC
Intel x520
These NICs can be had for less than $50 and are supported by TrueNAS and Pfsense. They are only PCI-e 2.0 at 8X (32Gb/s)so you will want to connect them to an 8x slot on the motherboard. Running this on a 4x slot will run at 16Gb/s and will bottleneck a dual port card. I tested this card with Cisco transceivers and will work with an annoying warning message. I recommend going with an Intel transceiver for this annoying reason.
I also went with MM fiber for the cost but nothing is wrong with either. Just keep in mind if you want to use OM1 like I have it is limited to 10Gb and at short distances.
Conclusion:
Upgrading your homelab to 40Gb can be done relatively cheaply, or for about the cost of a used enterprise 100Gb Switch. 40Gb will allow you to fully saturate the bandwidth capable of Nvme or 8 SATA III SSD. As 40Gb equipment continues to be dumped into the used market in truck loads expect this downward trend of switch cost to continue.
