2. DIY HOME SERVER - Rack Mounted Case
My basement already contained a 19 inch wallmounted enclosure capable of storing maximal 18 U. This rack is normally used for switches and patchpanels and has an (outside) depth of only 600 mm. Most server chassis have an (inside) depth of 700, 800 or even 900 mm. Because the maximum (inside) depth in my rack is only 500 mm, I had to find a short depth server chassis.
There’s not that much choice in short depth chassis. I selected the INTER-TECH IPC 4U-4408, also being sold as the LOGIC CASE SC-43400-8HS.
This 4U case has 8 hot-swappable 2.5/3.5 inch drive bays, one internal (combined) 2.5/3.5 inch drive bay and has a depth of 480 mm. The 8 drive bays are connected to 2 internal SAS backplanes with SFF-8087 connectors. The case comes with several mounting frames to allow the use of 2U, 2U redundant and 3U Power Supply Units (PSU). Without mountings it fits a standard ATX PSU.
Being a 4U case, it gives you plenty of free height inside the case to install well dimensioned CPU coolers. The case accepts full ATX motherboards. It has LEDS on the front to monitor power, HDD activity, alert status and 3 Network Interface Cards. This comes in very handy because, as you’ll see later on, the selected motherboard has indeed 3 NICs.
The case features one internal (combined) 2.5/3.5 inch drive bay. Because I wanted to mount two SATA boot SSDs inside the case, I ordered an ICY DOCK EZ-Fit Lite MB290SP-1B Dual 2.5″ HDD & SSD Light Weight mounting bracket for internal 3.5″ drive bays.
I’m pleased with this case. It is not a low cost case but had a good quality and is well build.
The only minor point is the air flow inside the case. All larger server cases have one complete row of noisy high powered fans right behind the harddisk bays to assure high flow air pressure over the CPUs and extension cards. This case only has two 80 mm fans at the back of the case. I would not advice you to install power-guzzling CPUs in a dual CPU configuration because things might heat up a little.
Tip :
Although theoretically the case would have been small enough for my 600 mm depth rack, it turned out I could not perfectly install the server case inside the rack because the cables on the back were pushing against the back wall. So, when you ever by a server rack, be sure to buy at least a 700 mm dept rack. For now my case is placed on 2 aluminium L-profiles. This works well enough until I buy a deeper rack.
3. DIY HOME SERVER - Power Supply Unit
Building a server requires choosing reliable hardware. Especially the PSU needs to be stable and power efficient. I chose the CORSAIR RM550X fully modular ATX PSU.
This 550 Watt 80+ Gold labeled PSU uses high-quality Japanese components and comes with a 10 year warranty. Now that’s what I call a statement.
4. DIY HOME SERVER - CPU
Choosing the CPU is the most crucial operation in the build. I wanted to purchase a used server CPU that would have cost a small fortune not all too long ago. I immediately skipped all consumer grade CPUs and looked for a INTEL XEON CPU. I first wanted to create a dual CPU system with just one CPU installed. This way I would easely be able to upgrade the system by adding the second CPU. But the case ventilation prevented me from going down that road.
I then studied the different CPU generations and narrowed the search down to an INTEL E5-26xx v3 or v4. At last I turned to eBay and checked the available CPUs sold by professional server gear resellers. I ended up ordering one INTEL XEON E5-2680 v4. This is a 14 core 28 thread CPU, situated in the middle of the INTEL XEON E5 v4 CPU family.
5. DIY HOME SERVER - Mainboard
The mainboard had to be a server grade FCLGA2011-3 socket board, dictated by the chosen CPU. At first I looked at ASROCK Rack but I ended up buying a SUPERMICRO X10SRi-F. This board supports IPMI and has 3 onboard 1GB NICs, wich would come in handy because I use a separate VLAN for my IOT devices. I bought one used on eBay, again from a professional server gear reseller.
Tip :
If your server case draws the air from the front to the back, it helps when you select a motherboard that has its memory slots beside instead of below and above the CPU.
The mainboard came in an excellent condition and even included a SUPERMICRO SNK-P0050AP4 CPU cooler featuring a SUPERMICRO BKT-0050L-RN narrow ILM mounting bracket. This CPU cooler needs at least a 3U case height.
The mainboard BIOS was up to date but the BMC firmware needed updating.
Tip :
In case you purchased a used motherboard always make sure all jumpers on the mainboard are in their default positions. At first boot, restore the BIOS to optimized defaults, it will revert back to Supermicro default settings.
Tip :
When the server is powered off, the dedicated IPMI NIC is negotiating at 1 GBit and both LAN1 and LAN2 NICs are negotiating at 100 MBit. Once powerd on, all 3 NICs will negotiating at 1 GBit.
6. DIY HOME SERVER - ECC RAM
This one was not that easy to solve myself. Sure, ECC memory was a no-brainer, especially because the OS would use ZFS as a filesystem. Server rated mainboards mostly are a little picky regarding the memory they prefer. The SUPERMICRO X10SRI-F mainboard has a Tested Memory List but was not that helpful. I ended up searching YouTube content using the same mainboard and checking if they used mainstream memory.
I ordered 4 x 16 GB Kingston KSM24RS4/16MEI 2400 MHz ECC memory, being 64 GB in total.
7. DIY HOME SERVER - HBA Controller
On eBay I bought a used DELL PERC H310 HBA Raid Controller. This card is fully LSI Logic 9211-8i compatible, but a lot cheaper.
Because Proxmox uses ZFS as a file system and therefore uses software RAID to address the drives directly, I had to bypass the HBA onboard RAID controller by flashing the HBA controller to IT mode instead of RAID mode. Thanks to TECHMATTR for his excellent post on SAS HBA crossflashing or flashing to IT mode, Dell Perc H200 and H310.
I ordered two INTER-TECH SFF-8087 SAS cables to connect the two SAS backplanes to the HBA controller.
Tip :
This HBA controller is designed for use in server chassis with high flow air pressure. To use this PCIe card in a normal case or a desktop/tower enclosure, you definitely need to add active ventilation. So I renewed the thermal paste on the main chip and installed a small 40 mm NOCTUA fan on the heatsink.
Remark :
The HBA controller has a small connector indicating the connected drives activity. I chose not to connect this output to the case HDD activity LED because this LED was already occupied by the mainboard HDD activity connector indicating the onboard SATA drive activity, in my case being both internaly connected SSDs occupied by Proxmox OS and VM storage. The activity of the drives connected to the HBA controller can be monitored by LEDs directly on the hot-swappable drive enclosures.
8. DIY HOME SERVER - Storage
I use two 500 GB CRUCIAL MX500 SSD, installed internaly in the ICY DOCK EZ-Fit Lite MB290SP-1B enclosure and connected directly to native onboard SATA connectors. These two drives are configured as a software ZFS RAID 1 mirror and contain a 400 GB partition to boot the Proxmox OS and store the VMs . I like these SSDs because they are reliable, cheap and feature integrated Power Loss Protection and Parity Protection.
In the 8 hot-swapable drive bays I use several WESTERN DIGITAL Red and SEAGATE Ironwolf drives in sofware ZFS RAID 1 mirrors (with Hot Spares). All HDDs are connected to the HBA and passed through to the TrueNAS VM. This way the HDDs are not accessible to Proxmox but managed completely by the TrueNAS VM.
In the future I’ll be replacing all WESTERN DIGITAL Red drives in favor of SEAGATE Ironwolf drives.
9. DIY HOME SERVER - GPU
This server build is ment to be used as a headless system. This means it’s only managed by remote control and does not need a keyboard, mouse or monitor. Thanks to the IPMI interface on the mainboard, the system is always manageable by remote access, even when the machine is turned off.
The only reason I integrated a graphics card is PCI pass-through. This way the GPU can be dedicated to certain VMs to be used for hardware accelerated rendering and streaming.
I chose a DELL NVIDIA QUADRO P620 featuring 2 GB DDR5 memory. This PCI Express 3.0 x16 card is capable of transcoding H.264 and H.265 streams, uses only 40 Watt and a Passmark score of 3663.
10. DIY HOME SERVER - 10 Gbps Networking
After upgrading my home netwerk to 2.5 Gbps/10 Gbps, I added a SUPERMICRO Dual Port 10 Gbps Ethernet Adapter SUPERMICRO AOC-STGN-I2S X520-DA2 Rev 2.10 bought on eBay.
This card is connected to one of the two 10 Gbps SPF+ ports on my TP-LINK Omada SG3218XP-M2 switch by a SPF+ Twin Axial Direct Attached Cable (DAC) from fs.com.
Tip :
Be aware the DAC Cable is a custom made version, having an INTEL transciever on one side and a GENERIC transciever on the other side.
Tip :
This 10 Gbps Dual Port SPF+ NIC is designed for use in server chassis with high flow air pressure. To use this PCIe card in a normal case or a desktop/tower enclosure, you definitely need to add active ventilation. So I renewed the thermal paste on the main chip and installed a small 40 mm NOCTUA fan on the heatsink.
11. DIY HOME SERVER - Noise reduction
The case came with two 80 mm fans mounted on the back. To make the server as silent as possible, I exchanged both stock case fans with a NOCTUA NF-A8 PWM 80 mm 12V fan.
I also replaced the stock fan on the CPU cooler with a 92 mm 12V NOCTUA NF-A9x14 PWM fan.
At last I installed two small 40 mm 12V NOCTUA NF-A4x10 PWM on the passive heatsinks of the HBA controller and the 10 Gbps NIC.
All replacement fans have 4 pin PWM connectors and a 6 year warranty. All fans are recognized and automatically managed bij the mainboards IPMI.
During operation, the two case fans turn at 900 RPM, the CPU fan at 1000 RPM and the HBA controller fan and 10 Gbps NIC fan at between 1700 RPM and 2300 RPM, depending on the load. Fan Mode is set to Optimal in IPMI.
Tip :
At first startup all NOCTUA replacement fans were constantly switching between low RPM and full throttle. This is because those PWM fans run at much slower rotations then expected by the BIOS. The BIOS interpretes the low RPM as being below critical and puts the fans in full speed. Then the fans throttle down because there’s no need for so much ventilation. Then the cycle repeats itself endlessly.
Updating the BMC/IPMI Firmware from 3.62 to 3.91 solved the issue.
The BIOS now uses low threshold settings for the fans, as shown in IMPI → Server Health → Sensor Readings → Fan Sensors.
If your mainboard BIOS and BMC/IPM Firmware are already up to date and the fans are still ramping up and down, then try setting the threshold for the fans manualy as explained in this video.
12. DIY HOME SERVER - Power Consumption
Over a period of one year in normal day to day operation, having installed :
- 2 SSD CRUCIAL MX500 500 GB
- 1 HDD SEAGATE Ironwolf 8 TB
- 3 HDD WD Red 3 TB
and running :
- ProxMox VE
- TrueNAS VM
- Windows 10 VM for BlueIris 5 camera surveillance
- PiHole LXC
- Heimdall LXC
- NTP LXC
- UPS LXC
- Linux Mint VM
- Plex Server VM
the system uses an avarage 152 Watts and a maximum of 218 Watts, measured by a power meter on the AC input.
This includes, beside the server, the following additional hardware :
- 1 cable modem with build in (but disabled) WiFi AP
- 1 NetGear 16-port (8 PoE) Ethernet GigaByte switch
- 1 UPS APC Back-UPS BX1400U-FR
- 3 PoE IP camera’s
13. DIY HOME SERVER - Parts List
Category | Product | Brand, Model | EAN | Quantity | Price |
---|---|---|---|---|---|
Housing | Case | INTER-TECH IPC 4U-4408 | 4260133128620 | 1 | 203.90 € |
PSU | CORSAIR RM550X | 0843591034562 | 1 | 79.90 € | |
Case Fan Replacement Fan 80 mm | NOCTUA NF-A8 PWM | 4716123315476 | 2 | 33.90 € | |
Server | Motherboard | SUPERMICRO X10SRI-F | 0672042156092 | 1 | 199.00 € |
CPU | INTEL XEON E5-2680 v4 | 0675901383714 | 1 | 246.05 € | |
CPU Cooler | SUPERMICRO SNK-P0050AP4 | 0672042103492 | 1 | ||
CPU Cooler Fan Replacement Fan 92 mm | NOCTUA NF-A9x14 PWM | 163121136789 | 1 | 19.95 € | |
RAM ECC | KINGSTON KSM24RS4/16MEI | 0740617278255 | 4 | 449.70 € | |
HBA Controller | SAS Controller | DELL PERC H310 | 5397063766703 | 1 | 49.95 € |
SAS Cable | INTER-TECH CABLE SFF-8087 | 4260133125612 | 2 | 32.68 € | |
SAS Controller Additional Fan Fan 40 mm | NOCTUA NF-A4x10 PWM | 9010018100440 | 1 | 13.45 € | |
Storage | SSD | CRUCIAL MX500 500GB SSD | 0649528785053 | 2 | 134.75 € |
HHD | |||||
Graphics | GPU | DELL NVIDIA Quadro P620 | 5397184288863 | 1 | 99.00 € |
Network | NIC 10 Gbps | SUPERMICRO AOC-STGN-I2S | 1 | 48.39 € | |
DAC Cable | 2 m DAC Cable | 1 | 27.23 € | ||
Fan 40 mm | NOCTUA NF-A4x10 PWM | 9010018100440 | 1 | 21.85 € | |
UPS | UPS | APC Back-UPS BX1400U-FR | 0731304315926 | 1 | 175.00 € |
Total : | 1842.70 € |
14. DIY HOME SERVER - IPMI Configuration
a. First login
By default the systems IPMI interface card will get a DHCP address. Scan your network using NetScan to retrieve this address. If you want to give the IMPI NIC a fixed IP address, you can do so in the system BIOS.
Go to the IPMI URL. Accept the security warning and log in to the IMPI interface. The default username and password are both ADMIN.
The IPMI interface is a very handy feature. It allows you to remotely manage your system, even when the server is powerd off. Take your time to explore the interface and as I did, you’ll probably like it a lot.
Hi,
you are writing that the system consumes in idle only 53.2 watts and in daily operation around 100 watts.
Are these values correct?
If I try to roughly estimate:
4x HDD = 4x 7W = 28W (spinning)
2x SSD = 2x 3W = 6W
2x Fan = 2x 3W = 6W
1xGPU = 1x 10W (idle)
1x HBA = 1x 10W
We have in sum around 60 watts. To these 60W we have to add the mainboard (let’s say 10W) + CPU (10W in idle) + RAM (4 modules, 3W each) –> 60W + 10W + 10W + 4x3W = 92W
With this the 100 watts could be reached if the system is running mostly in idle. But do you really have an idle load of around 53 watts?
The values were correct. I use a power meter on the AC input to verify the power consumption.
Together with additional hardware (1 cable modem, 1 PoE switch, 1 UPS, 3 PoE IP cameras’), the whole system uses an avarage 152 Watts.
The 3 HDD dedicated to data are idle most of the time, except during data access.
The 1 HDD dedicated to the camera system is recording 24/7.
All devices were selected to minimize power consumption.