First post, by DosFreak
- Rank
- l33t++
This thread is for helping others to build their own TrueNAS servers, thread may be cleaned up over time. Please leave divergent discussions to other threads.
First we'll start with:
SERVER SPECS
CASE
Ideally you'd use a case that has removable hard drives that can also show the status of those drives (power on, failure state) and also so you can label the drives with which port it is attached to so when there is a drive failure you can pull the right hard drive.
If all you need is M.2 NVME or 2.5" you can use internal bays or PCIe just make sure you label them!
You want a case that has good airflow but obviously if you have a ton of hard drives taking up space in the front of your case then that may not be possible, I've had 13x hard drives in the front of mine for years and although the temp of the HDs is high it is still within spec which is all that really matters.
Also look for a case that is easy to work on and that will last your for 10+ years. I've been using my SUPERMICRO CSE-743T-650B since 2007.
PSU
Size this appropriately , you don't want this too low but also not too high due to heat and energy bills. I always size mine too high since I think I may do more with my server (add a graphics card for passthru or transcoding) but never do.
MOTHERBOARD
You'll see people state that you need tons of memory for TrueNAS which is not true. 8GB or less can be fine depending on your uses. Something to keep in mind though is that just like SQL TrueNAS will actually use your memory so the more memory you have then the more in can store in memory. For a home server this usually isn't that big of a deal. For TrueNAS you should be buying a server motherboard anyway which should have the potential for alot of memory so you can upgrade later.
Find a board that has a lot of SATA ports, PCIe cards would be more heat, energy, affect air flow, potential failures, speed or compatibility issues.
If you want to run TrueNAS from a DOM then make sure there is a power connector for a SATA DOM or a USB2 or USB3 header on the board. You can also waste an onboard M.2 if you want but TrueNAS runs fine from a DOM. If you are using a USB or DOM I'd recommend storing the system dataset elsewhere.
It's likely that you'll use up all your SATA ports so you may need or want to use your PCI express for storage. I use "ASUS Hyper M.2 X16 PCIe" which store 4x m.2 nvme drives but requires that your motherboard supports bifurcation. You can find pci cards that don't require bifurcation, looks for the ones that have 2x slots and dedicate 4x to each m.2 and as a last resort use the ones with bridge chips.
Make sure the motherboard is either headless with an IPMI interface or has onboard video. No sense in wasting heat or energy on a video card.
Use a motherboard with 10GBE, you may not need it now but you don't want to have to spend money later and eat up a PCIe slot, if you find a motherboard you like that doesn't have it then make sure that you'll have enough PCIe slots to use it.
PROCESSOR
You don't want a processor that's too low or too high but you do need to take into account what you want to do with the server.
If it's pure file storage with no VMs or transcoding or dedupe then go lower.
If you plan on using any of the above then go higher.
If you are building a new server with a server class motherboard you'd have to actually go out of your way to find something that's not sufficient.
If you are repurposing an old motherboard with an old processor then it'll likely be only good enough for file storage.
OS
You can install TrueNAS to whatever you want as long as it's bootable. Something to keep in mind is that TrueNAS has a "System Dataset", if you are booting TrueNAS from USB then change the system dataset to some other location. If you are using a NAS and don't want your drives to spin up alot then also change this to an SSD if you have the option.
MEMORY
ECC memory should be used with TrueNAS but is not a requirement.
Memory is cheap and your files are important so use ECC memory.
Again TrueNAS doesn't need alot of memory for home usage, if you are using VMs or dedupe then add more.
If you are unsure then start with 8GB (or whatever makes sense since you don't want to throw away that stick when you add more memory) and only upgrade if you need to, not because you want to.
STORAGE
For my data I use raidz3 which uses 3x drives for parity out of 13x Seagate 16TB drives. It's recommended to not have so many drives in one pool for performance reasons but I can only put so many drives in this case and I want to keep everything in one case so 13x drives in one pool works for me.
For this pool for dedupe I have a Asus Hyper M.2 x16 PCI 4.0 card with 4x Seagate FireCuda 530 1TB that is used for my Special VDEV to store my DDT for dedupe.
For my Windows VMs that I access via SMB I use a ASUS Hyper M.2 x16 PCIe 3.0 x4 with 2x Sabrent Rocket Q 8TB M.2 (I store the original VMs on my raidz3)
For my TrueNAS bhyve VMs I store them on my 1x onboard M.2 Samsung SSD 970 EVO 2TB
BOOT DRIVE
Should be at least 8GB
It's preferred that this not be to USB but I haven't had any issues as long as I set the "System Dataset" to SSD/Platter HD.
DATA DRIVES
If you want the most affordable then this can't be SSD, use SSD for applications that need it such as Databases, Virtual Machines, etc.
If this is just for home use you may not even need a SSD depending on your usage.
Options are Mirroring,raidz1,raidz2,raidz3 (# = number of disk failures)
Only use mirroring if two drives or for performance reasons.
Only use raidz1 if you are limited in the number of drives you can use.
So plan on using either raidz2 or raidz3. If you have a lot of data and a lot of drives go with raidz3.
If you have too many drives you should create multiple pools. I've been fine with raidz3 with 13 drives for years but that is pushing it a bit, you usually see less than 12 recommended in a pool.
For home use the main factor will be cost so likely most home users will have a single pool instead of multiple pools, so plan for that.
DATA
You'll need to figure out how much data you'll need to store and to take into account how much it will grow over time.
The more data you have then you'll need larger drives and more of them.
VMs
If you plan to run VMs using bhyve on TrueNAS then you should use a SSD so account for that.
If the VM is simple to setup and backed up then you don't need to mirror them just use a single drive.
Bhyve is not as feature complete as other VM products and kind of annoying but it may be good enough for your uses. I use it for a Windows and Linux VM where I have the online game clients installed.
VMs DATA
You can setup a zvol (a volume) for your VM as a different drive and this can be stored wherever. Likely your SSD space is minimal so you may need to store this on the same drive as your other data. For my uses which is storing online games this works fine.
CASE: SUPERMICRO CSE-743T-650B Black 4U Case 11 Bays (Replaced case fans with Noctua Fans)
CASE ADDON: Supermicro CSE-M35TQB 5-in-3 Hot Swap SAS/SATA Mobile Rack (Black)
PSU: Supermicro 1200w ATX12V Power Supply - 1 kW, 1.20 kW - Internal - 110 V AC, 220 V AC (PWS-1K25P-PQ)
MOTHERBOARD: SUPERMICRO MBDH11SSL-I Rev 1.01
PROCESSOR: AMD EPYC 7351P 16-Core
PROCESSOR COOLER: Noctua NH-U9 TR4-SP3, Premium CPU cooler for AMD TR4/SP3
MEMORY: 4x Supermicro Certified MEM-DR432L-SL03-ER26 32GB DDR4-2666 LP ECC REG DIMM
ADDON CARD: ASUS Hyper M.2 x16 PCIe 3.0 x4 Expansion Card V2 supports 4 NVMe M.2 (2242/2260/2280/22110) up to 128 Gbps for Intel ...
ADDON CARD: ASUS Hyper M.2 X16 PCIe 4.0 X4 Expansion Card Supports 4 NVMe M.2 (2242/2260/2280/22110) up to 256Gbps for AMD 3rd Ryzen sTRX40, AM4 Socket
NIC: Intel Ethernet Converged Network Adapter X540-T1 10GBE
HARD DRIVE: 13x Seagate IronWolf Pro 16TB NAS (ST16000NE000)
DOM: 1x Supermicro 64GB 520 MB/s Solid State Drive (SSD-DM064-SMCMVN1)
ADDON CARD: 2x Sabrent Rocket Q 8TB M.2 for ASUS Hyper M.2 x16
ADDON CARD: 4x Seagate FireCuda 530 1TB Solid State Drive - M.2 PCIe Gen4 ×4 NVMe 1.4
ONBOARD M.2: 1x Samsung SSD 970 EVO 2TB - NVMe PCIe M.2 2280 SSD (MZ-V7E2T0BW) for onboard M.2