10T Backup Server Project

Play

Craig Severson explained his backup system one time. it seems over the top until you hear, “I can take that drive over there, plug in, reboot and I’m back to five minutes ago.”

I needed a backup device that transparent like Apple Time, was more cost effective then a appliance NAS drive, give me the most storage for my money and it had to be something I could self repair. I ended up building a 10 terabyte server powered by FreeNAS for under a $1000.

the problem that caused the need for this project presented itself in a rude way. a hard drive died inside a MacBook Pro which had a backup but it wasn’t up-to-date. this was caused by the movable nature of notebook computers. you have to remember to plug things in so that other things can happen automatically. the back up drive for Time Machine won’t work otherwise. and because nobody thinks like, “hey, so I’m going to lunch I guess I will plug this in so it will do it’s thing.” we have to make sure that backups just happen.

which brings up an ugly rule: the only time you think about backups is when you need one.

the “backup” that you make is really only as good as four things: how fast you can use it, how simple it is to do, how cheap it is to use it and finally how redundant redundant it is. generally when a drive dies there a few files that a person really wants now if they had access to them now they could keep working. it’s usually something like my bookthesispaper, pictures, music, project and email. and not always in that order. and when it’s borked is not the time to know if your backup software is doing it’s job. you have to check it once in a while and even better you have to test restore on some random hardware to make sure it’s really working.

the G4 NAS drive MK3 is still doing work in the office but is showing it’s age. it will be forever stuck at Mac OS X 10.4, has limited options for SATA cards, and if it dies it will stay dead because it’s 12 years old. even though it’s old I won’t retire it because as long as it’s running it will stay running. think mighty machine that serves files.

it would be easy enough to use a Time Capsule. but a quick bit of math showed that the office would outgrow it. it’s not really made for a work group / office environment. but there are other problems that made me dismiss it. you can read why I’m not just using a Time Capsule here.

a better product are mini servers called NAS drives. these offer redundant media options, cross platform file sharing, printer sharing, even music sharing without lots of complicated configuration to go through. commercial products like ReadyNAS and Drobo are interesting. the ReadyNAS is a stand alone box that connects to your network. while the Drobo can either attach to a computer or operate stand alone with an additional network adapter. you can buy either product with or without drives. for my application either would work and I was interested until I priced it out. either box with no drives cost about double what a basic motherboard, case, processor and RAM cost.

philosophy
as grown up as these products are it’s still to you to make sure what you are buying is compatible. in the past Macs and NAS drives didn’t get along. the file system didn’t support Mac names as they were tuned to work specifically with Windows. this would cause files saved on the server to become corrupt. they would look like they were there showing the proper file size and icon so all looked well until you actually tried to open the file. the buying tip here is if the NAS only supports SMB keep looking.

there are some things that I know about technology products 1) support really doesn’t help you unless its to issue an RMA number 2) drives fail 3) long term and proprietary do not go hand in hand. 4) while the parts inside aren’t any different then parts you can get yourself you don’t know if the company cheap’d out to cost reduce their product. if a power supply goes bad I should not have to wait a week for a new one. and if a drive goes bad it needs to be replaced now. 9-14 days doesn’t cut it. if I don’t have to haggle with support about getting these things fixed I’m even happier.

remember my goal is to get as much for my money as possible. so I’m willing to forgo company technical support and possible future repair from a single source. besides, building a box is fun.

the downside of rolling your own box is there are lots of unknowns. will this work with that and for how long. fortunately we don’t have to rely on “press” reviews of products. the users community that feedback on sites like NewEgg are beyond helpful. sorting by 1 stars to see potential problems as well as looking at 5 star reviews for praise. both should be evaluated with your specific requirements in mind. what doesn’t work for somebody’s application might be perfect for another.

software is another consideration that must be understood before buying a bunch of parts. I ruled out Windows right away because of cost. adding this OS would put the box I was building near the same cost as the other commercial solutions. there are lots of choices of open source software. Open Solaris, FreeBSD and Linux are all server capable. of course these OS may be beyond your skill set. the command line isn’t obvious to everyone. fortunately there are two projects that make it easy to get a file server running without having to using a CLI.

those projects are OpenFiler and FreeNAS. installing either is as simple as downloading an ISO, burning a CD, booting it on the target machine and following the instructions. which is as simple as it gets. both tools auto-detect network interfaces and hard drives. all you need to do is enter a fixed IP address and decide which format you want your storage drives to use.

we know that open source software can be a pain to work with because of sparse documentation and unfinished features. fortunately both projects have documentation that is good enough to help you get setup in case you can’t figure it out on your own. I installed both tools on my hardware before deciding on either.

the build
based on my research and testing I made a 10 terabyte server for under a thousand bucks. it’s been running for a while now so I have hindsight about the build, care and feeding and what I would change. here’s the specs:

SUPERMICRO MBD-X7SPE-HF-O Flex ATX Server Motherboard

2 gig of RAM up’d from 512M because ZFS complained

Western Digital Caviar Green 2TB 3.5″ SATA 3.0Gb/s Internal Hard Drive -Bare Drive

a 4G SATA SSD based on a Compact Flash Card

Antec Three Hundred Illusion Black Computer Case

power supply / whisper fans
powered by FreeNAS

building the 10T wasn’t any harder then making a PC from a box of parts. it’s simple. attach a motherboard, bolt in hard drives, and connect everything to a power supply. something I’ve done a hundred times. it was built and posting in less than 30 minutes. installing the OS on the compact flash drive didn’t work when I tried doing it on a Mac. however, it worked without a hitch when I did the install it from a proper PC. it took about 5 minutes

so there were problems right off. which I attribute to the nature of open source software. it’s not always obvious how to set it up! I did the install of the OS 4 times before settling on a version that worked. I partially blame this on the documentation. OSS doc is notoriously out of date. for example the use of “compact flash cards” was discouraged because they fail after a number of writes. except modern CF cards don’t suffer the same way cards used to.

performance issues and solutions
the next steps included formatting the drives, pooling them for ZFS, making users and then creating Time Machine mounts for all. between the false starts and learning how not to do it I spent about 2 hours with this setup. and now that I’ve done it a few times I could easily set up a new server in about 15 minutes.

the first big problem that came up after the initial configuration was that file copies were incredibly slow over the network. copy was at lest 5X slower compared to copying to the G4 NAS. poking around the UI for settings didn’t reveal an obvious solution but I found one in the freeNAS forum and it was in two parts.

the first problem was that the Western Digital Green drives that I choose can be slow on any system because they use a modern 4K block. you need to tell your OS that this drive uses that otherwise it won’t go fast. after the drives were reformatted and re-ZFSd performance increased. but it wasn’t go fast yet. checking the a box labeled “Enable tuning of some kernel variables” in the Advanced tab got copying to network saturation.

as you poke around you’ll find other tools and options that you can turn on. one configuration option is for email notifications. I made a new email address on my server. then I filled in the form with the normal log in information then clicked Send Test Email. it worked. 10 days in and then 4 months in.

10 days later I got an email that… oh no. one of the drives failed! it emailed me that a failure had happened. and I was like, oh YEAH! it worked!!! hell ya. I shut down the server, replaced the drive, then pointed ZFS to the new device allowing it to resilver the zpool. next I filled out the WD replacement form is on the web, packaged the drive and send it back. it’s replacement arrived back in days. I keep spare drives are in stock as a precaution. I bought 2 more because they were on sale. although I will very likely use these drives for other tasks because the shelf life of a idle unspun drive has proven to be short.

I set the server to email status updates once a week. at first I read them every time. now I don’t bother unless it’s not from a scheduled time. the only thing to note here is to be sure to enter the DNS settings otherwise the messages will never get sent.

all the problems I mentioned earlier got solve because I wasn’t in a hurry to get this running. I didn’t rush to get it done. because this software was a new thing for me I wasn’t promising when it would the project online. I had a bunch of false starts, reformats, and re-doing everything was okay. had this been a production file server instead of a back up server I might have been even more cautious about announcing it’s “on” to the people using it.

cutting costs
could the 10T be made for less money? certainly. I’ve got an idea of what would I change if I was building it with my new found experience. the case and power supply is the first stop in cutting costs. rather than using a new parts I could have used an old “found” case and supply. and if I found the case there might certainly be a motherboard loaded with a CPU and RAM inside. I could have chosen cheaper parts trading off looking good for basic functionality.

it might be nice to be able to swap a drive without opening the case. but “hot swap” drive holders start at $20 per drive and you have to have the right case to use them. I think if it’s job was production file server this would be worth the extra expense.

I might not use this specific motherboard. you don’t need lots of power for a file server. anything 1GHz and up will do. although 2G of RAM is a must if you plan to use ZFS. the SuperMicro board has 6 SATA ports and the processor is built-in which swayed my purchase decision. it’s certainly easy enough to find a cheaper solution using a hardware search tool. remember that you you don’t need a “RAID” card because freeNAS does this job in place of specific hardware and a dumb 6 port SATA card is cheap.

finally, the cost of drives has plummeted since I made the 10T. I’ve seen the specific drive for half of what I paid in July of 2010. the specific thing to watch with drive is the length of the warrantee. a bargain drive will often have just 1 year from the manufacture. this is not a drive you want to use for a 24/365 production server. and it’s certainly not good for storing backups. always mind the age of your drives with a plan to retire them as they as they approach the end of their warrantee.

wrap
Time Machine backups from all the users happen transparently. since it’s been in place I’ve used the going back in time feature a few times to get files that I stupidly changed without making a new version (open somefile_r4, save as, somefile_r5). having this much storage on the network means that nobody can make the excuse “I forgot to plug it in”.

the 10T has been in service for 9 months and has proven itself beyond my expectations in terms of cost, performance and uptime. I did expect drive failures because that is the nature of drives. that the server tells it’s status via email means means I’m not having to actually check it’s logs periodically for failures is a feature that I know works. hopefully you will have a similar experience if you decide to roll your own 10T (or bigger).

good luck. have fun.
YMMV.