Embedded Files
NOTE: This page from 2011 is severely outdated, but I'm leaving it here for the sake of posterity because it's still consistently visited
I've always had a bit of a fetish for Linux clusters, having heavily used them for my research since graduate school, and -- like any bona fide self-respecting geek -- fantasized for some time about having my own Linux cluster at home. I then found a number of people on the Web (in particular, Janne and Tim Molter) who have each built a Linux cluster into an Ikea "Helmer" file cabinet at a fairly low cost (around 2,500 USD). I was excited like a 12-year-old who had just discovered the lingerie section of his mom's department store catalog, and was naturally inspired to build my own 26-core Helmer cluster.
I wanted to build an SGE (Sun Grid Engine, now called Oracle Grid Engine)-based cluster, because that's what I used in graduate school and was most comfortable with (I've also worked with platform LSF, which is similar). SGE requires a "master host" that handles all job submissions & queueing, and a set of "execution hosts" that do the actual processing.
After reading the comments other people had left on Tim Molter's page, I decided to make all execution hosts diskless using DRBL (Diskless Remote Boot in Linux), with all files being stored in the master host and shared via NFS (Network File System). This means each execution host only requires a CPU and RAM; everything else it obtains via the LAN. How cool is that?
I should note that I previously had almost no admin experience with Linux. I'd never even installed a Linux OS onto a computer before, let alone set up fancy stuff like DRBL and SGE. It took quite a bit of trial and error, but I finally got everything working, and I am going to try to share the details of everything I did so that it may be of help to anyone interested.Since the hard drive on the master node is to be shared by more than 24 processes running at the same time, I also decided to use a RAID array -- also something I've never tried before -- for better performance.
Since Helmer can accommodate six micro-ATX motherboards, I decided to use Helmer to house all my execution hosts and use an extra micro-ATX tower to house my master host. I did think about using one of the Helmer drawers for my master host, but as everyone else points out, there is very little space in them. I wanted to be able to add additional hard drives to the master host in the future, and have easy access to the USB and monitor ports if I ever needed to connect a USB hard drive or directly manipulate the master host.
After some research, I drew the overall schematic shown on the right. I added an additional LAN card to the master node's mobo (motherboard) to connect it to my home network router, which is connected to the Internet and my Windows XP PC. This way, I could keep both the Helmer and the master node in my closet, and login to the master node via SSH using PuTTY and do everything from my Windows PC.
Here is a list of parts I purchased for this project in January, 2011, with all costs including shipping and/or tax. I was in a bit of a rush to build this system, so a lot of the parts were purchased at a store instead of online, and without shopping around for the best prices as a cheapskate like me would normally do. Of course, this cost me more, but I couldn't let a few extra bucks slow me down on my way to Helmer Heaven!
- IKEA Helmer file cabinet x 1 : $44
- Intel Core-i5 2400 CPU 3.1 GHz x 7 : $1148
- ASUS P8H67-M LE motherboard x 7 : $770
- TRENDnet TEG-S80G 8-Port Gigabit Ethernet Switch x 1 : $55
- Antec EA380D 380W ATX PSU x 6 : $300
- Antec NSK 3480 Micro-ATX Case : $95
- StarTech ATX Front Bezel Wire Kit x 3 : $17
- G.SKILL F3-10666CL9D-8GBRL 8GB(2x4GB) RAM x 3 : $269
- G.SKILL F3-10666CL9S-4GBRL 4GB RAM x 1 : $41
- Seagate 250 GB HDD x 1 : 44$
- Intel EXPI9301CTBLK PCI-E Network Adapter x 1 : $29
- Power Supply Cord x 5 : $30
- 5ft Cat6 Network Cable x 10 : $9
Grand Total: $2,851
I already had an old 250 GB hard drive, so I bought one more to create a 500 GB RAID 0 array. I decided that 4GB RAM for each execution host would be enough for my purposes, so I bought three dual-channel 2x4GB kits and installed one 4GB module on each mobo, and this worked perfectly. Later on, if I want to upgrade to 8GB, I can simply pair the three kits together, and buy three more kits for the other hosts.
The Helmer cabinet comes in three colors: red, silver, and white. All the other guys seemed to have used silver. I then remembered Doctor Brown's words from Back to the Future: "The way I see it, if you're gonna build a time machine into a car, why not do it with some style?"
The way I see it, if you're gonna build a Linux cluster into a file cabinet, why not do it with some style? Hence, I chose red!
As Tim points out, I had to cut off the plastic wrapping around the LAN cable connectors in order to bend them and fit them into the LAN ports of the execution hosts housed in the Helmer drawers. I want to put in a good word here for RiteAV.com for their great prices and fast shipping. I found their black LAN cables to be particularly good for this purpose, because their connector heads are short compared to the other cables. Cutting off the plastic wrapping turned out to be a pretty delicate job, because it's really easy to cut through the wiring inside. I ended up desecrating about 12 cables (including some that I had lying around the house) in order to get 6 neat ones. The other 6 were still usable, but I wanted perfection!
Since my execution hosts did not need hard drives, I was able to fit a mobo and power supply into each drawer with ease. Janne wasn't kidding when he said he wondered if IKEA designers had ATX specifications in mind when they designed Helmer; the mobo and power supply fit inside perfectly. I tried to roll and tie up the wiring from the PSU as neatly as possible, and also cut out pieces of cardboard from the box that the Helmer came in and laid them underneath the mobos to prevent electrical shorting.
I originally wanted to cut out the front and back panels of each drawer, just like Tim did, but 1. I didn't have the right tools, and 2. I wanted them done more professionally so that there wouldn't be any sharp edges, etc. I took the pieces to a metal workshop and showed them pictures of Tim's designs, asking if they could do the same for me nicely. They quoted me $500 for the whole job, so I decided to scrap it. The next time I visit Korea, I'm gonna take those panels with me to the Cheonggyecheon district in Seoul where I'm sure I'll find someone who will do it at a much cheaper price. For now, I'm just leaving the front and back sides of the drawers wide open, and as you can see from the pictures, it doesn't look too bad.
For the power LEDs and switches, I just purchased some ATX case front bezel wiring kits and connected them as you see in the photos, and that seems to work just fine. I only needed to buy three, because each kit contains two switches (reset and power) and two LEDs (HDD and power). I did also purchase some chassis fans, but will have to leave them out until I can get the back panels cut out. Cooling doesn't seem to be a problem right now anyway, since I have good ventilation. At some point, I may also cut out the unnecessary wiring in the power supplies like Tim did.
The only BIOS setting I had to change on each execution host was the PXE boot option, which had to be enabled. From that point on, the only work I needed to do was on the master host. I installed Debian Linux (Lenny) onto the master host, and installed DRBL and SGE following various resources on the Web. I allowed the master host to also be an execution host with two job slots, because I thought I could spare two of its four cores. The other execution hosts were allowed four slots each, so that one job would run on one core, resulting in a total 4x6+2=26 job slots.
I have documented the software installation and configuration (Debian Linux, DRBL, and SGE) here. As of February, 2011, the system is working beautifully, is blazing fast, and I really love it. I also want to thank Janne and Tim for being my inspiration and Guiding Light throughout this whole experience!
Any comments, suggestions, and questions are welcome!
-----------------------------------------------
This page last updated Feb 12, 2011
Page updated
Google Sites
Report abuse