ANODE new RAID file server--notes

Several months of notes leading up to ANODE: This computer is awaiting drop in HDD prices around 1Q2013 when Thailand factories recover from the big flood

Recycled Case available: Thermaltake VH6000 with room for 5 3.5" drives & 7 5.25" devices

This computer might be built to allow use of Flightgear flight simulator. Might use $48 video card ASUS GT610-SL-1GD3-L GeForce GT 610 1GB 64-bit DDR3 PCI Express 2.0 x16 HDCP Will be 16GB RAM to support VM quad-core or six-core uP A builder's recommendations such as pwr supply to suit your video card http://gallinazo.flightgear.org/technology/build-your-own-hot-flightgear-pc/ and he is partial to Linux & AMD just like me

April 6 2013 the HDD price drop happened last 6 months, I have six Western Digital RE4 RAID Edition 250GB drives at $90 each, these are labeled "enterprise storage." These were received Ap 10, tested in WinXP with WD Data Lifeguard and confirmed good. SMART Data, seen in Ubuntu without even mounting these drives, showed no reallocated sectors.

June 12 2013 Bought 4x4GB SRAM PNY DDR3 1333MHz $30 each & three 500GB Seagate Barracuda SATA 6Gb/s 7200RPM ST500DM002 2yr warranty $55 each at Best Buy. Two Seagates are intended for Highpoint RAID card, with another Seagate held back for replacement in case of real or simulated failure. These were made in China Mar 2013, 50000 contact start/stop cycles, 16MB cache, 6W typ & .8W standby/sleep. Seagate says they are good for home server & desktop RAID.

The other thing holding up ANODE computer is Linux getting on board Microsoft-mandated Secure Boot, part of UEFI. Feb. 8 2013: http://www.zdnet.com/linux-foundation-releases-windows-secure-boot-fix-7000011084/ so maybe this is going to work out, but new problem is that HighPoint RAID controllers mention Linux & Win7 support but don't mention Win8, and I find that Win8 has RAID alternative, Storage Space. Don't know how this is going to play out in ANODE build. Found Adaptec RAID table http://ask.adaptec.com/app/answers/detail/a_id/17243/~/what-adaptec-products-are-supported-in-windows-8-and-server-2012%3F where Microsoft put some Win8 drivers for Adaptec RAID on the Win8 install CD. So it may be best to go with the better-known RAID controller brands.

http://social.technet.microsoft.com/Forums/en-US/w8itprohardware/thread/8b3ad63f-943b-41d3-a451-806c1244d423:

With the Storage Space in Win 8 which seems to be some sort of "software" RAID that utilizes software drivers, is there a place for a hardware RAID which utilizes disk controller?

Well, sort of... if there are insufficient drive connections available on the motherboard, a disk controller card with it's RAID features disabled, would provide you with a way to connect enough disks for an array [in Storage Space]. The card must be able to have any of it's functions disabled to be compatible with Storage Spaces. So, only a disk controller is necessary.

In order words, am I making a wrong assumption that spending money on RAID controller cards may become unnecessary in Windows 8?

If you prefer managing your storage with the hardware card and accompanying software features, that would be a reason to still use it.

Storage Space can be managed via control panel, shell commands, System Center 2012 and Server 2012, providing full platform control. Not so, with a hardware card. [Microsoft is discounting the (often) extensive controls, monitoring, and e-mailed notifications available from hardware-RAID software.]

Another contributor: Storage Spaces is glacially slow compared to a real hardware RAID card. The management features of Storage Spaces are very limited in comparison to hardware RAID. With a hardware controller, I can see all the stats of all my drives, perform background verification, and the storage architecture is clearly defined and under my control. Likewise, the array is completely independent of the OS. I quit using Storage Spaces because the internals are too obscure, and I was very disappointed with the speed, as well as bewildered by the bizarre variability in transfer speed when copying files. Even for simple mirroring, my hardware controller is far faster and consistent in behavior. Storage Spaces has potential, but until it resembles something like a Windows version of ZFS I doubt it will challenge dedicated controllers. Friday, April 05, 2013 JE on Apr 7, 2013 Note how this critical user posted his message just two days ago. It may be that Storage Spaces does background maintenance once in a while that temporarily impairs file access, and the user mistakes this as being unexplainable.

(Continuing ANODE) Looks like I am an early adopter of hardware RAID in Win8. ]Well, that may be because so many are going away from hardware RAID. See below.] As I look for name brands & Win8 support, I can't see prices at the same time to let me make a choice toward the middle capabilities, and price when there is Win8 support tends to be $500. So the plan has to be: motherboard that can secure boot between Ubuntu & Win8, new RE4 drives split between Win8 Storage Space & Ubuntu HighPoint 620 $35 RAID, & the arrays won't be visible to the other OS, then over months see if $200 RAID controller drivers become available.

Web sites say that Win8 needs to be installed before Linux. Do I have to buy Win8 Pro early on? No, my intention is to reinstall Ubuntu often, in part to find how RAID setups work well or maybe don't work, and Ubuntu-only is OK at the beginning of the ANODE computer.

Plan for building this computer is shifting from Atlanta Frys to Augusta Computer Exchange; if a motherboard can't be made to work in May or June, I can buy another and still save money compared to having to travel to Atlanta and be overnight. May 23, 2013 RFQ went to Computer Exchange, they can do AMD Pile Driver with enough PCI/PCI-e, like MSI 970A-G46 $80 which has Winki3 built-in, a stripped-down Linux that is good for browsing Internet with no OS loaded! (990 chip set with more PCI-e may not be needed, especially at double the price.) New news: Win8.2 ("Blue") due out this Fall. What I read recently on Internet makes me think it is the right time to get a motherboard. If there is video trouble from NVIDIA, for example, popular cards can be had for only $45 and I could swap out for Radeon without even returning NVIDIA. Win8 must be Pro 64 bit System Builder OEM and choose the activation option for personal use. Don't know if Ubuntu 12.04.2 LTS or 13.04 is best, or even if either takes advantage of Feb2013 secure boot helper files from Microsoft. I have live-boot FLASH USB memories loaded with both to try out. SDRAM is more than I thought, >$130 for 16GB. Enermax (?) web site PSU advisor says 419W is OK for 7 drives & my intended HW lineup, so 600W supply $130 is probably fine. GPT partitioning is a given, managed through mdadm or a GUI version of that. ZFS has good press but >=1 user says it is too early for ZFS on Linux, so maybe do ext4 file system. But reading about ZFS is making me turn away from hardware RAID and think about software RAID! LVM on top of RAID may eventually happen.

June 17, 2013 With no response to RFQ from Computer Exchange, we left for family wedding in TX and went by Fry's near Atlanta and made the purchase. The 970A-G45 was there but both boxes had been opened, only $15 reduction. John at Fry's counseled for MSI 990XA-GD55 with rebate to get it nearer the 970 price. This has 6 PCI-e but only one PCI slot, but that will be OK. Antec HCG750 supply (high current gamer) with four 12V rails. FX 6300 processor, 6 cores, 14MB total cache, "Black Edition." I hope this is one of the 95W uP, not the 125W. I paid for them to assemble these plus the PNY 16GB SDRAM I brought from Best Buy and it got into BIOS OK. When I took time to check on Internet, I find that the Southbridge chip SB950 has given USB trouble on multiple mobo suppliers under Linux. Will need to find if this has been patched in the half year since the posts I was looking at. Need to get more of the $4 USB memories from Cyberguys to back up multiple BIOS versions, eventually. Also on Internet, this mobo has no full-GUI EFI which I was really looking forward to, but on the other hand this mobo is described as being low cost, so you get what you pay for. Mobo sys fans 3 & 4 are 3-pin but don't even sense speed, so don't use them. Sys fans 1 & 2 have PWM on 4th pin, good. JFP2 has both speaker & buzzer pins, don't know how much power available to speaker & what type of info would come out the speaker, so don't know if putting a speaker on those pins is worth anything. I am designing a fan-thermistor-3rd pin board to enhance cooling of ANODE especially when Linux doesn't provide PWM signals on the sys fans 1 & 2. This is to go along with FP708 bay device through NewEgg, but I want added thermistors on middle HDD in the HDD cage and on the Southbridge sink. There is a TPM, trusted platform module, header on mobo and MSI site sells it for $13, am inclined to get this security device and see what it does. Look ahead to Dec 7, 2013 on this web page to see that Secure Boot started working with Win8.1.

June 29, 2013 Took the mobo tray that Fry's built up for BIOS test with SDRAM & supply, slid it into case, mounted fan, built a stand for the three Seagate 500GB SATA purchased from Best Buy & put an 80mm fan under that until I get a 140mm fan, hooked up one of the Seagate HDD purchased June 12 to see if Linux can install. No big problems, despite reading about Southgate dropping USB & scare stories about video. Am not sure how to get to BIOS, I try various keys & sometimes can get into BIOS. EFI BIOS responds to mouse. In BIOS, CPU is not saving power & sink does get hot. Also, Northbridge, which has a big sink, is even hotter & mem is warm. Added two fans, on Northbridge & mem, they are much cooler, now. Will need to come back and neaten up. Tried Ubuntu 13.04 install from USB 4GB, that worked and it got 66 files thru Internet to update during install. System Monitor shows all six CPU cores! Running Linux, the blue LEDs on mobo show lots of power savings (I enabled pwr savings in BIOS). All 16GB of SDRAM show up in BIOS. The $35 video card has enough capability to let Unity desktop transparency show up when certain overlaps of windows happen, PNY GT210 1GB NVidia.

July 5, 2013 Orders to Cyberguys and NewEgg are in for fans and a fan/temp bay device. The cooling plan, using seven case fans, including fans on Northbridge & SDRAM, is very complicated because I want to do feed-forward control of exhaust fan when CPU power-phase LEDs 6-8 come on, use the FP708SL temp alarm to make some fans come up to full speed, and other refinements. It is also complicated because of the mix of 2-, 3-, and 4-pin fans. It turns out that there is a lot of analog in the custom fan PCB.

ANODE

July 16 Received TPM, trusted platform module $13 from MSI Store, installed on mobo pins but the software is probably Win only, so don't install software until I get Win8.1. Looking at the manual, there are 4 passwords that make TPM work, and it looks like you encrypt one partition. Found a serial (9 pin) connector & mobo connector at Computer Exchange, that is installed. So just about every connection on the mobo is now used. FP708SL bay device: using an alarm threshold of 106 deg F for the Northbridge sink even with 60mm fan blowing on it. Reading about Secure Boot on Internet, I haven't seen Secure Boot on/off control in the new mobo BIOS but read a tip to look for Legacy BIOS and turn it off, that may be an alternative name for Secure Boot. Also, M-Flash of BIOS is not recommended by MSI unless using Windows 8, there may be a BIOS or EFI upgrade that will let Secure Boot work. Ubuntu 13.04 is definitely set up on Seagate HDD with MBR partition table, not GPT, so there are at least two things that are not ready for Win8. Maybe I need to download a Microsoft readiness app for Win8 that might give some answers about these things. Looking for GPT for Ubuntu, I find that the older fdisk etc. programs don't do GPT, web sites by Rod Smith & his GPT fdisk look promising. There is no hand-holding built into the MSI 990XA-GD55 for novice Win8 installers, and to get GPT with Linux you really have to look around for help. My Official Ubuntu Server Book has a little about LVM on top of RAID but it does look like LVM is for administrators of big systems. And LVM is used for Windows and Linux, you can do just some of your partitions for LVM, doesn't have to be all your partitions. 50-LED, automotive-style, 12V white LED one-meter strip installed into ANODE case, the lighting is non-directional and so nice. Got GIMP and InkScape installed, they work fine. Ch. 18 in Negus Fedora 5 and RHEL 4 Bible 2006 says NFS is the network file system used to serve Linux workstations, but SAMBA tends to be used on a Linux file server when a netwok has Windows workstations.

The custom-etched fan controller for ANODE, to fit into a 5.25" bay, block diagram showing interactions--

This PCB project got pretty complex when I decided to develop my own pulse-width-modulated signal, with input from pot or thermistor temp sensor, and the other complicating factor is the ability to listen to almost all fan speeds plus an indication of the PWM signals coming from the two case-fan jacks on the mobo, neither of which I am actually using since I want the option of manual control. The board is so complex that it will not be easy to get all it's functions working. But it is a neat project since it has an audible function and a fan-speed function.

Windows 8.1 books--at Barnes & Noble, and Amazon, Win8.1 books are not due out until Dec 2013. Normally, such books come out before the software can be purchased. But I noticed in a Win8.0 book that you have to enable TPM in BIOS, and I was able to do that.

Advice from Brad is that ZFS is mostly freeBSD, LVM may be in the Linux kernal, LVM function may be in Win8 but not by the LVM name, and in LVM the logical volumes act as partitions, you don't pre-make partitions in the part of the volume group that you are doing LVM on.

A web site says LVM is mostly a Linux thing.

According to http://www.freebsd.org/cgi/man.cgi?query=gvinum&sektion=8, LVM is also freeBSD.

Wikipedia says ZFS is a combined file system and logical volume manager designed by Sun. A site says there is no OS-level support for ZFS in Windows.

But http://en.wikipedia.org/wiki/Logical_volume_management says there is widespread support for LVM (see big matrix), and in Windows it is called Logical Disk Manager but in Win8 is replaced by Storage Spaces.

July 27, 2013 Update on custom fan controller--the PCB etched well, the Atmel CPLD fit OK & programmed OK, the ripple-counter countdown from 50kHz works and the logic for both the audio MUX and state machine seem correct. All the circuits except PWM generator are built up & working. The area around the CPLD, 44-pin TQFP, is tight but within my soldering ability. Cabling all the fans and sensors to this board may be hard, I hope to plug on the cables with the board pulled out the front of the bay.

Aug 5 Yes, plugging cables can only be done with the board pulled out the front, had to make some cable extenders. All channels of fan-speed MUX are working though it is hard to pick out frequencies when they are square waves. Seeing the board's temp controller work awaits cold weather. Manual PWM with the pot, for the two, 140mm, PWM fans is fine. The 5-bay HDD temp sensor is working great, sensing one of two WD RE4 drives, with 82 deg F room temp; with setpoint of 99 deg F, the drive temp goes over 99 deg in ten minutes and the front fan goes to full speed to cool it down. This happens over 15 seconds and, as a bang-bang controller, repeats though with increasing duty cycle, as expected. Labelling is in place to show the speed MUX order and mapping of the FP708SL channels to actual connections.

The row of LEDs is very helpful to understand what is going on. The objective of controlling fan speeds independent of OS is accomplished. The 9-pin Sub-D jack has two ground, two +12V, one +5V to make custom pwr adapter for external SATA.

Above is recording from the front-panel sub-D jack, pin 9, the fan-tach MUX. All 12 channels are shown. The waveform is displaced up or down depending on the duty cycle of the channel. (DVD-fan tach is homemade and very narrow, negative pulses.) The two Case Fans use the PCB's PWM-to-freq converter to give an audible indication of the PWM commands coming out of the mobo's Case Fan Jack 1 & 2. I am not using these jacks to power up any fan since there is no driver to control these jacks in Linux. The seven fans sensed by the custom PCB go from CPU at center to window (side) fan at right. The CPU fan is probably controlled by BIOS and the board merely senses the tach pin, #3 of 4 pins. Northbridge/SDRAM fans and DVD fan are speed-controlled by FP708SL. The other four fans are controlled by the custom PCB.

The image above is from GETTER computer, the 2003 video-editing computer with the Intel mobo and striped RAID. Adobe Audition recorded the waveform coming out of pin 9 of the sub-D jack on the fan-controller PCB.

The following image zooms in on the transition from displaying the rear fan to the window fan.

What followed was a long period of finding if md0 was made, how to see it, and realizing it was mounted in /media. Then I found I needed to partition the md0 array, so I made a 10GB partition in the middle to play with.

The 63GB block device is RAID 1, 63GB in each of three disks. The orange highlight is of the 10GB initial partition I first made in the array. But notice that the name is not md0, it is md127p1. I have to read the manual about that naming. But the mount point is not in /dev, it is in /media/scinstr.

The md127 name actually shows up above if you look for it.

I needed to do a sudo chmod a+w md127p1, I think it was, or maybe sudo chmod a+w 10G_in_RAID1, to give permission for user scinstr to do mouse clicks to copy files to the RAID array.

After copying some .mpg files to the array, the above is how it looks.

This is good success for making a RAID array, on the way to putting LVM on top of it.

There was a lot of poking around in the file system and in Disk utility and System monitor to get to this point.

I see that many software arrays can be set up at once with mdadm when you first establish partitions on your physical disks. There can be some RAID 0, RAID 1, and probably RAID 5 & 10, but there is surely no need to have such a variety. I want to set up Ubuntu on sda and sdb, and I also need to use some of the mdadm commands. I am foggy about whether Ubuntu needs to be set up on sda and sdb before or after making LVM on top of RAID. Probably after.

August 22 to 27, 2013 Progress to adding a striped array, RAID 0, on four of the five RE4 drives

Reviewing the first use of mdadm to make a mirror array, two screens above, 3-way mirror on sdc,d,e. (It warned of metadata at start and this was due to my earlier choice of MS-DOS (MBR) partition table rather than GPT.)

scinstr@ANODE1:/dev$ sudo mdadm --create md0 --level=1 --raid-devices=3 sd[cde]1

Record some details of what happened. Little of this detail shows up in the man pages for mdadm, there is a lot more partitioning going on than I would have expected.

Above shows how I got GParted to make extended partitions, blue outline, done in sdb,c,d,e & 58.59GB ext4 within the extended partitions in preparation for striped array.

Disks GUI utility still shows three Other Devices so far, as shown four images above.

Now for the big step ahead, the second use of mdadm, try for 4-way stripe:

(Use md1 because I had used md0 in the first use of mdadm. Use the metadata option because of the warning in the first use of mdadm, though with GPT in current RAID volume it could maybe default to 1.2.)

scinstr@ANODE1:~$ sudo mdadm --create /dev/md1 --level=stripe --metadata=0.90 --raid-devices=4 /dev/sd[bcde]5

[sudo] password for scinstr:

mdadm: /dev/sdb5 appears to contain an ext2fs file system

size=61440000K mtime=Wed Dec 31 19:00:00 1969

mdadm: /dev/sdc5 appears to contain an ext2fs file system

size=61440000K mtime=Wed Dec 31 19:00:00 1969

mdadm: /dev/sdd5 appears to contain an ext2fs file system

size=61440000K mtime=Wed Dec 31 19:00:00 1969

mdadm: /dev/sde5 appears to contain an ext2fs file system

size=61440000K mtime=Wed Dec 31 19:00:00 1969

Continue creating array? y

mdadm: array /dev/md1 started.

scinstr@ANODE1:~$

The creation of the 4-way striped array was successful. Next, use GParted to make five partitions in the new Other Device. One of them is a second Linux swap in case I need it, like if I try to install Ubuntu 13.04 onto the stripe.

Mdstat_after_making_RAIDstripe_with_mdadm_Aug242013.txt

md1 : active raid0 sde5[3] sdd5[2] sdc5[1] sdb5[0] 245757952 blocks 512k chunks

md127 : active raid1 sde1[2] sdd1[1] sdc1[0] 61407104 blocks super 1.2 [3/3] [UUU]

*

Mdstat_Aug27_stumbled_upon_little_mount_arrow_in_Disks.txt

md126 : active raid1 sde1[2] sdc1[0] sdd1[1] 61407104 blocks super 1.2 [3/3] [UUU]

md127 : active raid0 sde5[3] sdb5[0] sdc5[1] sdd5[2] 245757952 blocks 512k chunks

note: the md names of raid0 & 1 swap depending on whether RAID0 is mounted. Who knows why.

At the * above and in the two images above, I stumbled upon a little arrow button within Disks utility. It does an on-line mount, partition by partition.

Stripe & mirror RAID arrays, shown in /dev and /media, respectively:

scinstr@ANODE1:/dev/disk/by-label$ ls

10G_in_RAID1 availAug22 Pri_sdb stripe_md1_sw [swap] stripe_md1_t stripe_md1_u stripe_md1_v stripe_md1_w

scinstr@ANODE1:/media/scinstr$ ls

10G_in_RAID1 availAug22 stripe_md1_t stripe_md1_u stripe_md1_v stripe_md1_w

scinstr@ANODE1:/dev$ ls (just the interesting part)

block a directory disk a directory mapper a directory md126 md126p1 md126p2

md127 md127p1 md127p2 md127p3 md127p4 md127p5 md a directory sda sda1 sda2

sda5 sdb sdb1 sdb2 sdb5 sdc sdc1 sdc2 sdc5 sdd sdd1 sdd2 sdd5 sde

sde1 sde2 sde5 ubuntu-vg a directory, the volume group of LVM I did first for Ub 13.04 install

Looking at the Disks window above, we see two RAID Members in sdd. The left one is the mirrored array on sdc, sdd, & sde and the right, orange one is the striped array on sdb, sdc, sdd, & sde. But for both RAID arrays, GParted has been used to allocate just a portion of the arrays to active partitions. More arrays could be made by setting up more partitions in the existing mirror and stripe and using mdadm on them. This fact does not leap out at you from the GUI tools, you have to know where to look. I am still sort of fuzzy about it, myself. The place to check it out is the image below, looking at the 63GB and 252GB block devices which are md126 and md127.

Now look at Disks utility, again. Click on the gears button and you get access to the Mount Options window.

Above is where I noticed the gears icon. If you turn off auto mount, and ask for mount at startup, it shows a star for the partition and the fstab file is added to as follows. (If no star, you have to mount it after each power-on). Here is fstab, file system table.

# <file system> <mount point> <type> <options> <dump> <pass>

/dev/mapper/ubuntu--vg-root / ext4 errors=remount-ro 0 1

# /boot was on /dev/sda1 during installation

UUID=925a5fe8-6db1-492c-839c-a97a61402f8d

/boot ext2 defaults 0 2

/dev/mapper/ubuntu--vg-swap_1 none swap sw 0 0

/dev/disk/by-uuid/241fcfb3-7a7a-4117-8b5a-8a892f1b0bc8

/mnt/241fcfb3-7a7a-4117-8b5a-8a892f1b0bc8

auto nosuid,nodev,nofail 0 0

/dev/disk/by-uuid/8dd6bdd5-0954-49fd-b7cc-f17b0b64d620

/mnt/8dd6bdd5-0954-49fd-b7cc-f17b0b64d620 auto nosuid,nodev,nofail 0 0

/dev/disk/by-uuid/10dc3ba1-5023-4723-b2f4-a5d2d84ef97a

/mnt/10dc3ba1-5023-4723-b2f4-a5d2d84ef97a auto nosuid,nodev,nofail 0 0

/dev/disk/by-uuid/66806fd4-2551-4d9e-acf0-b6c8e9544be2

/mnt/66806fd4-2551-4d9e-acf0-b6c8e9544be2 auto nosuid,nodev,nofail 0 0

These operations have shown that the use of mdadm is just the start of making a RAID array useful.

If you choose to not show the RAID partitions in the Unity left-hand panel, how can you make a symbolic link on the desktop so you can use the partitions conveniently? Use ln -s /mnt/UUID /home/scinstr/Desktop/man-readable-name

Think about whether Ubuntu 13.04 could be installed on the stripe array, to make startup faster. Maybe GRUB could handle that. But if I ever remove the LVM on RE4 sda, where booting has been happening from for a while, would that remove what is needed for the (potential) boot from stripe? In other words, is ANODE boot dependent on sda, even though the 4-way striped RAID array skips sda, using sdb, sdc, sdd, and sde?

Sept 18 2013: current thinking is that dual booting Win8.1 & Ubuntu 64 bit with secure boot on & GPT partition table is not a prime goal after all. Emphasis should be more to have either Win8.1 or Ubuntu loaded on storage at any one time, and use secure boot and GPT. With Win8.1, attempt to use TPM. The goal is to experience both OSs with RAID, not necessarily dual boot.

Nov 15: Topic ANODE. Received Win8.1 64 bit Pro OEM (system builder) DVD $130 from NewEgg, also HighPoint 640L 4-port SATA JBOD card (4-lane PCIe) $70 to substitute for 620 2-port so I can get all three Seagate 500GB $55 HDDs available for Microsoft Storage Spaces, and leave five WD RE4 drives for mdadm software RAID with Ubuntu. For new file server ANODE, the bootable, FAT32, USB drive with files in /boot/efi/ to update BIOS from 22.5, an M version Oct 2012, to 23.2, an N version of March 2013, was not recognized at boot time by ANODE. (USB drive was prepared by unzips of files from MSI, and MSI warns that the BIOS updater program or script should be the one that comes in the unzipping, not the old one on the BIOS desktop.) I saw that F11 gets the EFI shell (command line) that is in BIOS 22.5. Help -b shows available commands (about 50 of them), and it is either Linux or close to it. One of the commands, map, shows drives, including the USB drive, and there is a mount command that probably is necessary to list (ls) the files on the drive, just to check, and maybe even to run. I fiddled with mount a little bit but left the system in some state that no longer recognizes a Linux live install from USB (one that used to work). I can still get to BIOS 22.5, but the only other thing showing up is a command to insert a boot device. I tried putting the five /boot/efi/ files in the root--no advantage. Maybe there is a mount reset, and maybe there is a BIOS reset, and maybe I need to change over the mobo jumper and do rear-panel reset button. This may even benefit from a workover by Augusta LUG. I did try a 12-yr-old Win XP Pro install from CD, after adding back in one HDD to mobo first SATA connector, which starts fine but stops in a blue-screen error in two minutes. Since I haven't updated BIOS, I don't think I have done anything irreversible. Another thing to try is the Win 8.1 DVD, just to see what happens with the sub-Win-8 version of BIOS (with EFI enabled).

Nov 16 2013: in EFI shell (do F11 during boot), found map -r to reset the mapping. Whether that was needed or not, Win8.1 DVD starts but terminated at partition time using the USB drive with MS-recommended unattend file. I think map -r was needed to let Ubuntu 13.04 start up from USB drive, whereby I got the first RE4 drive cleared out of all partitions (251GB free space, contents unallocated space MBR) to where, maybe, Win8.1 won't terminate at partition time. Still with unattend USB trying to automate install, it quit again at partition time, so I tried it without the unattend USB and it went ahead with full install, though I don't think some things will work, like BitLocker because BIOS is still not updated. But Win8.1 is functioning on ANODE, a big step ahead, and the hardware seems to be compatible with Win8.1 Next step is to put in Highpoint 640L and see if the three Seagate drives can be used for data from Win8.1.

In BIOS, found that BBS is BIOS boot spec, and BEV is boot entry vector.

It seemed to want an additional MS account, e-mail address with pswrd Fro... (at bottom of our block), and MS sends back a 7-digit, numeric security code they may need for future identity verification.

MS sent numerous update files that installed automatically.

Nov 17: performed the BIOS update. I re-did the bootable USB drive, with new understanding from MSI downloads screen, after search for 990XA-GD55, that when they say update BIOS only when you use Win8 they may mean to do the updater download & unzip within Win8. So I did that. First try didn't work, second try ended up with 8MB bin file in root of USB drive. What I hadn't done before was take the .exe file in /efi/boot/ and run it, which places the bin file in root of USB drive. The updating process was supposed to be described through an SOP link in the warning paragraph for the 23.2 BIOS level (which says leave the USB drive there through restarts), but that link was error 404. Another link at bottom of page gave screen saves through a process that had more steps than what I encountered when I merely clicked on a command on the Win8.1 screen to perform the BIOS upgrade. (Note: Win8.1 did install through the 22.5 BIOS, from about Oct 2012, and I think EFI was in effect, but I still (on Nov 18) don't see whether partition table is GPT or MBR.)

The BIOS upgrade took maybe three minutes, a time of anxiety. The screen left Win8.1 and was a simple screen with progress bar. Then there was a restart and I got back to BIOS setup, where the new BIOS level, 23.2, was displayed. Give thanks to the Lord. Not much difference in BIOS menus, but there is a Windows 8 entry. Within that is Secure Boot. I enabled that, but upon restart, it complains that EFI requires GOP video and turns Secure Boot back off. Investigating GOP video, it is Graphics Output Protocol and is a several-years push to replace VGA and interrupts legacy with something better. (Not that VGA connector is going away, maybe.) But I can't find any video card with Win8 qualification and GOP. But on Nov 18, GeForce GTX680 in the $350-600 class, two-wide, which is much more video than I want or have space for.

With the new-level BIOS, checked Ubuntu 13.04 via USB installability, it seemed to start up. I doubt that Secure Boot will work for it.

Returning to the Win8.1 scene, with Secure Boot off, tried DVD again to check installability. From a Microsoft web site http://www.microsoft.com/OEM/en/installation/downloads/Pages/Windows-8-ADK.aspx#fbid=Nen1RE8uhHz a default product key to get install started for Windows 8.1 Professional can be B478Q-FNMXX-FKYT6-D6FC8-HMGTD. Yes, it took off and installed. MS sent a new, seven-digit security code. Secure Boot may have to wait on cheaper, GOP video to come out.

Starting at Control Panel, I can find Bitlocker, TPM support (pretty much required for Bitlocker), and Storage Spaces. Don't know if those will succeed. Only have one RE4 drive (in the five-drive bay) hooked up for now, I can see how five-total RE4s connected to mobo SATA might not be the thing to do right now. Check out the 1.5-yr-old Wacom tablet with Win8.1, also need to put in Highpoint 640L card & hook it to three Seagate $55 drives.

Looking at Dell.com to see what type of Win8.1 towers they have, they are still back in Win7! Is Win8 usage so low?! (Dell uses Win8 for their all-in-ones which have touch screens and little need for enhancements.)

Found Win8.1 Disk Management. It shows a 100MB EFI system partition which might indicate GPT. The C: drive is NTFS, simple, basic, healthy.

For Wacom Bamboo Fun tablet, it took a 4-min driver download, now shows up in right area of Start screen. Setting up preferences. Responds to single finger, also two-finger scroll and two-finger zoom.

Nov 19: Got Wacom Bamboo Fun tablet installed, had to go to Wacom.com to get driver myself. Pressure works in Corel Painter Lite 15-day free evaluation download. But I did a sleep shutdown from kybd moon button and tablet didn't wake up when system came back on. Full shutdown brought tablet back.

Printer in back bedroom with AirLink print server: I got it working from ANODE Win8.1 but it took a long time, had to approach it through troubleshooter.

Nov 21: Microsoft Windows 8.1 Storage Spaces is working. I had to get the Highpoint Rocket 640L 4-port SATA JBOD adapter installed, and that was aided by this card being a newer release. With the three Seagate $55 drives connected to that, it turns out that Storage Spaces needs unallocated drives (no partitions) to form a storage pool. Once that was done through Disk Management, it was easy to set up two Storage Spaces, the first 300GB and the second 200GB. There is something like 800GB still available in the pool to make more spaces or maybe the existing two can be extended. I am sure the spare room on these three drives cannot be shared by Linux, they are locked into Windows Storage Spaces. If usage gets to 75% of capacity, Storage Spaces is supposed to ask you to add a drive to the pool, or two for one of the three setups (two-way mirror, stripe, parity I think) so Storage Spaces doesn't run out of room.

Getting Storage Spaces working is probably more important than getting Secure Boot working, so this is a big step ahead. To remember the process of getting Window 8.1 working, I took photos and made up three .pdf documents. Titles are Win 8_1 on GRID in pics.pdf, Win 8_1 on GRID in pics 2nd half.pdf, and ANODE computer Win8_1 prtsrver & StorSpaces Nov23 noon.pdf. Other informative documents: http://blogs.windows.com/windows/b/extremewindows/archive/2013/02/15/high-performance-storage-solutions-with-windows-8.aspx High Performance Storage Solutions with Windows 8 Feb 15, 2013 by Gavin Gear EFI System Partition (ESP) is harmed by Ubuntu through at least 12.04, so do not use old Ubuntu (http://www.rodsbooks.com/gdisk/booting.html).

How many Storage Spaces scenarios don't make sense? If you do two USB Flash memories as a pool, and one of them has a little trouble, does Storage Spaces ask you to replace the one with the trouble? Can you take the two USB Flash memories to another Windows 8 box and expect them to be recognized? (This would be a good experiment. In fact, can you remove a USB pool, either hot-unplug or while power is off, then come back later and hot plug them?) Does it make more sense to have external e-SATA drives as a pool or use a NAS box? (Which is better and cheaper?) In fact, is a NAS box accessible to multiple computers at once? Is it possible to take a two-HDD pool, connected to a little JBOD PCI-e adapter, to another Windows 8 box, or is it better to use NAS? (Like for video editing, which works faster?)

Internet: My 4 new 2TB WD My Passport USB drives [2.5" HDD with emphasis on continuous backup of files on the system connected to, formatted with NTFS] are not recognized by the Storage Spaces manager, even though they show up and work flawlessly in the usual Disk Management. A quick call to WD support confirms: These drives will not work with Storage Spaces. So, I will just have to return them.

Someone: Backup those 4 WD external disks. Delete each volume on the 4 disks in Windows Disk Management. Now open the Storage Spaces control panel. You should be able to use the disks to create pools and spaces.

I tried... I deleted the partition, I cleaned the disks with diskpart, I reinitialized them, tried MBR or GPT partitioning, with and without partitions on the disk. I even played with diskpart's setid. These disks simply don't work due to the way the USB controller lists them on the bus... And WD support confirms this.

Storage Spaces seems to be a Microsoft approach to sofware RAID. It is customized to use consumer-grade HDD which might have 90-second error recovery time (whereas WD RE4 drives have 7-second TLER) and it has the ability to declare storage spaces that are much bigger than the HDDs you initially connect, it just advises you add more HDD when usage reaches 75%.

Storage Spaces continues the trend toward software RAID. I picked up on this nine months ago and got onboard. My Linux mdadm experimenting on ANODE shows it is workable, but I don't know of GUI Linux tool.

Dec 7 2013: Windows 8.1 Secure Boot is working on ANODE file-server computer, with just Windows 8.1 installed (no Linux). As described on this web page at Nov 17, I had already done the MSI BIOS flash for Win8 support. The breakthrough today was substituting the Win8-logo-on-the-box Gigabyte $100 Radeon HD7750 video card for the NVidia 210 $35, reinstalling Win8.1, installing video driver from Gigabyte, and getting on a Microsoft web site http://technet.microsoft.com/en-us/library/dn479183.aspx by searching for Windows 8.1 Secure Boot isn't configured correctly, which is the watermark on the Win8.1 desktop that I have been staring at. The Microsoft web site said to use Powershell as admin and confirm-securebootuefi. This showed "false," and the troubleshooting guidance on the web site told me how to get to BIOS from desktop (yes, it can even be done through Settings... after a restart, without pushing DEL a lot) and there was a clue to use Secure Boot Mode Custom, which I had seen but been afraid of. Doing Custom got me to Key Management in the MSI BIOS. I did Factory Default Key Provisioning, then Install All Factory Default Keys, and the four of them changed to Installed. Save and Reboot brought up Win8.1, and going to desktop no longer showed the watermark, Secure Boot isn't configured correctly! Going to Powershell and doing the two commands confirm-securebootuefi and get-securebootpolicy showed true and the proper publisher (77fa9....764b). So this is a big breakthrough, and Microsoft did well to give guidance on the web site. Getting Secure Boot working has been one of the goals since June 17 of this year, when the ANODE computer started coming together (see June 17 above, on this web page). The Highpoint 4xSATA non-RAID (JBOD) adapter (640L) is recognized by Device Manager as a Standard SATA AHCI Controller and is not needing a driver from Highpoint, and, further, is not blocking Secure Boot since the driver is signed by Microsoft. There is another Standard SATA AHCI Controller, the one in the mobo for the six on-board SATA jacks.

Documentation of what happened is a 343MB folder named 2013-12-07 Win8_1 install where Secure Boot worked. See the .txt file within that folder for guidance to what happened, it was not smooth.

Went ahead to try HDMI cable from new RADEON video card to Sony TV, at same time as VGA CRT display. Well, the HDMI cable wasn't inserting enough into the card slot. I trimmed off a little plastic on the HDMI cable. Through Control Panel, am able to add TV as a duplicated display, though it puts CRT display into a 16:9 format that wastes a lot of screen. But it does duplicate onto the Sony flat-panel TV. But having the HDMI connection all the way from power-on blocks booting, so leave off HDMI during boot.

I notice in Win8 that Help tends to come from Internet. Skydrive is an important offering. All in all, Internet is integral to the utility of Win8, but that means Win8 has an always-on Internet connection, which makes a computer so much more susceptible to malware and intrusion. This may be why Secure Boot and BitLocker/TPM are there, to bring back security despite the always-on Internet connection.

Video cards that are Secure Boot enabled use the GOP (graphics output protocol) more than legacy VGA. I found an AMD 2011 conference presentation that has this info: GOP is enabled by UEFI driver to support graphic console output in the pre-OS phase (and yes, my MSI mobo's BIOS is graphical and mouse enabled in BIOS setup, & I have seen indications that higher-price mobos, while in BIOS setup, can go out on Internet for drivers)...ultimate goal of GOP is to replace legacy VGA BIOS and eliminate VGA hardware functionality....no more INT15 handshaking...GOP is written in C, not assembly.

Dec 9: tried booting from a live-boot Ubuntu 13.04 USB flash drive with boot-from-USB HDD temporarily placed first; no, it went to Win8.1. I was looking for an error seen by a Dell Win8 user, Image failed to verify ACCESS DENIED, when he tried live-boot Linux. http://technet.microsoft.com/en-us/library/hh824987.aspx has a line that is a bit ominous: the OEM locks the firmware from editing, except for updates that are signed with the correct key or updates by a physically present user who is using firmware menus, and then generates a platform key (PK). The PK can be used to sign updates to the KEK or to turn off Secure Boot. I don't know if I have a platform key. Hmmm--should the USB flash drive with Ubuntu 13.04 be

Got Bamboo Fun touch-sensitive tablet working with Win8.1. A Wacom driver wasn't specifically listed for Fun, I used another one and it is working, though two-finger swipe doesn't work well. Finger-based mouse function with finger left click is fine.

Dec 13: Turn on BitLocker, which is aided by the TPM I have on the mobo. Reading Microsoft Q&A about BitLocker, which is about 20 pages long, there are a lot of things to consider about BitLocker, and each drive encrypted has a password and a separate, 48-numeral recovery key to keep track of. Encryption of, like, a 300GB drive is 40 minutes. Sounds like Ubuntu GRUB or whatever Ubuntu might use to dual boot would mess with boot of Windows and trigger BitLocker, so before changing boot order or anything in BIOS, it would be best to turn off BitLocker, which seems easy to do, but I don't know if a long re-encryption is needed. Have to try that. There are so many passwords--these BitLocker passwords, Microsoft account password, Win8.1 password to get to desktop.

Jan 19 2014: Purchased Microsoft Press Windows 8.1 Inside Out by Tony Northrup. It has a lot of info and acknowledges difficulties. Page 261 compares Storage Spaces to RAID. I have never seen this before: You can't use your system drive as part of a Storage Spaces pool. Storage Spaces is only for data, it seems. If you want redundancy for the system drive, you must go with RAID.

Hyper-V, which is Windows 8 virtualization, can handle Ubuntu, see http://www.servethehome.com/run-ubuntu-windows-8-hyper-v-quickly/ . This seems to be an alternative to dual booting ANODE. "Ubuntu now includes Hyper-V integration components, it is an easy Linux distro to install in Windows 8 Hyper-V. For this guide we are using the 64-bit desktop version of Ubuntu 12.10. Just after this, we did the same with Ubuntu 12.10 server. No major difference." But can mdadm work from within an Ubuntu virtual machine? So many questions.

Nov 8 2014: Got the 3 Seagate HDD mounted into the Thermaltake case, clustered around the cavity for plug-in adapters. (They had been pushed out on the floor since the Highpoint Rocket 640L 4 SATA adapter went in.) Covers back on. Curious about Ubuntu on Hyper-V that is part of Win 8.1. Used my book, Windows 8.1 Inside Out as a guide. Had to enable Hyper-V, no problem. Used 20GB of unallocated room on first HDD to give a G: drive to put the VM folder in. Proceeded to Ubuntu 10.04 install from a CD. No problem. The Ubuntu VM computer name is ARGON, user name penguin.

It is important to see Hyper-V's note, to release mouse press ALT CTRL left arrow. To get to Internet, had to have Hyper-V add a legacy network adapter. OpenOffice works, was able to print to network printer. Chose to make Ubuntu monitor size 800x600, requires some scrolling that is a little cumbersome. The AMD uP six cores seem to be highly active when VM is running. This is a very neat experience. Could put more than one VM in Hyper-V.

Dec 11 2014: Found that when dual displays are "duplicated" in the Project menu, with HDMI cable connected to a TV, even with TV off, when you restart computer, the VGA display is ignored in favor of the TV, making you think the computer is broken. Unplug the HDMI cable at the Video Card output & VGA display wakes up, maybe with a restart.

Further experiments may show that another choice in Project makes VGA display work without hand wringing. Yes, select 2nd Screen Only.

But continued trouble after doing Project, 2nd screen only. With HDMI TV Sony connected but not powered on, black screen on VGA display even after disconnecting HDMI cable from video card, and multiple power off-power on.

But there was added unknown of some .2s AC power interruptions this morning. Got back VGA by turning computer power supply off with AC-input switch for 3 minutes, no HDMI cable to video card. It is disconcerting that VGA display is dark most of the 70 seconds that it takes to get motherboard going, & Win 8.1, with the two choices I have made to not do fast startup.

Tried speech recognition with quite a bit of success. Hard to find voice training through control panel but stumbled on it through Access. That training has some good tips. Help is hard to find, and a list of commands. "show numbers" is good.

Jan 6, 2015: Windows 8.1 Search for files in File Explorer has not been working, like through the search charm. Found the best way, a search bar that is always near top of File Explorer window, see this screen print.

By the Search Tools tab, you have access to familiar ways to limit the search. Plus, in the search bar it shows which limits you are using, see "by-laws type: odt datemodified:last year."

Feb 14, 2015: ANODE experienced third Win 8.1 system recovery, after locking up (no mou, kybrd; frozen screen) during an on-line PPTX to PDF conversion. Like the last time this happened, it couldn't repair itself & offered options. This time, the try at refresh showed lack of room on C:, & I took option of cmd line to delete OpenOffice directories & even refresh file (had to del both to open enough room). Then refresh worked, but it needed the install DVD. I took screen photos, & after it came back I did 3.5GB of data bkups onto a DVD.

Middle 2016 Microsoft regularly pushes out free opportunities to upgrade to Windows 10. I was always declining because I didn't know if Storage Spaces (encrypted) or TPM would cause an incompatibility with an upgrade. But one upgrade push got through without me realizing it. So ANODE became Windows 10. And it still seemed to be secure boot. Things were working.

But the system upsets (the modern equivalent of Blue Screen of Death) continued, maybe once a quarter, and once Win 10 was in there, I didn't have an install DVD for Win 10, and there came a time when there was no way to let the system recover. I installed Win 8.1 from DVD and it lost all data files.

I wondered if SRAM was at fault. I found online a free memtest86 (there is an advanced version you pay for, that probably tells you which DIMMs are failing) and in running it for hours on ANODE I found two DIMMs that had an error about once in three hours. All four DIMMs were the original PNY brand I had bought at Augusta Best Buy. At a Fry's I bought replacements. Since then, no system recoveries. But I did not go to the trouble of making Storage Spaces.

In the meantime, the deluxe homebuilt Win XP computer from about 2003, GETTER, suddenly failed. Probably the motherboard. I lost Microsoft Flight Simulator X, movie editing Edius, the use of the expensive video hardware, the use of the big graphics tablet, and Corel suite. After moving to Austin before Mason was born to Stephanie and Joe, I took all the insides out of GETTER, which included old parallel ATA IDE drives, bought new hardware (all SATA), and got Ubuntu 16.04 working around May 2016. I took some of the enterprise-grade 250GB Western Digital HDDs over from ANODE to newGETTER. NewGETTER with Ubuntu 16.04 is now my workhorse computer. Stellarium works, and LibreOffice, and Inkscape, and GIMP. and NetBeans with Ruby. Inkscape is now my printed circuit board layout software, and the expensive Wacom Bamboo tablet worked without driver problems in Ubuntu 16.04! But audio on YouTube doesn't work. Banshee DVD creator works, the first DVD data disk writer that has worked for me in all my Ubuntu installs. Spending the money to get new hardware to match up with Ubuntu 16.04 was a good deal.

With newGETTER working so well, I don't turn on ANODE any more. I used LVM for newGETTER but I haven't added partitions from other HDDs to get software RAID.