Server, Take Two

July 2 to September 10, 2011

I love my server. It serves files and printers, backs up my PCs, and serves as a fairly large Personal Video Recorder. I use its PVR function daily, watching recorded HD and SD TV shows while being able to quickly skip through commercials. It's just like a giant TIVO, any combination of up to 8 HDTV tuners or 8 analog TV tuners is available for simultaneous live TV or recordings, and enough storage to keep about 80 hours of HDTV for later viewing.

After about 18 months using the heck out of it, I realized that I had one nagging problem, I would see one or two little drop-out/stutters in almost everything it recorded. Something was grabbing the CPU, basically locking it into one task for several (sometimes many) milliseconds. The result of this is transparent under almost any normal computer use. I know there are are other apps that are sensitive to this misbehavior, I notice it primarily under these conditions:

1. When using rendering software, or 3D design software and rotating or panning the view, the program seems to hang momentarily (This is just as often caused by exceeding the system's capabilities causing pauses while it catches up.)

2. When using SoundEasy, if this occurs during an MLS measurement, the results will be seriously affected.

   1. If the interruption lasts long enough during video recording or playback, the input buffer of the program will be emptied of data, and the result will be a small skip. For interruptions lasting a second or more, this skip may be large, causing anything from odd video behavior to a system hang on the machine trying to play the media. THIS INCLUDES STREAMING VIDEO APPLICATIONS LIKE BEYOND TV

What I was experiencing were occasional dropouts, where the video might hang while the audio continued, then everything would skip forward a few frames and continue running in-sync without issues after the skip. Occasionally, this would happen long enough to simply stop the show for a second or two. This occurred more often during playback, but about 25% of these errors occurred during the recording process (easy to tell, just replay the glitched portion, if it was not in the recording, the glitch wouldn't happen again).

I finally got irritated enough to really pursue the cause of this behavior on my WHS machine. I hooked up a monitor, keyboard and mouse, logged on and started running DCP Latency checker. What I found looked like this.

The groups of three to five consecutive latency measurements near 1000 µs are periods when something was capturing and holding the CPU to a single task while the remaining services and programs are forced to wait. If a streaming application is playing or recording a video, the buffer receiving data may run empty. When the buffer overrun happens, you get a skip/hang/video glitch. If smaller groups of 1000 µs peaks occur repeatedly, the buffer will be operating almost empty for some longer period of time, repeatedly being overrun, causing stuttering.

On my systems, this has never been a problem with wave audio playback, and occurs very rarely during DVD play back. It finally became bothersome enough during HD TV recording and playback to prod me into looking for the cause.

I stopped all the services I wasn't afraid to stop, disabled everything on the server but the NIC and TV cards, everything I could think of, no joy... I always had the groups of high latency periods. Even with the device drivers for the NIC & TV cards turned off, the system still had the groups of high latency periods.

MY SOLUTION AND RESULTS

I decided to trounce the problem. Most WHS setups run fine on really light hardware, especially audio file servers. Hardware performance starts to become a problem when you expect the machine to transcode served content or actively record streaming media with bit rates above about 10 Mb/sec. The problem for me is the extra performance required by Beyond TV. The subtle problem I described above requires quite a bit more horsepower for error free operation.

Also:

Seldom are my choices vindicated, but I just couldn't bring myself to learn Sage TV and stayed with Beyond TV when I decided to build my server the first time. Google bought Sage TV back in June 2011. As of today minor updates and bug fixes are being released, but you still can't buy the software or hardware they were selling back in June. It's silly for me to speculate (but I'm gonna); It appears as though Google is not going to sell the product. I don't know if they are simply killing it for some reason, or plan to use the technology some time in the future, but it looks like Sage TV, as a viable server/client PVR solution is now way deader than BTV.

I didn't think twice about using BTV this time around, you can still purchase the software, it's stable as a flat rock, fast and responsive. Their Electronic Programming Guide is used in their current business software offerings, thus it is supported and appears to be usable for the foreseeable future.

The fact that the program interface hasn't changed in years is of little concern to me when a program performs as well as BTV. Until something extraordinary happens to how we receive TV, BTV looks like the program of choice (of course, now that I've said this, I've probably jinxed it and BTV will disapper...)

If you are interested in BTV's software, look here, it's hard to find on their website.

• • The resulting Server/TV Tuner Farm can record up to 8 channels simultaneously, in any combination of clear QAM or Analog formats (striaght off the wire, no set-top box required) from my Cox Cable Basic Service account.

  • With this hardware, Beyond TV can record to a pooled drive without problems. I ran the system for a couple of weeks using a share in the drive pool and noticed no problems.

  • Of course, Drive Extender Migrator (DEM) still needs to be turned off in WHS whenever BTV is recording or playing content.

  • If I'm watching something at midnight, I'll see about 2 to 5 minutes of mild to bad stuttering while Ckdisk runs.

  • The boot problem with BTV remains; The Advanced Admin plugin is still required. When booting the server, you must log in and minimize the WHS Connector window. Only then will BTV's start-up run.

  • New this time around. I discovered a thread about BTV loosing its QAM channel maps. The solution seems to be working so far (about 60 days), but I've gone longer than that without problems in the past. The solution? In the Set-Up Wizard:

    • Delete all cards and channel maps.

    • Install a QAM tuner and get a good QAM map, then install the remaining QAM tuners.

    • Install an Analog tuner and get the Analog map, then install the remaining Analog tuners.

    • Turn Lineup Change Detection OFF.

    • It's fine to leave Guide Data Updates on.

The WD Black drives are quite a bit faster than the WD Green drives I replaced, especially in the drive pool.

For the Green Caviar Drives, large file read performance was (as reported by HD Tune Pro v4.61):

• 1. 500 GB System Drive, average read speed was 78.0 MB/sec (Max - 105.1 MB/sec, MIN - 36.6 MB/sec)

  2. 500 GB NON-POOLED Drive, average read speed was 74.5 MB/sec (Max - 97.9 MB/sec, Min - 45.2 MB/sec)

  3. Two 1 GB POOLED Drives, average read speed was 61.4 MB/sec (Max - 78.4, Min - 36.6 MB/sec)

For the Black Caviar Drives:

• 1. 750 GB NON POOLED Drive, average read speed was 105.3 MB/sec, a 41% increase (Max - 132.6 MB/sec, Min - 64.7 MB/sec)

  2. Two 1 TB POOLED Drives, average read speed was 104.6 MB/sec, a 70% increase (Max - 132.2 MB/sec, Min - 61.38 MB/sec)

In the same Antec 300 case I used for the first build, they run about 10° F hotter. They idle at about 105° F and run up to 115° F under a heavy recording load.

I was afraid of noise, but there's no problems with the 500 GB and 1 TB drives, they are as quiet as their Green counterparts as far as I can tell.

The 750 GB drive was a real disappointment, it sounds like a little bitty box of tiny rocks being shaken violently and and occasionally being banged on a little counter top as it works. So far, the SMART parameters look OK, but I'm glad I'm not depending on it for serious data storage, I'd have to replace it. It occasionally sounds that bad.

What I Did (details if anyone needs them)

I upgraded the hardware a bit (still mainstream, non-bleeding-edge stuff though):

• • CPU

    • Intel i3-2100 (3.2 GHz, 65Watt) dual core Sandy Bridge architecture (the cheapest one in the line-up, it does provide video if used with an acceptable MB)

• Motherboard

  • • ASRock z68 Extreme 4

    • 5 PCIe slots (1x cards work in 16x slots. It looks funny, but works fine.)

    • z68 BIOS supports both overclocking (I couldn't care less) and video (just what I needed for a headless server)

    • SATA v3.0 HD support (total of four v3.0 ports)

• • RAM

    • Two 2 GB sticks, allowing two channel operation

    • The 4 GB of RAM didn't affect the 2250 TV cards

    • Also, the Intel Video chip solution in the z68 BIOS will grab about 1 GB of system RAM, even with everything set to minimums. Windows Server 2003 reports 2.7 GB of RAM available.

  • TV Cards

    • Hauppauge 2250 dual hybrid tuner cards (two hybrid tuners for Clear QAM, ATSC (OTA), or Analog TV and one FM radio tuner per card)

    • Had great success with two of these cards the first time, Installing two more for a total of four this time.

  • Hard Drives

    • Using Western Digital Caviar Black drives

    • SATA v3.0 drives, 6 Gb/sec throughput

    • 7200 RPM

    • 500 GB for system drive

    • 750 GB for TV recordings (not included in the drive pool)

    • Two 1 TB drives for the pool

    • These are not advanced format drives

  • Intel PCIe NIC

I stayed with WHS v1 because I'm used to it and I like the drive pool. The only programs running on the server are WHS and the Beyond TV recording service.

I put everything together and installed AHCI drivers for the HDs this time around. I had to get a little 1.44 MB Floppy Drive (USB Port interface) and do a real, honest to God "floppy" install (press F6 when Windows installer asks you if you have any SCSI drivers to install). I couldn't get any of the slipstream builds to work, but the USB floppy does work.

Some things to remember for the installation:

• 1. Install only the HD to be used as the system drive during the WHS installation.

     1. WHS will place all installed HDs into the pool automatically, if you intend to keep any permanent hard drives out of the pool, install them after WHS is up & running. I could have left two 1 TB drives connected during the install, it just seemed easier to add them later.

  2. Plug in the USB floppy and insert the floppy with the required AHCI (F6) drivers BEFORE booting the PC. The BIOS needs to register this as a floppy and assign the traditional A drive letter during the boot. (Guess how I know...)

  3. In BIOS, set system HD to AHCI mode

  4. In BIOS, TURN OFF ALL hardware you are NOT going to use, for me

• • 1. On-board NIC

    2. On-board 1394 port

    3. On-board USB 3.0 chip & ports (I'm using an eSATA dock for external hard drives)

    4. On-board serial port

    5. On-board IR port

    6. The On-board floppy drive port was turned off after the install.

       1. I doubt it matters, but this was turned ON during the install (I was too tired to care, I just assumed that it might matter).

       2. This was also causing 1000 µs latency spikes every 90 seconds or so, even with the floppy device turned off in Device Manager. After turning the the floppy drive port off in BIOS, they disappeared (they came back later, I can't eliminate them).

Step-by-Step, this is what I did for a headless WHS:

• 1. I started with USB port floppy drive, NIC, system drive, keyboard, mouse, & monitor installed.

  2. Installed AHCI drivers with the F6 command during the initial stages of the WHS install (when the old XP style blue screen with yellow text comes up)

  3. After WHS is installed & running, I logged in to the WHS & installed Chipset inf files & rebooted.

  4. Installed .NET (at least through 3.5, you'll need it to run the Intel NIC driver installation program and BTV also uses .NET)

  5. I ran Intel's Rapid Storage set-up program

  6. Installed the Intel NIC driver

  7. Install other AHCI HD drivers as necessary (Marvel SATA ports on MB)

  8. Go to Windows Update page and install updates, OS first, then .NET (about 130 in all, daum)

  9. After updates are complete, shut down & install TV cards. The next boot (and all boots after this point) will take a looong time while the BIOS & WHS negotiate all the new PCIe connections.

  10. Install TV card driver. IMPORTANT, use v7.6.27.27323. The newest driver is buggy with many systems, reporting incorrect file run times & lengths.

  11. Install sound card driver

      1. This driver is only necessary to hear the channels during the Beyond TV setup. If you can do without during the BTV setup, just turn the sound chip off in BIOS, and do not install a sound card driver. BTV doesn't need one.

  12. Install printer drivers if you are serving printers (Do NOT install any printer "monitoring" programs, these caused stuttering on my machine).

  13. No other drivers are necessary, the default Windows VGA driver is good enough for the WHS and BTV installations

• • 1. The Beyond TV Setup Wizard will generate several video errors when it first tries to generate a running TV screen. Ignore these (click on OK) and BTV will default to a non-overlay mode that works with the default windows driver and the Intel graphics chip.

    2. Also, when you run BTV on the server to make sure it works, you will get the same error messages. Again, click through them and BTV will run fine. Full screen HDTV looked quite nice.

A few things I finally figured out while configuring Beyond TV this time:

• 1. Don't worry about how the card tuners are used in BTV.

  2. I have four 2250 TV Tuner cards, with two hybrid TV tuners per card. This provides me with 8 TV tuners and 4 FM Radio tuners.

  3. Simply install all 8 tuners as QAM tuners 1-8

  4. Then install all 8 tuners as Analog TV tuners 1-8

  5. Then install all four FM tuners as FM tuners 1-4

With this configuration, BTV will record any combination of up to eight simultaneous recordings, QAM or Analog, assigning tuners as needed. So far, this has allowed me to watch live TV, or listen to live FM music during heavy recording periods, without being blocked by a "need to stop recording" message like I used to get.

THIS IS ABSOLUTELY REQUIRED:

You don't need to use my schedule, but you should set the Drive Extender Migrator Service to "stopped" and "disabled" when you want to record or watch TV shows. I use Scheduled Events on the server to do this.

I set DEMigrator to turn OFF every week day at 5:00 PM

I set DEMigrator to turn OFF every weekend day at 6:00 AM

I set DEMigrator to turn ON every night at 2:00 AM

I use Scheduled Events to run the following batch files at the appropriate times.

The exact contents of DEStart.bat:

C:\WINDOWS\system32\sc.exe \\localhost config DriveExtenderMigrator start= demand

C:\WINDOWS\system32\sc.exe \\localhost start DriveExtenderMigrator

The exact contents of DEStop.bat:

C:\WINDOWS\system32\sc.exe \\localhost config DriveExtenderMigrator start= disabled

C:\WINDOWS\system32\sc.exe \\localhost stop DriveExtenderMigrator

The results? Pretty good for cheaper lower level hardware.

This is now the lowest latency system I own. To be fair, after everything was installed and working well, the actual latency looks like this.

The system operates at an avg latency of 40µs when running idle, with BTV service running in the background (not recording). Something is still grabbing the system for 1000 µsec about every 60 to 90 seconds.

The good news? This really isn't a problem. The hardware is so fast that data buffers in running programs have plenty of opportunity to stay full before each spike, and can easily recover after the spike occurs. The result? My drop-outs have finally disappeared :-)