How to Install Maestro 4.x

While much of the installation process for Maestro 4 mirrors that for Maestro 3, there are enough differences to merit a separate accounting here. Maestro 4 comes with a very simple installer that copies the required files to the Maestro "home" directory and creates a Windows registry key with that directory's path. As with the Maestro 3 installation, you must have "Administrator" privileges to install and configure the RTX64 runtime environment.

Before you begin...

Make sure you purchase a workstation that satisfies the system requirements of RTX64 and Maestro 4. In particular, a multi-core (at least 4, preferably 8) processor is a must, with 16GB RAM. Note that the RTX64 3.4 Runtime may not support the latest and greatest Intel processor, so be sure to check RTX64 documentation for processor compatibility! You'll need at least two open PCI-Express (PCIe) slots on the motherboard to hold the PCIe-6363 and the NIC dedicated to the RMVideo link, plus an additional PCIe slot for a second network card if you will integrate Maestro with the EyeLink tracker.

Verify that the system's advanced programmable interrupt controller (APIC) and hardware abstraction layer (HAL) are compatible with RTX64. RTX64 does not support Intel's x2APIC specification. Windows 10 may boot into x2APIC mode if the motherboard supports it; in that scenario, you must disable x2APIC in the BIOS (or select xAPIC instead). Chances are the HAL will be "ACPI x64-based PC", which is compatible with RTX64.

For more information about the APIC and HAL, and how to verify and/or change the HAL via Windows Device Manager, consult the "Check the HAL" section in the installation procedures for Maestro 3.x. Be sure to change the HAL before installing RTX64. If you discover that you must change the HAL after installing RTX, be sure to uninstall RTX before proceeding.

Disable Intel SpeedStep, Turbo Boost, Hyper-threading, and C-States in the BIOS. These are various features that have been introduced by Intel in an effort to improve the performance and/or reduce the power consumption of Intel processors. IntervalZero recommends disabling SpeedStep because it may increase timer and interrupt latencies in RTSS applications. In initial Maestro 4 testing we disabled Turbo Boost (which dynamically raises the operating frequency of selected cores to respond to demanding tasks) because it was not needed and it was associated with a badly behaving driver and application (Intel Turbo Boost Max 3.0). Intel's hyper-threading technology is intended to improve performance by creating 2 "logical cores" per physical core in the processor. Thus, an 8-core processor with hyper-threading enabled appears as a 16-core processor to Windows. This is likely helpful only for a highly parallelized software application, which Maestro is not; so we recommend that the feature be disabled. However, if you choose to leave it enabled, IntervalZero strongly recommends that an even number of cores are dedicated to RTX64. Finally, "C-States" are idle, power-saving processor states; when a core is not being actively used it may enter one of these states (there are several levels). Returning to an active state from a C-state introduces more latencies. Furthermore, in initial testing our Windows 10 workstation would mysteriously reboot itself without warning or a memory dump, and some Intel forums indicated that this might have something to do with C-states on the Skylake-X series processor in our test machine. To be safe, be sure that C-states are disabled in the system BIOS.

Disable the Windows weekly maintenance task that defragments the hard drives. Over two months of testing Maestro 4, we observed that once a week the system would crash a number of times over the course of the day while Maestro was running; otherwise, it ran trials continuously for 8+ hours a day without issue. The periodic nature of the problem eventually led us to disable an automatic maintenance task (that is configured when Window 10 is installed) that would analyze and defragment the system hard drives on a weekly basis -- including the drive to which Maestro was streaming data files during testing. Once that task was disabled, no further crashes occurred.

To disable scheduled defragmentation, first open the Defragment and Optimize Drives application (from the search bar next to the Start menu). Ensure that "Scheduled optimization" is turned off. Then open the Task Scheduler application. In the tree view on the right, select the node Task Scheduler Library > Microsoft > Windows > Defrag, select the task labeled "ScheduledDefrag" and disable it. You should restart the computer to ensure these changes have taken effect. Of course, this also means you may want to manually defragment your hard drives from time to time (but note that solid-state drives do not require defragmentation).

Disable the Windows weekly memory diagnostic task. IntervalZero support engineers recommended that we disable this diagnostic task which, like the defragmentation task, is a part of Windows "Automatic Maintenance". To disable it, open the Task Scheduler, find the node Task Scheduler Library > Microsoft > Windows > MemoryDiagnostic, and disable the "RunFullMemoryDiagnostic" task.

Use default English language pack and system font!

Unfortunately, Maestro is NOT an internationalized application. It does not have support for languages other than English, nor fonts other than the default system font.

After installing Windows 10, be sure to verify that the language is English and the system font is the default system font. It was recently discovered that the parameters panel on the Trial Definition Form may be cut-off if this is not the case.

Install the RTX64 3.4 runtime environment

Follow instructions provided with the RTX64 3.4 runtime package (through Update 1 dtd 20 Jul 2018) from IntervalZero. As mentioned in the System Requirements section, you'll need a runtime license that supports 2 or more dedicated RTSS cores (with the rest dedicated to Windows); you'll also need a companion license to use the RT-TCP/IP stack. For a quad-core system we recommend the Entry edition (1-2 RTSS cores); for an 8-core system we suggest the Professional edition (up to 7 RTSS cores). There is no need to purchase the far more expensive SDK license, unless you plan to do code development with the RTX64 software development kit.

After the RTX64 package is installed, you will be asked to activate it by entering the license keys -- one for the RTX64 runtime and one for the RT-TCP/IP stack (activation requires an Internet connection). You then define how many cores are dedicated to RTSS and how many to Windows, and restart the computer. You'll now have two boot configurations: Win10 + RTX64 and Win10 only. When running Maestro, of course, you must always use the former configuration, which is the default.

After the restart, open the RTX64 Control Panel to configure RTX64 as follows:

• • Configure the RTSS Subsystem > Change internal system behavior: Set startup type to Automatic. Ensure the HAL timer period is 100us, and the default thread time quantum is 0. Check Free the Stack on TerminateThread call and Use priority inversion.

  • Configure the RTSS Subsystem > Configure Memory Allocation Behavior: Select Request from Windows. This should be the default. Leave the other settings unchanged. Note that this is contrary to IntervalZero's recommendation for deployed applications. However, Maestro has very modest memory requirements, allocates all memory at startup, and does not require any additional memory over the course of its operation.

  • Configure the RTSS Subsystem > Configure the RTSS Watchdog Timer: Disable the watchdog timer.

  • Configure the RTSS Subsystem > Configure Exception Support: Check Use Structured Exception Handling and Terminate the Faulting Process.

  • Configure the RTSS Subsystem > Configure Power Management Settings: Ensure that Windows idle detection is disabled. In addition, click on Windows Power Management and verify that the system is configured with the RTX64-Recommended power plan, which should be created when RTX64 is installed. This power plan disables hibernation, sleep, and other power-saving features that could impact real-time performance. For more information, see the page Real-Time Subsystem > Hardware Considerations > Causes and Management of Interrupt Latencies in the RTX64 Runtime Help.

  • Configure the RT-TCP/IP Stack > Configure RT-TCP/IP Stack Behavior: Ensure that the stack always starts with the RTSS subsystem by checking the box at the top of this page. Leave all other settings on this page alone. However, if you have problems connecting to RMVideo when you run Maestro, you might want to check the Run in Verbose mode box. Doing so will post information and error messages from the RT-TCP/IP Stack in the RTX Server Console window.

Unless explicitly addressed in the above list, you should leave all other settings in the RTX64 Control Panel unchanged. You will need to specify the interface for the RMVideo NIC, but that will be done in a later step (see below).

Install Maestro

Download the Maestro 4.x archive from the file cabinet on the Downloads page, unzip it, then open the resulting folder and run the self-extracting installer, Maestro4Setup.exe. The installer copies the Maestro program and support files to a "home directory" you specify on the local hard drive, sets up the program group for the application, and stores several Maestro-specific settings in the Windows registry. It also registers an uninstaller tool with the operating system.

After installation, you will find a shortcut to the main application executable in the Windows Start menu program group "Maestro.4x"; you can elect to pin this shortcut to the Start menu or the task bar. Since the installer modifies the Windows registry, you will probably need Administrator privileges to run it. [NOTE: If the installer fails to write the registry correctly, Maestro might start but it will expect the program files to be in a (hard-coded) default directory, C:\Maestro. If that directory does not exist, the application will be unable to find and start its RTX64-based hardware controller. Worse yet, if it does exist but contains an incompatible version of the hardware controller, you'll get some odd behavior or a system crash!]

As of version 4.1.1, Maestro gives the user a modicum of control over the timing of the three device-level register writes required to deliver a 16-bit "latched" digital command to any external device in the Maestro DIO interface rack (write DO<15..0>, set the active-low TTL "DataReady" latching signal to 0V, then restore it to 5V). Prior to this change, there was only a hard-coded software busy wait of 2.5 µs inserted after the second write in an effort to ensure that the DataReady pulse was at least that long. However, testing showed that, under certain circumstances, a register write could be delayed on the order of several microseconds. As a consequence, the three register writes were apparently "queued up" and then executed in rapid succession -- so that the DataReady pulse was less than 0.2 µs long. While this may not be an issue for the DIO interfaces in use in the Lisberger lab, it is a problem in labs using a different DIO interface design. If a latched digital device fails to detect an incoming command, you'll see missed marker pulses or rewards, problems synchronizing with the Plexon or Optiplex system, etc.

To address this issue, the installer includes a wizard page that lets you specify the busy wait duration after each of the three stages of a DO command. Each busy wait time is specified in microseconds, between 0 and 20; the default values are 0.5, 2.5 (for the DataReady pulse length), and 0.5. The busy waits are stored in the Windows registry; Maestro retrieves them each time it launches.

To uninstall Maestro, go to the Start menu and open the Settings panel. Select System, then Apps & features. [Another way to get here is to use the search bar to find "Add or remove programs".] You'll see a list of applications installed on your workstation. Scroll down the list until you find the entry for Maestro. Click on that item and select Uninstall.

Note that, as of version 4.1.1, you do NOT have to uninstall Maestro prior to installing a new release. The installer will detect the existing installation and simply replace the program files in the existing home directory, then update the Maestro registry settings as needed. However, if you want to move the installation to a different directory on the hard drive, or downgrade to an older release, be sure to uninstall Maestro first.

Install and configure hardware

Background. Prior to Maestro 3, successfully installing and configuring the DAQ and network cards in the computer proved to be the most elusive step in getting the program up and running. This is because we had to ensure that the DAQ board and the network card linked to RMVideo each had exclusive access to an interrupt request (IRQ) line. With IRQs a scarce PC resource, this was often difficult to do. This problem went away with the advent of devices supporting message-based signal interrupts (MSI). The current required hardware boards all support MSI.

Hardware vendors typically recommend that you install their device driver and support software before you install the device itself. However, Maestro does not rely on any vendor-supplied device drivers; its RTX hardware controller talks directly to each device using RTX device-management functions. For this to work, the device must be transferred from Windows to RTX64 control. For any RTX64-supported network adapter, this transfer is very easy because the necessary driver INF (setup information) and CAT (security catalog) files come with the RTX64 runtime installation. The CAT file is appropriately signed and cross-signed, a Windows requirement since Windows Vista.

For the PCIe-6363, we generated the ni6363_rtx64.inf file based on an IntervalZero-supplied template; this essentially maps the PCIe-6363 device to the generic RTX64 plug-n-play driver (rtx64pnp.sys). Using the Windows Driver Kit tool inf2cat.exe, we created the corresponding security catalog file, ni6363_rtx64.cat. However, Windows 10 will not accept the CAT file unless it is signed and cross-signed using a code-signing certificate with Windows Authenticode support. In lieu of applying for and purchasing a code-signing certificate, we elected to leave the CAT file unsigned and circumvent the issue by temporarily disabling the driver signature enforcement. This complicates the procedure, detailed below, for transferring the PCIe-6363 to RTX64 control.

Steps to install and configure Maestro-specific hardware:

• 1. Turn off the computer. Carefully install PCIe-6363 DAQ card and Intel Gigabit CT Desktop adapter (or any other RTX64-supported NIC that will serve the Maestro-RMVideo communication link). The PCIe-6363 and the Intel CT are installed in PCI-Express x1 slots (you can use a x4, x8 or x16 PCIe slot if necessary). If you plan to use Maestro with the EyeLink tracker, you'll need a second network card for that connection. We do this separately (see step 5) because it may be difficult to identify which NIC is which in the Windows Device Manager, especially if they are the same brand. Restart the computer.

  2. Transfer ownership of the Intel Gigabit CT adapter from Windows to RTX64. Open the RTX64 Control Panel and click on Manage RTX64 devices to open the Device Manager. Find the device node for the Intel CT under the Network Adapters section; it will be labelled "Intel 82574L Gigabit Ethernet Controller" or "Intel Gigabit CT Desktop Adapter". Right-click the node and select Update Driver Software. In the popup dialog, choose Browse my computer for driver software, choose Let me pick from a list of device drivers on my computer, choose Have Disk... and use the browse controls to select the file rtxpnpnet.inf in the inf folder within the RTX64 runtime's installation directory. This the setup information file for the RTX64-supplied NIC driver. Choose Next, and Windows will automatically install the RTX64-compatible NIC driver, which is labelled "Intel 82574L Gigabit Ethernet Controller (RTX64)" for the Intel Gigabit CT. The NIC is now under RTX64 control. The same process applies to any RTX64-compatible NIC; only the device driver names are different.

  3. Temporarily disable driver signature enforcement, then transfer ownership of the PCIe-6363 to RTX64. As described above, we need to disable driver signature enforcement since the driver security catalog file we supply, ni6363_rtx64.cat, is not digitally signed. The required steps are outlined in the text box on the right; it requires a system reboot. When the system comes back up, follow the same procedure outlined for the network card in step (2). The PCIe-6363 device node will likely be located in a section called Other devices or Unknown devices. Right-click the node and select Update Driver Software, choose Browse my computer for driver software, then use the browse controls to find the drivers folder in Maestro's installation directory. The required driver files are in this folder. Choose Next, and Windows will automatically install the RTX64 generic plug-and-play driver as the driver for the PCIe-6363. It will be labelled "National Instrument PCIe-6363 (RTX64)" and will be found under the RTX64 Drivers section in Device Manager. The DAQ card is now under RTX64 control, although it's a good idea to reboot the computer after completing this step.

  4. Configure the RT-TCP/IP subsystem to use the Intel Gigabit CT network adapter. Open the RTX64 Control Panel and navigate to the page Configure the RT-TCP/IP Stack > Manage Interfaces and Filters. All network interfaces in use are listed on the left-hand side of the page, while the properties for a selected interface are shown on the right side. RTX64 comes with a virtual network adapter, RtVirtualNic, which Maestro does not use. Ensure that interface is disabled. Next, click the black "+" icon to add the interface for the Intel Gigabit CT. In the popup dialog, select Device Name = "Intel 82574L Gigabit Ethernet Controller (RTX64)"; the Driver Name will be "RtE1000.rtdll" (of course, these entries will be different if you elect to use another NIC). Set the Ipv4 address to 10.1.1.2, which is the only IP address from which RMVideo will accept a client connection (the NIC on the RMVideo side must be assigned to 10.1.1.1). Set the Netmask to 255.255.255.0 and leave the Location unchanged. Click OK to confirm the entries and add the interface. You should now see the RtE1000 interface in the right-hand list and it should be enabled. Verify that the interface has these property values (these are all defaults, so you should not have to make any changes!): Interrupt thread priority = 64; MTU = 1500; receive thread priority = 63; # of receive buffers = 256; # of transmit buffers = 256; interrupt type = MSI-X (or MSI). To ensure the changes take effect, you must restart the RT-TCP/IP Stack.

  5. If you plan to use the EyeLink 1000+ eye tracker with Maestro, you will need to install a second network card for the Ethernet connection to the EyeLink's Host PC. Turn off the computer, install a suitable NIC (such as the Intel Gigabit CT) in an available slot (and label it on the outside of the computer!), and restart. Windows will automatically install the driver for the NIC. However, you'll need to manually set the IP address for that NIC to 100.1.1.2. In Windows 10, go to Control Panel > Network and Internet > Network and Sharing Center. Click on Change adapter setting, double-click on the network card icon that represents the NIC you just installed, and select the Properties button. Select the list item "Internet Protocol Version 4" and click the Properties button below the list. In the pop-up dialog, select "Use the following IP address" and enter 100.1.1.2. Enter a subnet mask of 255.255.255.0. Leave the default gateway and all other settings blank. Click "OK" (several times) to save the changes.

Run Maestro and verify operation

Start the program and wait for the main frame window to come up. Open up the Message Log (if it is not already open) and check the startup messages. Ensure all of the boards you installed have been recognized and are available for use. Both Trial and Continuous operational modes should be enabled.