HMC in VirtualBox

Note that this document is just a theoretical assumption about an otherwise unauthorized use of the HMC code.

From "IBM License Agreement for Machine Code":

IBM grants Licensee a nonexclusive license to use Machine Code without Circumventing Technological Measures and only as follows:

a. to enable the Covered Machine to function as designed by IBM;

b. to access and use Authorized Built-in-Capacity; and

c. to execute and display Machine Code as reasonably necessary to maintain the Covered Machine in accordance with the IBM-designated maintenance package for such Machine type.

No other licenses or rights (including licenses or rights under patents) are granted either directly, by implication, or otherwise. No other use of Machine Code is authorized.

Information valid as of January 2015

Requisites

  • VirtualBox OSE
  • Free disk space (see below)
  • At least 4GB 2GB free memory - see references
  • 64-bit host CPU
  • HMC ISO recovery images from IBM.com

An example setup:

  • Linux 64bit host
  • Intel Core i7 @ 2.4GHz CPU
  • 8GB of RAM ("Host RAM: 7882MB total, 6830MB available")
  • 120GB SATA-150 2.5" 5400rpm disk via USB2.0 enclosure, ext4 filesystem
  • VirtualBox 4.2.18 OSE
  • HMC recovery v8.8.2.0 (Procedure tested on v7.7.9.0 +SP1 as well)

Create VM

General

Linux other 2.6 64bit (v7: 32bit)

System

4GB RAM (v7: 2GB)

Boot order: CD, HDD (uncheck FDD)

Chipset: ICH9

Processor: 2 vCPUs, 100% cap (unlimited), enable PAE/NX

Acceleration: enable VT-x (set automatically)

Display

64MB video memory (no particular recommendation)

Storage

IDE: attach recovery ISO (v7.x and below)

SATA: Create disk image

Type: VDI

Size: 80GB dynamically expanding (v7: 40GB) - v8 will not accept a 40GB disk (no matter if fixed or dynamic)

Network

Network 1 (HMC public) - this will be eth1:

- Intel Gigabit server adapter

- NAT

- Cable connected

Network 2 (HMC private) - this will be eth0:

- Intel Gigabit server adapter

- NAT (this will be later connected to host physical adapter and "HMC network")

- Cable not connected (not required yet)

Audio, serial, USB, shared folders

Not relevant/not applicable (maybe only USB)

NAT and port forwarding

You need the guest's IP address assigned via DHCP, and make sure nothing is listening on these ports in the host.

Host firewall settings should be changed accordingly.

Privileged host ports =<1024 will be allowed in the VirtualBox NAT settings without warning, but will not work anyway.

To check which ports are open on localhost, use netstat or (Unix) lsof.

Forwarded ports:

ssh 22 -> 127.0.0.1:22222

https 443 -> 127.0.0.1:20443

https-12 12443 -> 127.0.0.1:12443

https-11 11443 probably unnecessary

wasadmin 9060 probably unnecessary

Adjust VirtualBox BIOS parameters

The HMC boot scripts check if the machine model is supported by the given HMC release.

$ VBoxManage setextradata "vmname" "VBoxInternal/Devices/pcbios/0/Config/DmiSystemVendor" "IBM CORPORATION"
$ VBoxManage setextradata "vmname" "VBoxInternal/Devices/pcbios/0/Config/DmiSystemProduct" "7042CR5"
$ VBoxManage setextradata "vmname" "VBoxInternal/Devices/pcbios/0/Config/DmiSystemSerial" "06AAAAB"

HMC v8 requires at least a 7042-CR5 (existing other models are 7042-CR6/CR7/CR8, as of Jan 2015).

The serial number is needed for the new pre-GA GUI only. The classic GUI runs fine without it.

See also: http://omnitech.net/reference/2013/05/01/installing-hmc-in-virtualbox/

Install HMC

The installation takes about half an hour.

See this screenshot tour: http://imgur.com/a/8AO2X

  • Boot from the ISO
  • At the Install Wizard panel, choose "Install"
  • At the Install Confirmation panel, choose "Finish"
  • If you get the "An error occurred during the install/upgrade", you will be put in a debug shell.
  • Look at the error log: cat /tmp/HmcInstall.err
  • In my case, the error ("No partitions found error") didn't come up during the next install attempt
  • At the "Restore critical console data" panel, choose finishing without restoring data

The VM will reboot and initialize the graphical interface. The initialization takes a few minutes. Note that the growing filesystems are now on LVM.

Post-install steps

  • At the Change Locale window, either choose "Exit and don't prompt" or "Change Locale" if you wish
  • At the Accept License windows, choose "Accept"
  • At the Guided Setup wizard, choose Yes (but you can run this later)
  • Set timezone, date and time
  • Set hscroot and root password (note that hscroot can change root password anytime)
  • Configure interfaces. Identify them by MAC address.
  • Public interface: Set as DHCP client.
  • Private interface: Run DHCP server, select a range that's different from Virtualbox's DHCP (10.0.0.0/36)
  • Wait for the Console Application graphical interface to restart
  • Skip/Uncheck Call Home Wizard

Initial configuration

A the logon window, you can choose from the interface style, differences are explained in a pop-up window.

I assume the use of the Classic UI, unless noted otherwise.

Work under "HMC management".

  • Change network settings > LAN adapters > eth1 (public) interface: enable SSH (22) and the remote console (443, 12443, 9060)
  • Change network settings > Name services > Disable DNS and disable DHCP DNS
  • Disable DHCP DNS seems to have no effect
  • Routing requires
  • Change hostname
  • Remote command execution (enable SSH)
  • Remote virtual terminal (enable remote vterm)
  • Remote operation (enable Web UI)
  • Manage certificates > Create/New certificate (self-signed): see below, not required for HMC v7

Verify the IP address acquired via DHCP and adjust NAT/port forwarding as described above. Ports should be available on the host's loopback immediately.

Getting the v8 new "Pre-GA" UI to work

It shows "Power Management Console" which predicts another rebranding... The new UI is based on WAS 8.5.

Serial number

Although the Classic UI is fine without a serial number, the new one does require one.

File: /var/hsc/log/wlp/FFDC.log

<2048>1 2015-02-01T14:58:25.499+01:00 vmhsc02 REST - PMC00001 [pmc.jaxb.reflection@2 C="com.ibm.pmc.jaxb.impl.common.GenericPattern" F="GenericPattern.java" L="56" M="getValue" P="WARN" T="Liberty PMC-thread-24"] PMC00001: The supplied value failed the pattern check with error message "ERROR: Pattern class SerialNumberPattern with raw value "0" failed to match the regular expression pattern "[\w\-\._]{2,11}".".

Serial number has already been added to BIOS settings, see above.

SSL certificate

None of the menu items worked, they just didn't appear. I examined the logs via ssh, and I found that there's a problem with the server cert, perhaps because the hostname was changed.

File: /var/hsc/log/wlp/messages.log

[2/1/15 13:35:25:993 CET] 000000b6 com.ibm.ws.channel.ssl.internal.SSLHandshakeErrorTracker     E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

Upon closer examination, I found a script compareHostnameWithCertificate.sh which attempts to dump the 'hmcserver' key to an X.509 certificate from the default keystore, then find a line with "DNS:" in it.

File: /var/hsc/log/wlp/wlpd.log

#--- compareHostnameWithCertificate.sh: The HMC /etc/hosts content is:
127.0.0.1 localhost.localdomain localhost
192.168.128.1 hmc
#--- compareHostnameWithCertificate.sh: The HMC SSL certificate DNS hostnames are:
#--- compareHostnameWithCertificate.sh: Comparision results:
ERROR: None of the HMC SSL certificate DNS hostnames match the current HMC hostname!

So I generated a new cert in the Classic UI.

Update HMC

Download the latest fix pack ISO from Fix Central. For V8.8.2.0, you need the mandatory fix MH01454 (as of Jan 2015).

Load the ISO in the virtual IDE CD drive.

Updates > Update HMC > Removable media > CD-ROM or DVD-RAM

After the fix is installed, you might be prompted for a reboot.

The details of the installation are in the file /tmp/HmcInstall.log .

Version information

hscroot@hmc:~> lshmc -V
"version= Version: 8
 Release: 8.2.0
 Service Pack: 0
HMC Build level 20141006.3
","base_version=V8R8.2.0
"

- Kernel 2.6.32-358.23.2.53.hmc7_4p.x86_64 - maps to RHEL 6.4: https://access.redhat.com/articles/3078

- New UI: WebSphere Application Server 8.5.5.0 (wlp-1.0.3.20130524-0951)

After installing MH01454:

"version= Version: 8
 Release: 8.2.0
 Service Pack: 0
HMC Build level 20141104.1
MH01454: Required fix for HMC V8R8.2.0 (11-05-2014)
","base_version=V8R8.2.0
"

Resource usage

CPU: Typically ~30% while idle, max when initializing console

RAM: v8.8.2 RSS kb. 3,5GB

HDD: dynamic disk ~10 -> 11GB (v8), ~12GB (v7)

v8 VirtualBox process running the Classic UI, initializing:

VIRT   RES   SHR S CPU% MEM%
5218M 3776M 3658M S 98.5 47.9

v8 VBoxHeadless process, idle:

VIRT   RES   SHR S CPU% MEM%
4560M 3367M 3267M S 12.6 42.7

Memory use in the VM meminfo:

Total 4GB

Free 1.2GB

Buffers 52MB

Cached 764MB

Hacks

Root access

- In the boot loader, edit the menu entry and append " init=/bin/bash"

- Appending "1" or "single" will not work (it reboots the HMC)

- After you are dropped in the shell, login as root.

# mount -a
# mount -o remount,rw /
# mount -o remount,rw /var

Now you can copy a shell binary to the restricted path, preferably with a distinct name.

# sync; sync; sync

Note that the shutdown/reboot command will not work and you might have to reset the VM.

You can now use an unrestricted shell as hscroot and su to root.

Note: PAM settings restrict su to hscroot only.

Unrestricted shell from local browser

Locally in the new UI, you can break out/execute any command by right-clicking on Pins -> Save As... -> Open with..., see screenshots.

User 'browser' running Firefox can gain access to an unrestricted shell, by executing /usr/bin/xterm, for example.

Watch SSH

Watch SSH port open from the Linux host

$ while : ; do echo "Attempting ssh" ; ssh -o ConnectTimeout=120 hscroot@127.0.0.1 -p 22222 2>/dev/null; echo Sleeping 10; sleep 10; done

List and terminate Web browser sessions

Standard HMC commands

$ lslogon -r webui -u -F session_id
$ lslogon -r webui -u -F session_id | while read sess; do termtask -r webui -s $sess -t all; done

VirtualBox headless mode - runs in the foreground

$ VBoxHeadless -s hmc8
Oracle VM VirtualBox Headless Interface 4.2.18_OSE
(C) 2008-2014 Oracle Corporation
All rights reserved.
(...)

URLs

https://HMC:443/preloginmonitor/index.jsp

https://HMC:443/hmc/connect

https://HMC:443/hmc/connects/ <-- this seems to be the correct URL

https://HMC:443/hmc/connects/mainuiFrameset.jsp

https://HMC:443/dashboard/#user/log-on?resources/systems

https://HMC:443/hmc/taskcontroller?reason=connectionClosed

https://HMC:443/license/en_US/HSC_License.html

https://HMC:12443/rest/ui/static/RedirectCCFWLogon

Logs

Install

/tmp/HmcInstall.log
/tmp/HmcBaseInstall.log

Apache at :443

[root] # ls -ltr /var/log/httpd/
total 3092
-rw-r--r-- 1 root root       0 Jan 31 13:34 access_log
-rw-r--r-- 1 root root 1662681 Feb  1 21:06 ssl_request_log
-rw-r--r-- 1 root root 1415365 Feb  1 21:06 ssl_access_log
-rw-r--r-- 1 root root   65507 Feb  1 21:55 ssl_error_log
-rw-r--r-- 1 root root    5027 Feb  1 21:55 error_log

New UI WebSphere

/opt/pmc/log/wlp -> /var/hsc/log/wlp

-- note how they are mistyped as "PCM" !!

(...)
-rw-r--r-- 1 wlp  wlp    75146 Feb  1 12:28 messages_15.02.01_13.22.47.0.log
-rw-r--r-- 1 root root    8943 Feb  1 13:23 wlpd.log
-rw-r--r-- 1 wlp  wlp    11848 Feb  1 13:31 PCMFFDC.log
-rw-r--r-- 1 wlp  wlp    28000 Feb  1 13:35 console.log
-rw-r--r-- 1 wlp  wlp    70934 Feb  1 13:35 messages.log
-rw-r--r-- 1 wlp  wlp  3452648 Feb  1 13:37 FFDC.log
-rw-r--r-- 1 wlp  wlp  2974993 Feb  1 13:38 Security.log
-rw-r--r-- 1 wlp  wlp    45706 Feb  1 13:38 Audit.log

Possible GUI socket file

/tmp/hmc/nativeProcessChildrenListener.(pid)

Possible problems

LIC errors

To be examined...

E212E115 Licensed Internal Code failure on the Hardware Management Console (HMC).
Memory Alert; VSIZE value for a process is too big Error reason = VSIZE size .
E3551231 Licensed Internal Code failure on the Hardware Management Console (HMC).
The critical hardware event detection code is not operating.