With the Core-SDI Advisory on the SSH1 CRC-32 compensation attack detector vulnerability and the associated exploit running around in Fall 2001, many people are rushing to upgrade their vulnerable secure shell servers. The exploit appears to be easy to use and quite effective, rooting boxes right and left.
We were among those rushing around upgrading systems, and it was quite a chore. So we've written down some rough instructions that report how we did it on our own systems. These are not intended to be tutorial, but they might give somebody a start. These instructions only apply to UNIX systems, and in particular we have installed on Red Hat Linux 6.2 and Solaris 2.7.
And we wish to note that this tip was created years ago, and much has changed since then even though the fundamentals should be familiar. Always check your local documentation to see what's different in the version you're on now.
There are three separate packages that have to be installed, and each is found in a different place throughout the World Wide Web.
zlib
This is a great file compression library, and from the official home page the mirrors can be located. We used version 1.1.4 in tarball zlib-1.1.4.tar.gz. Note that there is a substantial security issue in version 1.1.3 - upgrade and rebuild your server if you've got the older one.
OpenSSL
This is a secure socket library, and it is easy to find at the OpenSSL source code specifically do not need the "engine" variant, which uses hardware that does the heavy-duty crypto. We've noted that this web site is often unavailable.
OpenSSH
This is the SSH client & server itself, and it's really easy to get the wrong version. We found ours atftp://ftp3.usa.openbsd.org/pub/OpenBSD/OpenSSH/portable/, and it's important to always look in the portable/ subdirectory - otherwise you'll get a BSD only version. Our tarball was openssh-3.0p1.tar.gz.
We typically do our work in a /home/source area, so the unpacking will look like this:
Table of Contents
# cd /home/source
# gtar -xzvf zlib-1.1.4.tar.gz
# gtar -xzvf openssl-0.9.7a.tar.gz
# gtar -xzvf openssh-3.5p1.tar.gz
The three packages have to be built in order, and we do them each in turn:
# cd zlib-1.1.4
# ./configure
# make
# make install
# cd ../openssl-XXXXX
# ./config Note: not './Configure'
# make
# make test
# make install
# cd ../openssh-XXXXX
# ./configure --with-ipv4-default --with-md5-passwords
# make
# no "make install" yet!
We chose the arguments to ./configure by feeling around, and we think this is the right set for our environment. These instructions have been working for us on a number of systems.
--with-ipv4-default appears to tell SSH to use only IPv4 -- "regular" internet addresses -- instead of the next-generation IPv6 addresses. We're not sure if this is required or not.
--with-md5-passwords enables the better password hashing as found in Linux, and it probably isn't required on other systems or if you only use RSA authentication.
We suspect that --with-pam might also be useful, but so far the system is working fine for us without it.
NOTE - for recent releases of OpenSSH, this is where the sshd user and group should be added to support privilege separation. See the end of this document for the details.
If this is a first-time installation of the Secure Shell on this system, it's safe to just do a make install in the OpsnSSH area and it will drop its files in the proper places.
But if there is a working SSH installation then it's wise to go find the key setup files and save them off to the side first. This installation also parks files in different target directories than the older one we used, so it's wise to poke around and save or delete the old stuff. We typically look in these directories:
/etc
/usr/local/bin
/usr/local/sbin
Pay particular note to the ssh_config and sshd_config files, which define important server and client options. Save these files to the side.
It's also wise to keep an alternate method of accessing the system, such as the console or telnet, until you're really sure that you have gotten SSH installed properly. It's a drag to find (for instance) that sshd doesn't start properly on a reboot when you have no keyboard or monitor attached to the server under the desk. Been there, done that.
Once these precautions have been taken, do a make install in the SSH source directory, and it will put everything where it belongs.
Once SSH is installed, there are still a few configuration steps required. We're really not terribly strong with this, and in any case cannot provide a tutorial for SSH here. But we will note a few things that ought to be attended to after upgrading secure shell.
Insure that sshd gets launched properly at system startup. In our installation, the daemon is located in /usr/local/sbin instead of/usr/local/bin, and the /etc/rc.d/init.d/sshd script has to be modified to reflect the proper location.
The file we use lives in /etc/rc.d/init.d/sshd, and it contains:
#
# secure shell daemon
#
# chkconfig: 2345 90 05
# description: secure shell daemon
#
SSHD=/usr/local/sbin/sshd
[ -x $SSHD ] || exit 0
case "$1" in
start) echo "Starting secure shell daemon"
eval $SSHD
;;
esac
Once this file is in place (in /etc/rc.d/init.d/sshd), it has to be linked into the proper positions in the other runlevels so it actually: gets called. This is most easily done with the chkconfig program that manages the startup and shutdown locations in the various run levels.
# cd /etc/rc.d/init.d
# chmod +x sshd
# chkconfig --add sshd
If you prefer to do it yourself,
# cd /etc/rc.d
# chmod +x init.d/sshd
# ln -s ../init.d/sshd rc2.d/S90sshd
# ln -s ../init.d/sshd rc3.d/S90sshd
# ln -s ../init.d/sshd rc5.d/S90sshd
Important - do not forget the chmod step: the Secure Shell daemon won't start unless the init.d/sshd file is executable, and we've been burned by this more than once. It really stinks to be unable to get into your system remotely after a reboot because you forgot to chmod a file.
Make sure the sshd_config file is set correctly. In our installation we found that the file was installed to /usr/local/etc instead of/etc or /etc/ssh/, and that none of our customizations in the old version migrated to the new. In particular, we're careful to turn offPasswordAuthentication, though we are not sure this is a great idea if this is the first time you're using Secure Shell.
But in any case, be sure to disable the SSH version 1 protocol by setting Protocol 2 in the file. SSH1 is not secure and can be the source of all kinds of shenanigans.
We are not excited about allowing PasswordAuthentication for our secure shell daemons, because this provides limited system protection. It's true that it does forestall network sniffing -- and this is no small matter -- but it does not stop the bad guy from running a password guesser over the secure channel. In our work as penetration testers, we've gotten into systems via secure shell this way: by guessing or finding a password.
Much better is to use Public Key encryption, which has the potential for substantially better security as well as more convenience. We've been surprised how much we like public key support in SSH - much more than we expected.
In PK, each user creates a pair of keys: one public and one private, and the two are mathematically related. Both are very long strings of binary numbers, and they bear no relation to anything like a "password" or anything else human readable. The public key can be given away, but the private key is so important that it is further protected by a "pass phrase". This pass phrase actually encrypts the private key, and you'll need to type this key every time you need to use it.
The pass phrase should be something easy to type but impossible to guess: you'll be typing it a lot. At first it seems annoying to type this same phrase over and over, but quickly found that typing one long phrase for everything positively beats the pants off having to remember many individual passwords for all the sites we visit. This has turned out to be a big win.
Public key support revolves around the .ssh/ subdirectory in each user's UNIX/Linux home directory, and a couple of important files live here. These include (but not an exhaustive list):
authorized_keys - list of public keys (incoming)
identity.pub - SSH1 private key (outgoing)
identity - SSH1 private key (outgoing)
id_rsa.pub - SSH2 RSA public key (outgoing)
id_rsa - SSH2 RSA private key (outgoing)
id_dsa.pub - SSH2 DSA public key (outgoing)
id_dsa - SSH2 DSA private key (outgoing)
Formerly, authorized_keys held SSH1 keys while the transitional file authorized_keys2 held the version 2 keys, and for a time it was common to have both. But nobody should be using version-1 keys anymore, and though modern SSH servers will read either file, one only really need to useauthorized_keys.
IMPORTANT - Nothing will work right unless the ownership and permissions are set correctly. The directory and all files must be owned by the user in question, and must be unreadable by anybody but the user. The ssh server will simply ignore files that don't meet this stringent rule - it's a good security precaution. Just remember to chmod og= ~/.ssh ~/.ssh/* every now and then to be sure.
The ssh-keygen program is used to create a public key pair suitable for OpenSSH client and server use (there are other mechanisms for creating keys, we'll touch on the m later). It's possible to create both "user" and "host" keys, but since the host keys are usually created during the software installation process, we'll only touch on the user key.
There are many options for creating public key pairs, but we only care about making SSH version 2 RSA keys. This command creates the key pair, prompts for a pass phrase to encrypt the private key, and stores the two files in your ~/.ssh directory:
$ ssh-keygen -b 1024 -t rsa
-b 1024 specifies the number of bits to encrypt the key, and the consensus is that 1024 is fine: larger values just slow things down without adding appreciably to security. -t rsa specifies to use RSA encryption to produce an SSH version 2 key (the default is to create a version one key - bad news). It prompts for a filename to store the keys in (usually OK to keep the default), then asks for a pass phrase twice.
Choose a good pass phrase! Your private key is not very private if the very secret id_rsa file can be compromised.
Running the command should look like this:
$ ssh-keygen -b 1024 -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/friedl/.ssh/id_rsa): RETURN
Enter passphrase (empty for no passphrase): passphrase + RETURN
Enter same passphrase again: passphrase + RETURN
Your identification has been saved in /home/steve/.ssh/id_rsa.
Your public key has been saved in /home/steve/.ssh/id_rsa.pub.
The key fingerprint is:
67:2c:06:ff:5a:cf:36:64:11:20:74:90:9f:97:14:8e steve@unixwiz.net
Now the files ~/.ssh/id_rsa.pub and ~/.ssh/id_rsa contain your public and private keys, respectively. We'll use them shortly. The public key file is one line of ASCII, and it looks like this (though it's actually all one line):
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAx9aYnr7XwNw0Ei3X2ngxRjGg
kQpK/EqpkveGCvvbMoH84zqu3Us8jSC0D392JZAEohGSoe0dWMBFm9Y41VGZ
YmncwkfTQPFH1P0IvDw49aTAa2RJNFyVQANZCbSocDeTT0QpusuUj/v8hj7+
PqsUUpUXVQSDIhXBkWV+bJawc1c= steve@unixwiz.net
Again, there is no need to keep a public key secret.
When the ssh server is considering an incoming connection request via the public key mechanisms, it consults the ~/.ssh/authorized_keysfor version-2 keys (we're specifically ignoring version-1 key support).
To enable a remote user to the local OpenSSH server, simply edit the user's ~/.ssh/authorized_keys file and append the user's public key -- one line -- to the file and save it. Do make sure that the file is owned by the user and is not readable by anybody else, but there is no need to restart the OpenSSH server to make it take effect.
Then, from the remote system, connect with the ssh command and the name of the remote host.
$ ssh www.example.com
The authenticity of host 'www.example.com (10.1.1.1)' can't be established.
RSA key fingerprint is 50:29:ac:84:60:42:31:f7:e0:4c:bc:d2:ff:de:ba:89.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'www.example.com,10.1.1.1' (RSA) to the list of known hosts.
Enter passphrase for key '/home/steve/.ssh/id_rsa': passphrase + RETURN
$ uname -a
Linux www.example.com 2.2.12-20 #1 Mon Sep 27 10:25:54 EDT 1999 i586 unknown
$
The first time you contact any host, you'll be warned that your local SSH installation doesn't know about the remote. When you confirm that you want to connect to it with yes, the name and fingerprint of the remote host is added to the ~/.ssh/known_hosts file. Then you won't be prompted again for this host.
After confirming the host, you'll enter your passphrase to decrypt your private key. If done correctly, the SSH client and server exchange your key information, and if all is well you're granted access to the remote system.
There is no reason that a single public key can't be put in multiple user accounts on the remote system: it's possible to store your public key in both /home/steve/.ssh and /root/.ssh directories. So logged in as a regular user, we can login as root on the remote system:
$ ssh -l root www.example.com
We use the fantastic telnet/serial/SSH client SecureCRT (from VanDyke Software) to connect from our Windows (ME/NT/2000) boxes to OpenSSH, and setup is straightforward.
After SecureCRT is installed on the Windows box, a pair of public/private keys must be created. From the menu, select Options:Global Options, and from this dialog box select SSH2, then click Create Identity File. Key creation is done just one time after installation. NOTE: these instructions are for SecureCRT v4.x.
Click Next in the Key Generation Wizard, then select an RSA - not DSA - key and click Next. In the next dialog box, enter your pass phrase (twice), then a comment that appears in the file. By convention this is your email address, but it can be any descriptive text that helps you identify the key. Then click Next.
In the next dialog box, select 1024 bits for the key and click Next. Then you are invited to move the mouse around: this provides important random inputs that help generate a key pair that will be hard to crack. This takes perhaps 30 seconds, and when finished you are asked to clickNext.
In the final dialog box, you're asked to save your key files somewhere. It's not as important exactly where they go as long as you remember where that place is. You will need the public key file shortly. For our example we are using C:\identity.pub.
After the key has been generated, locate the public key file and get it over to the UNIX system. Since this file is small and ASCII, it's often easiest to open it in notepad, select the whole thing, and "cut" it into the Windows notepad. Then, logged into the remote server (perhaps with telnet or via ssh using a password), and pasting it into a file. For instance:
$ cat > /tmp/identity.pub
"paste" the file here
^D <-- control-D
$
Remember that the entry is one line, so you might need to use your favorite editor and join them together if pasting created multiple lines.
Once the file is over on the UNIX system, it has to be converted from the SecureCRT format (some kind of standard) to that used by OpenSSH. The ssh-keygen command is able to do this conversion for you:
$ ssh-keygen -i -f /tmp/identity.pub >> ~/.ssh/authorized_keys
Now your SecureCRT public key is installed, so you should be able to ssh into your UNIX box from Windows.
Once the public key mechanisms are known to be working correctly, including testing of proper restart after reboot, we recommend some additional steps.
Disable telnetd
Even with TCP wrappers limiting the IP addresses of incoming connection, we believe that telnet is just too dangerous to leave running. So we disable telnetd in the /etc/inetd.conf file by commenting out this line
and "kicking" inetd by sending a hangup signal to it:
#telnet stream tcp nowait root /usr/sbin/tcpd in.telnetd
# ps -fCinetd
UID PID PPID C STIME TTY TIME CMD
root 413 1 0 Feb01 ? 00:00:00 [inetd]
# kill -1 413
We believe that newer systems use the xinetd.conf file, but we don't have this on any of our systems: the idea ought to be similar.
Now telnet won't accept connections any longer. The brave and determined might actually remove the telnet daemon altogether. We've done this on most of our systems.
Disable PasswordAuthentication option
Magical Mystery Auth
Disable ChallengeResponseAuthentication option
Unless you use S/Key authentication (we never have), disable this — it's been the source of security troubles in the past.
Disable UsePAM option
PAM is the Pluggable Authentication Module mechanism on Linux, and it should be disabled if you wish to use key-only logins.
We're quite sure that other options in the config file are important, but we've not really looked at them in much detail. Shame on us.
Once OpenSSH has been installed on a platform, it's reasonable to want to install it elsewhere without having to actually build it. So we've noted the files that are installed by the make install process so that they can be copied (with permissions!) to some other system. This listing is relative to /usr/local.
... in the OpenSSH configuration file (often /usr/local/etc/sshd_config), find the setting for PasswordAuthentication and change the value to no. This then permits only public key authentication and prevents "regular" passwords from working. We feel strongly that allowing people to guess passwords is a really bad idea, and by insisting on RSA keys, a whole raft of shenanigans can be avoided.
The interaction and behavior of thePasswordAuthentication,ChallengeResponseAuthentication, andUsePAM options is tricky, and has changed substantially over times. We're very, very fuzzy on this other than to know we have to turn themall off to reliably disable the use of passwords. Check the documentation for your particular version to be sure.
$ gtar -tzvf OpenSSH-3.1p1z.tgz
-rws--x--x root/root 211396 2002-03-13 01:32:57 bin/ssh
-rwxr-xr-x root/root 26520 2002-03-13 01:32:57 bin/scp
-rwxr-xr-x root/root 75564 2002-03-13 01:32:57 bin/ssh-add
-rwxr-xr-x root/root 42428 2002-03-13 01:32:57 bin/ssh-agent
-rwxr-xr-x root/root 79640 2002-03-13 01:32:57 bin/ssh-keygen
-rwxr-xr-x root/root 136784 2002-03-13 01:32:58 bin/ssh-keyscan
-rwxr-xr-x root/root 46216 2002-03-13 01:32:58 bin/sftp
lrwxrwxrwx root/root 0 2002-03-13 01:32:59 bin/slogin -> ssh
-rwxr-xr-x root/root 230416 2002-03-13 01:32:58 sbin/sshd
-rw-r--r-- root/root 182 2002-03-13 02:31:06 sbin/generate-ssh-keys
-rw-r--r-- root/root 46525 2002-03-13 01:32:58 man/man1/ssh.1
-rw-r--r-- root/root 3261 2002-03-13 01:32:59 man/man1/scp.1
-rw-r--r-- root/root 4896 2002-03-13 01:32:59 man/man1/ssh-add.1
-rw-r--r-- root/root 6315 2002-03-13 01:32:59 man/man1/ssh-agent.1
-rw-r--r-- root/root 10081 2002-03-13 01:32:59 man/man1/ssh-keygen.1
-rw-r--r-- root/root 3890 2002-03-13 01:32:59 man/man1/ssh-keyscan.1
-rw-r--r-- root/root 7129 2002-03-13 01:32:59 man/man1/sftp.1
lrwxrwxrwx root/root 0 2002-03-13 01:32:59 man/man1/slogin.1 -> ssh.1
-rw-r--r-- root/root 43258 2002-03-13 01:32:59 man/man8/sshd.8
-rw-r--r-- root/root 2087 2002-03-13 01:32:59 man/man8/sftp-server.8
-rwxr-xr-x root/root 23712 2002-03-13 01:32:58 libexec/sftp-server
-rw-r--r-- root/root 568 2002-03-13 01:32:56 share/Ssh.bin
-rw-r--r-- root/root 88039 2002-03-13 01:32:59 etc/moduli
-rw-r--r-- root/root 1144 2002-03-13 01:32:59 etc/ssh_config
-rw-r--r-- root/root 2523 2002-03-13 01:32:59 etc/sshd_config
The only nonstandard file is sbin/generate-ssh-keys, which is a small shell script we have created: it simply creates the three key pairs that are required. Run this program one time after installation to create these keys.
The SSH daemon requires its own set of host keys, and these are created as part of the installation process. But when the binaries are copied to another machine, the keys must be generated manually. As with all keys, these are created with the ssh-keygen program, and the three sets of keys are created with:
# ssh-keygen -t rsa1 -f /usr/local/etc/ssh_host_key -N ""
# ssh-keygen -t dsa -f /usr/local/etc/ssh_host_dsa_key -N ""
# ssh-keygen -t rsa -f /usr/local/etc/ssh_host_rsa_key -N ""
This entire Tech Tip is very lame in comparison with the excellent O'Reilly book SSH, The Secure Shell: The Definitive Guide (2e), which covers everything in much more detail. Once you've gotten a handle on the basic configuration of OpenSSH, we recommend picking up this book to get to the finer points. We'll update our document as we pick them up ourselves.
There has been a problem reported in the ZLIB compression library that has prompted version 1.1.4. We've installed this and found it easy to do.
Fetch the updated ZLIB tarball: zlib-1.1.4.tar.gz from the ZLIB home page.
Unpack, build, and install the source:
# cd /home/source
# gtar -xzvf zlib-1.1.4.tar.gz
# cd zlib-1.1.4
# ./configure
# make install
Edit the OpenSSH version file. In the OpenSSH directory is a file version.h, and we want to remind ourselves that we've made this fix even though we haven't actually patched OpenSSH directly. So we tack a z to the end:
#define SSH_VERSION "OpenSSH_3.0p1z"
Rebuild OpenSSH. We first do a "make clean" to force the changes to take effect:
# vi version.h
# make clean
# make
The resulting sshd file is left in the current directory.
Install sshd. Doing make install updates everything, and we're not sure this is really necessary. So we just manually copy the file to the final place. Then we look for the "parent" SSH daemon (the one with the PPID of 1) and send it the hangup signal. Then restart the daemon.
# strip sshd
# cp sshd /usr/local/sbin
# ps -fCnamed
UID PID PPID C STIME TTY TIME CMD
root 4509 1 0 07:46 ? 00:00:00 /usr/local/sbin/sshd
root 4510 4509 0 07:46 ? 00:00:00 /usr/local/sbin/sshd
root 4534 4509 0 07:49 ? 00:00:00 /usr/local/sbin/sshd
# kill -1 4509
# /usr/local/sbin/sshd
We've managed to build OpenSSH on several other platforms with more or less the same instructions for Linux, but a few notes might be helpful here.
SCO UNIX 5.0.6 with SCO cc
This took a while to get around some build issues, but we ultimately got it working just fine.
zlib-1.1.4 built and installed without incident
openssl-0.9.7 required that we edit the Configure file and find the line for "sco-cc". Remove -lresolv from this line - we didn't find this file on our system, and our attempts to build BIND9 on this platform failed due to the lack of a "long long" integer type. Then build, test, and install per instructions.
openssh-3.5p1 fails the "configure" process because the default cc output is COFF, not ELF. So the command to set this up is./configure --with-ipv4-default CC="cc -belf". Then build and install per the usual instructions.
SCO UNIX 5.0.5 with GNU cc
This built and installed with the above instructions excepting configuring OpenSSH itself (SCO doesn't use MD5 shadow passwords).
# cd openssh-3.2.3p1 (with gcc)
# ./configure --with-ipv4-default
We installed the startup file in /etc/rc2.d/S99sshd. This information built as of:
zlib-1.1.4
openssl-0.9.6d
openssh-3.2.3p1
The sshd privsep user directory must be created manually with all the right permissions lest it complain about the modes.
# mkdir -m 700 /var/empty
# chown sshd /var/empty
# chgrp sshd /var/empty
Sun SOLARIS 2.7
This likewise built with the same procedure with the exception of the the OpenSSH configuration itself:
# cd openssh-3.2.2p1
# ./configure --with-ipv4-default --with-pam
Built with:
zlib-1.1.4
openssl-0.9.6d
openssh-3.2.2p1
Starting with OpenSSH 3.3.2, the privilege separation feature was introduced. This is a security provision that goes a long way to forestall full system compromise even if a vulnerability is found. The details are covered elsewhere, and our favorite is here.
But before SSHD is installed, both a sshd user and group must be added, and it looks in the password and group files like this:
/etc/group sshd:*:27:
/etc/passwd sshd:*:27:27:sshd privsep:/var/empty:/sbin/nologin
Under Red Hat Linux 7.2, these commands do the trick:
# groupadd sshd
# useradd -M -g sshd -c 'sshd privsep' -d /var/empty -s /sbin/nologin sshd
# passwd -l sshd
The -M parameter suppresses creation of the home directory, as this is done by the "make install" procedure for OpenSSH.
Note further that the UsePrivilegeSeparation yes option must be set in the sshd_config file - this may be the default for new installations, but upgrades don't usually touch current values. Make sure this option is set.