Software Updates

Software Update Guide

Overall software update guidelines for our cluster:

NOTE: to avoid kernel panic remove oldest kernels from /boot. Leave at least two kernels in case you have to fall back to a previous kernel after reboot.

• Please be aware of all the following directions in this section before proceeding

• For the key service nodes only update known security problems, and never reboot, just restart services:

  • Whenever possible don't do a kernel update because that requires a reboot, and we want to NEVER reboot these!

hepcms-hn.umd.edu

hepcms-ovirt.umd.edu

hepcms-foreman.umd.edu

• Any nodes not in base hiera group will need to have automatic SL6 software updates turned off

• Always do the following before applying any updates:

yum install yum-utils yum-plugin-ps

yum check-update

needs-restarting

• Be sure to keep a log of the update, occasionally a single part of the update will fail and you'll have to check what broke.

• General yum guidelines (a few more tricks on our cheat sheet also):

  • https://twiki.opensciencegrid.org/bin/view/Documentation/Release3/YumRpmBasics

• After update, do needs-restarting again to find out what services need restarting.

• Here is an example restarting commands from Trey (copy/paste into command line as root), these "Should eliminate all but the udev-d and tty processes from the list":

# Restart services - so far these are ones I've found

for s in nfslock messagebus haldaemon munge ntpd postfix sshd sssd zabbix-agent mcollective ovirt-guest-agent httpd tomcat6 atd crond ; do

if test -f /etc/init.d/${s} ; then

/etc/init.d/${s} status &>/dev/null

if [ $? -eq 0 ]; then

/etc/init.d/${s} restart

/etc/init.d/${s} status

fi

fi

done

# Dell tools too on Dell systems

if test -f /opt/dell/srvadmin/sbin/srvadmin-services.sh ; then

/opt/dell/srvadmin/sbin/srvadmin-services.sh restart

fi

• OSG notes:

• In general, you don't want to upgrade OSG, condor, or hadoop without doing this for ALL nodes

  • For OSG node updates of OSG software, don't reboot, just restart services, including OSG services (in the proper order: http://hep-t3.physics.umd.edu/HowToForAdmins/errors.html#errorsOSG needs rewritten)

• Make sure to have a saved version of condor configuration before upgrade

  • After upgrading condor, restart condor

• Before upgrading hadoop, put hadoop in safe mode on hepcms-namenode

  • Copy the current NameNode index to /data as well as your laptop

  • Do the upgrade, then restart services as needed (on hepcms-namenode, hepcms-secondary-namenode)

  • Make sure the full hadoop disk is still there and once the FULL upgrade procedure for the outage is done, take hadoop out of safemode

• Worker Nodes are also hadoop datanodes, so try not to reboot them, just restart services

• Interactive nodes should be restarted (planned outage to announce to users)

  • If an unplanned outage is needed, use the command w to see who is logged in

  • Also check ps ahux to see if anyone left a screen, tmux, or other long-running process (they're not supposed to leave long-running processes on interactive)

  • I have often done reboots during weeknights when nobody's logged in, or a weekend, but it always runs the risk of a bare metal IN not coming back online (see next note)

  • Note that this is nice to clear abandoned ssh logins, badly cancelled processes, and swap memory

• Warning about bare metal machines (worker nodes, hepcms-in2, main cluster service nodes)

  • Occasionally they won't come back up after reboot, either they hang on some command (or disk mount) or need input at the machine to fsck a disk (usually a data disk) by hand that failed disk check on reboot

  • Therefore you want to have at least one person on campus to intervene in the case of a reboot of bare metal machines

node cpus disk TB condor slots hadoop usage osg-version singularity

comp 5 8 3.5 7 4 3.4 yes comp five has proority slots so the number reads 14 but actual slots are 7

10 8 3.5 7 3.4 yes

7 8 3.5 7 3.4 yes

11 8 3.5 7 3.4 yes apparently never upfdtaed using dougs instrcution and was updated earlier with --skip broken

updated singularity to 2.6.1 but didnt work. just re,oved the /usr/libexec/singularity/bin/start-suid for now.

8 8 3.5 7 3.3

6 8 3.5 7 3.4 yes rm /usr/libexec/singularity/bin/start-suid instead

r510-0-9 24 22 nothing 3.4 yes

11 24 22 nothing 3.4 yes /dev/sda1 is 100% so cant install

rm /usr/libexec/singularity/bin/start-suid instead

6 24 22 23 3.4 yes rm /usr/libexec/singularity/bin/start-suid instead

1 24 22 23 3.4 yes

4 24 22 23 3.4 yes

10 24 22 23 3.4 yes

5 24 22 23 3.4 yes rm /usr/libexec/singularity/bin/start-suid

r720-0-1 32 22

3.4

2 32 22 31 3.4 yes

total 280 218

adding singularity to worker nodes compute-0-10,

r510-0-11, 6, 04,

ISSUES:

fixed some issues by changing

https://gitlab.cern.ch/SITECONF/T3_US_UMD/blob/master/JobConfig/site-local-config.xml

changes all instances of /sharesoft/cmssw to /cvmfs/cms.cern.ch

Genral update to osg 3.4 Sep 2018

remove emove the osg-wn-client-glexec package as we no longer need it.

yum remove osg-wn-client-glexec

and dont update the ones causing trouble.

yum y update --disablerepo=whatever is making the update fail

you might even have to diable these when getting one or two pacjkges

yum -y install --disablerepo=dell-omsa-indep,dell-omsa-specific https://repo.opensciencegrid.org/osg/3.4/osg-3.4-el6-release-latest.rpm

using Doug's instructions:

cd /etc/yum.repos.d/

mkdir SaveOSGRepos

mv osg* SaveOSGRepos/

yum clean all

yum -y install https://repo.opensciencegrid.org/osg/3.4/osg-3.4-el6-release-latest.rpm

yum clean all

yum -y install lcmaps vo-client-lcmaps-voms osg-configure-misc llrun voms-clients

yum -y update

osg-configure -c

ln -s /data/osg/scripts/grid-mapfile /etc/grid-security/

sometimes rpm would fail with an error that "will not update and nothing to do". A reinstall of rpm helped.

These are exact instructions that worked on r510-0-9

cd /etc/yum.repos.d/

mkdir SaveOSGRepos

mv osg* SaveOSGRepos/

yum clean all

yum -y reinstall --disablerepo=dell-omsa-indep,dell-omsa-specific https://repo.opensciencegrid.org/osg/3.4/osg-3.4-el6-release-latest.rpm

yum clean all

yum -y install lcmaps vo-client-lcmaps-voms osg-configure-misc llrun voms-clients

yum -y update --disablerepo=dell-omsa-indep,dell-omsa-specific

osg-version

osg-configure -c

ln -s /data/osg/scripts/grid-mapfile /etc/grid-security/

yum clean all

yum -y install singularity-runtime

emacs -nw /etc/singularity/singularity.conf

March 2019 servicenodes upgardes to OSG3.4

gridftp already has the osg 3.4

[root@hepcms-gridftp boot]# osg-version

OSG 3.4.17

[root@hepcms-gridftp boot]#

dec 12, 2018 update.

Critical vulnerability found update to singu;arity-runtime-2.6.1

yum -y install singularity-runtime

Singularity install

from https://opensciencegrid.org/docs/worker-node/install-singularity/

yum -y install https://repo.opensciencegrid.org/osg/3.4/osg-3.4-el6-release-latest.rpm

yum clean all

yum -y install singularity-runtime

to update

yum update singularity-runtime

[root@compute-0-10 ~]# vi or emacs -nw /etc/singularity/singularity.conf

configure

ENABLE UNDERLAY: [yes/no]

# DEFAULT: no

# Enabling this option will make it possible to specify bind paths to locations

# that do not currently exist within the container, similar to the overlay

# option. This will only be used if overlay is not enabled.

enable underlay = yes

# ENABLE OVERLAY: [yes/no/try]

# DEFAULT: try

# Enabling this option will make it possible to specify bind paths to locations

# that do not currently exist within the container. If 'try' is chosen,

# overlayfs will be tried but if it is unavailable it will be silently ignored.

enable overlay = no

MAX LOOP DEVICES: [INT]

# DEFAULT: 256

# Set the maximum number of loop devices that Singularity should ever attempt

# to utilize.

max loop devices = 0

validate

[jabeen@compute-0-10 ~]$

bash

singularity exec --contain --ipc --pid --home $PWD:/srv --bind /cvmfs /cvmfs/singularity.opensciencegrid.org/opensciencegrid/osgvo:el6 ps -ef

singularity exec --contain --ipc --pid \

> --home $PWD:/srv \

> --bind /cvmfs \

> /cvmfs/singularity.opensciencegrid.org/opensciencegrid/osgvo:el6 \

> ps -ef

WARNING: Container does not have an exec helper script, calling 'ps' directly

UID PID PPID C STIME TTY TIME CMD

jabeen 1 0 1 00:25 ? 00:00:00 shim-init ps -ef

jabeen 2 1 0 00:25 ? 00:00:00 ps -ef

[jabeen@compute-0-10 ~]$

Warning

If you modify /etc/singularity/singularity.conf, be careful with your upgrade procedures. RPM will not automatically merge your changes with new upstream configuration keys, which may cause a broken install or inadvertently change the site configuration. Singularity changes its default configuration file more frequently than typical OSG software.

Look for singularity.conf.rpmnew after upgrades and merge in any changes to the defaults.

GRID services nodes OSG software update: Feb 2018

hepcms-gridftp

yum update --disablerepo=puppetlabs-products

root@hepcms-gridftp ~]# service globus-gridftp-server restart

hepcms-ce

yum update --disablerepo=puppetlabs-products

root@hepcms-1 ~]# service httpd restart

Stopping httpd: [ OK ]

Starting httpd: [ OK ]

[root@hepcms-1 ~]# service condor-cron restart

Stopping Condor-cron daemons: [ OK ]

Starting Condor-cron daemons: [ OK ]

[root@hepcms-1 ~]# service rsv restart

Stopping RSV: Stopping all metrics on all hosts.

Stopping consumers.

Starting RSV: Starting 13 metrics for host 'hepcms-1.umd.edu'.

Starting 2 metrics for host 'hepcms-0.umd.edu:8443'.

Starting 1 metrics for host 'hepcms-gridftp.umd.edu'.

Starting 2 consumers.

hepcms-se

yum update --disablerepo=puppetlabs-products

[root@hepcms-0 ~]# service bestman2 restart

[root@hepcms-0 ~]# service xrootd restart

[root@hepcms-0 ~]# service cmsd restart

hepcms-gums

yum update --disablerepo=puppetlabs-products

[root@hepcmsdev-6 ~]# service tomcat6 restart

[root@hepcmsdev-6 ~]# service mysqld restart

hepcms-squid

yum update --disablerepo=puppetlabs-products

[root@hepcms-squid ~]# service frontier-squid restart

Full OSg update Oct 4

Yum update

or Yum update --disablerepo=puppetlabs-products

if the puppetlab repo was in the list of updates