Troubleshooting

Screenshots

of different hardware and software work in building the cluster as well as troubleshooting various issues 

Screenshots

Keep in mind

After the squid issue, as of Nov23, 2016,  taking R720-01 out of condor to keep the traffic low.

[root@r720-0-1 ~]# service condor status

condor_master (pid  9826) is running...

[root@r720-0-1 ~]# service condor stop

Stopping Condor daemons:                                   [  OK  ]

Account problems

1. Node lets you enter password and then kicks you off without prompt

2. Node doesn't let you enter password, just ssh closed - Denyhosts block

3. Can SSH from Lxplus but not directly to T3

4. public_html web dir not visible

CMSSW

1. 1 CMSSW

2. 2 Version of CMSSW the user wants is not found

3. 3 CVMFS

4. 4  cmsrel and cmsenv "commands not found"

5. 5 ROOT

6. 6 FWLite

OSG

Software update

were asked to add the following software:

from hn

  ssh-agent $SHELL

   ssh-add

    clush -w  @R510 yum install -y tcl tk zsh perl-ExtUtils-Embed compat-libstdc++-33

   clush -w  @compute yum install -y tcl tk zsh perl-ExtUtils-Embed compat-libstdc++-33

   clush -w  @R720 yum install -y tcl tk zsh perl-ExtUtils-Embed compat-libstdc++-33

   clush -w  @interactive yum install -y tcl tk zsh perl-ExtUtils-Embed compat-libstdc++-33

   clush -w  @hepcms-in1 yum install -y tcl tk zsh perl-ExtUtils-Embed compat-libstdc++-33

   more /etc/clustershell/groups

   clush -w  @INT yum install -y tcl tk zsh perl-ExtUtils-Embed compat-libstdc++-33

-y is for automatic yes to options. Make sure to test the install on one node first to see if everything works.

Grid/CRAB/Condor

1. 1 GRID /CRAB

   1. 1.1 CRAB Unable to check write permission

2. 2 Condor Jobs taking too long to process

Puppet/Foreman

1. 1 Puppet/Foreman

2. 2 How to upgrade the puppet master

3. 3 Running Puppet on Foreman - DON'T!

4. 4 Do Not run Puppet on hepcms-hn (The head node)

   1. 4.1 Oh no you ran puppet where you shouldn't have and want to roll back your file

5. 5 Keep puppet at the same version for the client as master 

6. 6 Work on a test node for hiera or puppet code changes:

7. 7 Backup puppet (as root on hepcms-foreman) before changing things (note this is aliased as backup to put a tar in /data on that node)

8. 8 Check what the changes were after you try something in Puppet:

9. 9 Test out hiera:

   1. 9.1 Did you change hiera.yaml?

10. 10 Test puppet changes without implementing them on a node:

    1. 10.1 Test puppet changes without implementing them on a node for just one feature (tags):

    2. 10.2 Implement puppet changes on a node:

    3. 10.3 Stop a puppet agent (these run automatically on a node either in kickstart or crontab):

    4. 10.4 Make that puppet agent not start automatically upon node reboot):

    5. 10.5 Start a puppet agent (these run automatically on a node either in kickstart or crontab):

    6. 10.6 Changes in Puppet not taking effect? 

    7. 10.7 Conflict error message in puppet?

    8. 10.8 puppet agent --test --noop reports failed dependencies?

    9. 10.9 Want to add a puppet class in base.pp or site.pp instead of on hepcms-foreman web?

    10. 10.10 Want to add a puppet class in a hiera yaml instead of on hepcms-foreman web?

    11. 10.11 Changed hostname on a machine, need to re-get puppet cert:

    12. 10.12 Puppet certificate error/made a new cert:

    13. 10.13 r10k make sure we don't lose changes updated locally and not in github:

    14. 10.14 Backup Foreman:

    15. 10.15 Add a puppet class to a host group in Foreman

    16. 10.16 Foreman kickstart telling you there's not enough disk space for partitions?

    17. 10.17 Kickstart on PXE boot failure:

    18. 10.18 Error downloading the kickstart file:

    19. 10.19 Kernel panic during kickstart:

    20. 10.20 Is the Foreman build of a baremetal machine working (checking during build):

    21. 10.21 Note the time it takes to build at the bottom of this page can be very long ~20-30minutes or more depending on disks attached to the node

    22. 10.22 Get r10k updates:

    23. 10.23 My github update to Puppet/Hiera didn't take effect?

    24. 10.24 Problem with r10k?

        1. 10.24.1 Go to the area it's complaining about (on hepcms-foreman) and do a git status. Currently there is no ssh key on hepcms-foreman, we are using git as read only To continue to use git as read only, do the following: Commit and push any changes you have made by hand to git using another server (not ideal, best to add a ssh key and config your hepcms-foreman git as root) Go to the affected areas that git complains about, i.e., and in each area, do the following git commands: /etc/puppet/hiera/production/hieradata /etc/puppet/environments/production/modules/profile git fetch --all git reset --hard origin master       or      git reset --hard

    25. 10.25 Add a puppet module by hand in an area (locally) where r10k & git won't affect it:

        1. 10.25.1 On hepcms-foreman Put it in: /etc/puppet/modules, that is a directory defined in /etc/puppet/puppet.conf as a basemodulepath path which is a path picked up by Puppet for modules but it is not touched by r10k

    26. 10.26 Command line puppet modify some value:

    27. 10.27 Add a module to what r10k picks up from git?

    28. 10.28 Puppet module complaining about operating system issues?

    29. 10.29 Missing some repo in your install?

    30. 10.30 Ensure a specific package is used in hiera .yaml:

    31. 10.31 Puppet order of applying things:

    32. 10.32 Do proper ordering of install to ensure program (i.e. facter) comes from puppetlabs instead of epel:

    33. 10.33 Change in some .yaml parameter or class not taking effect at all on a node?

    34. 10.34 Check puppet agent behavior for a specific module (on that node):

    35. 10.35 All your nodes in the hepcms-foreman web page suddenly orange for "not in sync"?

    36. 10.36 How to change a puppet configuration file in your hiera .yaml?

    37. 10.37 How to use a puppet class in hiera .yaml?

    38. 10.38 Did your hiera implementation give you something weird, like ["?

    39. 10.39 Hiera yaml files:

    40. 10.40 Hiera variable usage:

    41. 10.41 Why is my node stuck in blue A and always doing the same update?

    42. 10.42 Implementing a new puppet module and get an error about "Could not find class"?

    43. 10.43 Foreman error in getting r10k?

    44. 10.44 Puppet create a directory:

    45. 10.45 Puppet symlink a file:

11. 11 Ovirt:

    1. 11.1 If possible, please never reboot ovirt

    2. 11.2 Ovirt web management super slow?

    3. 11.3 Ovirt network configuration reverted from the web one and you have to change it at Rivertech:

    4. 11.4 Configure network for hepcms-ovirt:

    5. 11.5 Bring up virtual machines that didn't come up upon reboot:

    6. 11.6 Rebooting virtual machines in Foreman:  

    7. 11.7 Power management alert on hepcms-ovirt web page:

    8. 11.8 Change FQDN of oVirt machine:

    9. 11.9 Ovirt machine doesn't have installed:

    10. 11.10 Make a snapshot

    11. 11.11 Roll back to an ovirt snapshot

12. 12 Adding more Physical Guaranteed Memory to a VM

OSG/UNIX

1. 1 OSG:

   1. 1.1 Backup GUMS database/files

   2. 1.2 OSG version & repository

   3. 1.3 User grid certificate error "Couldn't find valid credentials to generate a proxy."

2. 2 Unix:

   1. 2.1 history

   2. 2.2 yum update

   3. 2.3 X11 forwarding request failed on channel 0

   4. 2.4 Yum: how to find group packages available to install

   5. 2.5 Yum: what provides this package:

   6. 2.6 Yum what's available?

   7. 2.7 Do not yum remove package!

   8. 2.8 Yum/Puppet groupinstall:

   9. 2.9 Yum and puppet manage repositories:

   10. 2.10 Yum troubleshooting:

   11. 2.11 Yum error repo: Metadata file does not match checksum:

   12. 2.12 Yum error with a particular repo:

   13. 2.13 Yum error: not enough space

   14. 2.14 Yum versionlock:

   15. 2.15 ROOT password:

   16. 2.16 Sudoers has been implemented (by hand) on:

   17. 2.17 Trying to install new software doesn't work due to user account issues:

   18. 2.18 Command to create/execute file/directory x through z

   19. 2.19 Command not found (such as chkconfig or service):

   20. 3.21 Check syntax of kickstart file

NETWORK

1. 1 Basic troubleshooting:

2. 2 Internal network working and not external for a node?

3. 3 Slow network or connectivity?

4. 4 What machine am I logged into?

5. 5 Some ethernet port is not working even when I modify it in Foreman?

6. 6 Bring down one ethernet port:

7. 7 What ports are open? :

8. 8 NFS disk mount/cannot find server problem:

9. 9 ssh_exchange_identification: read: Connection reset by peer:

10. 10 Kickstarted a node but Foreman's having trouble seeing an (external) port:

11. 11 TCP Tuning

Hardware Issues

1. 1 Dell Disks/OMSA

   1. 1.1 Does Dell OMSA work?

   2. 1.2 Cannot get to Dell OMSA webserver 

   3. 1.3 https://servername.privnet:1311 from firefox on the local network (i.e. hepcmsdev-1):

   4. 1.4 What's the status of disks?

   5. 1.5 Need to know which disk number in Dell corresponds to which /hadoopX?

   6. 1.6 Assuming drive cache: write through message on login screen at cluster for a disk

2. 2 Hardware Issues

   1. 2.1 Dell Firmware update guide: 

   2. 2.2 Dell OMSA Cheat Sheet

   3. 2.3 Physical Node

3. 3 Hadoop

   1. 3.1 Raid Disk swap May 4, 2015 (this is all ancient Rocks directions and some might be useful for Foreman cluster but some might not)

   2. 3.2 How to handle a stale mount of a NFS disk:

   3. 3.3 Debug NFS disk mount:

   4. 3.4 NFS disk mount/cannot find server problem:

   5. 3.5 Additional NFS disk mount problem: requested NFS version or transport protocol is not supported

   6. 3.6 Problems with writing to an nfs mount?

   7. 3.7 Did you reboot things in the wrong order? NFS disks not mounted?

   8. 3.8 Want to edit/look at files as root on NFS systems:

   9. 3.9 Hadoop filesystem not mounting:

   10. 3.10 Debug and fix a hadoop data disk:

   11. 3.11 Partitioning a disk

   12. 3.12 removing hardware disk

4. 4 R720 panic mode

   1. 4.1 Running diagonastics

   2. 4.2 Decommissioning

   3. 4.3 Puppet Giving libsmbios error when trying to install srvadmin-all

5. 5 Physical Hardware

   1. 5.1 Consult this web page for more technical information about hardware identity

   2. 5.2 KVM

   3. 5.3 One or both KVMs not working?

   4. 5.4 Machine has Orange blinking light, or orange "electrical", or orange "hard drive" symbol

   5. 5.5 Machine isn't visible remotely from reboot

Power Down and Power Up procedures for the hepcms cluster

1. 1 Power Down and Up procedures

1. 2 Power Down

2. 2.1 Emergency/Essential steps:

   1. 2.2 Not as essential but useful steps:

   2. 2.3 Commands for power down in either case:

3. 3 Power Up: Recover from a full cluster outage Outline

4. 4 Recover from HN reboot

5. 5 Recover from r720-datanfs reboot

6. 6 Recover from Ovirt/Foreman reboot

7. 7 Recover from Interactive/Worker Node reboot

8. 8 Recover from hepcms-namenode reboot (OSG 3.2)

9. 9 Recover from hepcms-se reboot (OSG 3.3)

10. 10 Recover from hepcms-ce reboot (OSG 3.3)

11. 11 Recover from reboot: Cluster checks

EXternal Links

External references/Links:

Disks/Dell:

Dell linux hot swap sysadmin reference: https://grox.net/sysadm/unix/linux_disk_hotplug_helpful_commands

Setting up xfs: https://www.percona.com/blog/2011/12/16/setting-up-xfs-the-simple-edition/

Dangerous - delete partition with fdisk: http://www.cyberciti.biz/faq/linux-how-to-delete-a-partition-with-fdisk-command/

Dangerous - fdisk management: http://www.thegeekstuff.com/2010/09/linux-fdisk/

RedHat quotas SL6: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/ch-disk-quotas.html

Dangerous: hadoop datanode disk repair: http://hep-t3.physics.umd.edu/HowToForAdmins/errors.html#errorsHadoopFsck

http://www.cyberciti.biz/faq/linux-disk-format/

http://www.ehow.com/how_1000631_hard-drive-linux.html

http://lissot.net/partition/ext2fs/labels.html

Main Dell landing page for server update: http://www.dell.com/support/article/us/en/19/SLN293301

Dell: Maintaining R510 drives: http://www.dell.com/Support/Article/us/en/19/HOW10062

Dell firmware: https://lonesysadmin.net/2011/03/07/the-easiest-way-to-update-a-dell-servers-firmware/

For PowerVault (r720-datanfs, hepcms-gridftp): http://www.dell.com/support/home/us/en/04/Drivers/DriversDetails?driverId=F96NR

R720 Manual

 http://topics-cdn.dell.com/pdf/poweredge-r720_owner's%20manual_en-us.pdf

Ovirt:

RAM snapshots: http://www.ovirt.org/Features/RAM_Snapshots#Commit_to_snapshot

Single disk snapshot: http://www.ovirt.org/Features/Single_Disk_Snapshot

Live snapshots: http://www.ovirt.org/Live_Snapshots

Puppet/Hiera:

Puppet getting started with hiera: http://docs.puppetlabs.com/hiera/1/#getting-started-with-hiera 

OSG puppet github: https://github.com/opensciencegrid/puppet-contrib

Looking for official puppet modules: https://forge.puppetlabs.com

How to hiera/module: http://garylarizza.com/blog/2013/12/08/when-to-hiera/

Refactoring puppet class to use with Hiera: http://rnelson0.com/2014/10/20/rewriting-a-puppet-module-for-use-with-hiera/

Puppet, getting different data types with Hiera: http://codingbee.net/tutorials/puppet/puppet-retrieving-data-from-yaml-files-using-hiera/

Hiera/r10k blog post: http://rnelson0.com/2014/07/21/hiera-r10k-and-the-end-of-manifests-as-we-know-them/

Puppet: languages and namespaces: https://docs.puppetlabs.com/puppet/latest/reference/lang_namespaces.html#autoloader-behavior

Puppet: beginner's guide to modules: https://docs.puppetlabs.com/guides/module_guides/bgtm.html#step-one-giving-your-module-purpose

Hiera yaml format: http://symfony.com/doc/current/components/yaml/yaml_format.html

http://www.puppetcookbook.com/posts/creating-a-directory.html

Grid/OSG:

CERN IT support status: old page http://it-support-servicestatus.web.cern.ch/it-support-servicestatus/default-dynamic.asp

CERN status (improved page, login for most info): https://cern.service-now.com/service-portal/sls.do

T2Admin guide twiki: https://twiki.cern.ch/twiki/bin/viewauth/CMS/T2AdminGuide

CompOpsStoreTemp policy for hadoop: https://twiki.cern.ch/twiki/bin/view/CMSPublic/CompOpsStoreTemp

Other CompOps information: https://twiki.cern.ch/twiki/bin/view/CMSPublic/CompOps

Puppet OSG from Trey: https://github.com/treydock/puppet-osg

Our old cluster (SL5): http://hep-t3.physics.umd.edu/

Linux/admin:

How much CPU/RAM do I have on this machine? http://geroldm.com/2012/06/find-out-cpu-memory-and-harddisk-info-in-linux/

25 most common iptables example: http://www.thegeekstuff.com/2011/06/iptables-rules-examples/

Marguerite's UNIX/ROOT/CMSSW cheat sheet: http://cmsdoc.cern.ch/~belt/CheatSheet.html

rsync: http://www.tecmint.com/rsync-local-remote-file-synchronization-commands/

rsync: https://www.digitalocean.com/community/tutorials/how-to-use-rsync-to-sync-local-and-remote-directories-on-a-vps

rsync: https://gist.github.com/KartikTalwar/4393116

10 useful commands to configure (and check) a network interface - you are better off using Foreman or ovirt to configure, but this is if you really need to do it by hand: http://www.tecmint.com/ip-command-examples/

Yum&RPM from OSG Release3: https://twiki.opensciencegrid.org/bin/view/Documentation/Release3/YumRpmBasics

Bootable Linux USB from Mac (used this sucessfully): http://borgstrom.ca/2010/10/14/os-x-bootable-usb.html

Alternate Bootable Linux USB from Mac: 

    Step one, need hdutil convert: http://unix.stackexchange.com/questions/114984/how-to-create-a-bootable-linux-installation-usb-from-an-iso-in-os-x

    step 2: http://blog.tinned-software.net/create-bootable-usb-stick-from-iso-in-mac-os-x/

Alternate Bootable Linux USB from Mac: http://osxdaily.com/2015/06/05/copy-iso-to-usb-drive-mac-os-x-command/

Bootable Mac Linux loader (makes liveCD where you can write things - may or may not be Linux bootable, but I hope it is): http://blog.sevenbits.tk/Mac-Linux-USB-Loader/

NIS:

Install and secure NIS master: http://www.setuptips.com/unix/install-configure-nis-master/

NIS tools (managing): http://docstore.mik.ua/orelly/networking_2ndEd/nfs/ch13_04.htm

NIS best practices: http://archive09.linux.com/feature/114201

NIS, what files are managed: http://docstore.mik.ua/orelly/networking_2ndEd/nfs/ch03_03.htm

NIS add new users: http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch30_:_Configuring_NIS#Adding_New_NIS_Users

complete guide to useradd: http://www.tecmint.com/add-users-in-linux/

OSG/Tier3:

The main link for all OSG stuff: https://twiki.grid.iu.edu/bin/view/Documentation/Release3/WebHome  

https://twiki.cern.ch/twiki/bin/view/CMSPublic/USCMSTier3Doc

https://twiki.cern.ch/twiki/bin/viewauth/CMS/T2AdminGuide (requires CMS authentication - T2, NOT T3 guide, some but not all info overlaps

Monitoring

a book on Ganglia, available online: 

https://www.safaribooksonline.com/library/view/monitoring-with-ganglia/9781449330637/

Miscellaneous from past emails

http://tldp.org/HOWTO/Clock-2.html

http://www.freeraidrecovery.com/library/raid-5-6.aspx

I actually looked at this last night, this is the manual for the

controller for the RAID6 disks on hepcms-hn:

http://www.flagshiptech.com/eBay/Dell/poweredgeh310h710h810UsersGuide.pdf

https://www.lsc-group.phys.uwm.edu/daswg/download/vmwareSL6.html I don't

recall what I looked at here

SE-0-2:

http://en.community.dell.com/support-forums/servers/f/906/t/19590720

ROOT Password

(Copied from old cheatsheet  Kak: Nov-4)

• Nodes with oldest root pass:

  • sl5, any node not part of Foreman that still has SL5 and Rocks 5.4 

    • SE-0-3 : broken disk

    • interactive-0-4: broken disk

  • public network switch web page, username admin, old root pass

  • private network switch web page, username root, old root pass

  • hepcms-gridftp

  • iDRAC

  • clush password

• Nodes with new root pass:

  • hepcms-ovirt web page, username admin, new pass

• Nodes with 2016 summer root pass:

  • hepcms-in8, hepcms-in1, hepcms-gums,hepcms-se, hepcms-squid, hepcms-ce, hepcms-hn-backup, hepcms-secondary-namenode, hepcms-namenode, hepcms-vmtest, foreman-vmtest2, r720-0-1, r720-0-2, r720-datanfs,compute-0-10, compute-0-5, foreman, head node, hepcms-in2, compute-0-11, compute-0-8, compute-0-6, r510-0-1, r510-0-4, r510-0-5, r510-0-6, r510-0-9, r510-0-10, r510-0-11

Sudoers has been implemented (by hand) on:

• compute-0-5, compute-0-11, r510-0-1

  • Kak, Margarita, James, Marguerite

• hepcms-hn

  • drjohn,treydock,reduardo, Shabnam, Marguerite, Margarita, Kak

Trying to install new software doesn't work due to user account issues:

• Check that the user didn't already exist, and if the software uses shared filesystem, the user must have the same UID and groupid across all systems. So check the information on hepcms-hn as well as the individual nodes

  • id hdfs

• Also can get list of groups on a node this way:

  • getent group