Home‎ > ‎Knowledge Base‎ > ‎

WHM 11 cPanel - HowTo - General Maintenance

As of WHM 11.28.xx and WHM 11.30.xx there are many constant changes occurring. Keep an eye for updates to this page.

Last Update: 02/25/2011

Requirements:
You will need a good understanding of cPanel administration, bash commands and general knowledge of the make command.

I am working on compiling my notes to produce a step by step guide for understanding and running maintenance on a cPanel server. We will be focusing on WHM cPanel version 11 with Red Hat Enterprise 5, however most of the issues contained within should be fairly universal and hopefully will stand the test of time as newer software comes out. For those that have seen my work, I strongly believe that if the server is not hacked, then the next focus is to keep the server on the standard upgrade path to prevent future issues. Hence any modifications to a server that strays from the standard future upgrade path should be avoided with EXTREME PREJUDICE! However contradictory to that, WHM is extraordinarily customizable without straying off the standard upgrade path, making it a very reliable robust and long standing package. So keeping these two perspectives in mind, cPanel is unique in that understanding in depth all the REAL issues of the server maintenance will fix about 90% of any WHM cPanel error. Or if you use the same methods as standard operation, you will then avoid 90% of the errors everyone else has down the road.

Part 1 - Perl

The core of WHM / cPanel is built upon Perl so keeping the internal Perl engine updated has one of the highest priorities. Doing a basic check on the health is fairly easy with the /scripts/ tools in WHM. This will run minor upgrades on the Perl Modules WHM thinks is important and report on the overall health of the Perl install.


/scripts/checkperlmodules

Using fast module check.....checked 112 modules in 4 second(s)
Tested 113, 113 ok, 0 failed.
Cpanel::FastMath: [INSTALLED=1] [VERSION=0.2]
Cpanel::POSIX::Tiny: [INSTALLED=1] [VERSION=0.7]
Cpanel::Cleanup: [INSTALLED=1] [VERSION=0.4]


The output is very long but at the end it will tell you if you have failed results. If you do have failed results you will most likely have to reinstall perl from the cPanel repository. The fact that this particular instance shows 113 modules is not relevant, that will depend on various options that you have enabled such as different modules loaded into EasyApache.

For the most part if /scripts/checkperlmodules does not report any errors then you are done with the Perl section of the maintenance. However every now and again you will have to come back and double check Perl. Either you just want to ensure everything in Perl is upgraded, or you just happen to have some bug that seems to be Perl related and you just need to dig deeper. Go back and run /scripts/checkperlmodules again and look at the full detailed output. This will report on the minor upgrade issues that most of the time can be overlooked. Generally they will fall into four categories. Since the first run is going to have a ton of extra output from the upgrades it already managed, run it a second time just to get a listing of failures without all the extra verbose information that you are not going to need. Trust me, it just makes life a lot easier.

1) /scripts/checkperlmodules does not check that particular Perl module
2) The Perl module requires a manual upgrade
3) The Perl module requires a manual removal and then to be upgraded
4) The location of the Perl module repository accidentally got moved to a new location and is not in the directory listing.

For the first part we will run a manual upgrade of the major Perl modules. From the CPAN shell the install command will check to see if a package is installed and if it needs to be updated. Again the actual output is very verbose. What you will do is run the commands twice. The first time through if there is an update it will begin the installation. The second time through if the upgrade completed then it will report {Ok}. If the upgrade failed then it will continue to try to update. Of those that failed, take note of which ones are broken.

The following list was borrowed from 2. Installing Perl Modules


cpan

reload index

install DateTime
install DBI
install DBD::mysql
install Class::Autouse
install Digest::MD5
install Digest::SHA1
install HTML::Template
install Image::Size
install MIME::Lite
install MIME::Words
install Compress::Zlib
install Net::DNS
install URI::URL
install HTML::Tagset
install HTML::Parser
install LWP::Simple
install LWP::UserAgent
install GD
install Mail::Address
install Unicode::MapUTF8
install XML::Simple
install IO::WrapTie
install Unicode::CheckUTF8
install Captcha::reCAPTCHA
install Digest::HMAC_SHA1
reload cpan

quit

For the second option you will have to scroll back in the code to look for packages that have not been upgraded, for example CPAN. Take not that while this is a single command, there are multiple modules that get installed from it. Again you will run these commands twice to see which modules failed to update.

cpan

install Bundle::CPAN
reload cpan

quit

Once done the repeat with /scripts/checkperlmodules and the manual cpan updates until no further updates are reported.

For the third part is when the upgrade from the CPAN shell fails, then you have to manually un-install the module before running the upgrade. At the time of writing this there is an issue with SQL::Statement where the error code looks like the following.

WARNING! You seem to have an older version of SQL::Statement already installed (1.20 <= 1.20).
This new version introduces a number of features that will impact operation of SQL::Statement and of DBD drivers for CSV, AnyData, and Excel.

at Makefile.PL line 47
No 'Makefile' created  REHSACK/SQL-Statement-1.23.tar.gz
  /usr/bin/perl Makefile.PL -- NOT OK
Running make test
  Make had some problems, won't test
Running make install
  Make had some problems, won't install
Failed during this command:
REHSACK/SQL-Statement-1.23.tar.gz            : writemakefile NO -- No 'Makefile' created

Now if you are like me and working as a professional the number one rule is you never manually rip anything out of a system unless you back it up first! The modules fall into a hierarchy as follows.

First locate the root of the Perl install. This can be frustrating and annoying because different versions of the same distributions will have different locations, or worse, on older server it will have multiple locations. Most of the time it is in one of two locations, and yes sometimes even both.

/usr/local/lib/perl5
/usr/lib/perl5

So if you find both look at the current distribution

/usr/local/lib/perl5/site_perl/5.8.X/Module/
/usr/lib/perl5/site_perl/5.8.X/Module/

Once will have many modules, the other will have only one or two. On the server I am checking on it is currently /usr/local/lib/perl5/site_perl/5.8.8/ but the point I am getting to is the Perl root is /usr/local/lib/perl5/ and we are looking for SQL::Statement.

/%perl root%/site_perl/%perl_version%/%Module%/%Sub_Module%/

In this example it is as follows. Note this may be different on other servers such as /usr/lib/ instead of /usr/local/lib/.

/usr/local/lib/perl5/site_perl/5.8.8/SQL/Statement/

ls -la /usr/local/lib/perl5/site_perl/5.8.8/SQL/Statement/
total 104
drwxr-xr-x 2 root root  4096 Jul 17 02:03 ./
drwxr-xr-x 4 root root  4096 Jul 17 02:03 ../
-r--r--r-- 1 root root 11585 Mar  5  2009 Embed.pod
-r--r--r-- 1 root root 19201 Mar  5  2009 Functions.pm
-r-xr-xr-x 1 root root 21775 Mar  5  2009 GetInfo.pm*
-r-xr-xr-x 1 root root  3133 Mar  5  2009 RAM.pm*
-r--r--r-- 1 root root 11812 Mar  5  2009 Structure.pod
-r--r--r-- 1 root root 18355 Mar  5  2009 Syntax.pod
-r-xr-xr-x 1 root root  2782 Mar  5  2009 Util.pm*

So first we want to make a backup and then rip out the old module and install the new version.

mkdir /root/backup/
mkdir /root/backup/perl/
mkdir /root/backup/perl/site_perl/
mkdir /root/backup/perl/site_perl/5.8.8/
mkdir /root/backup/perl/site_perl/5.8.8/SQL/
mkdir /root/backup/perl/site_perl/5.8.8/SQL/Statement/
cp -a /usr/local/lib/perl5/site_perl/5.8.8/SQL/Statement/*  /root/backup/perl/site_perl/5.8.8/SQL/Statement/.
rm -rf  /usr/local/lib/perl5/site_perl/5.8.8/SQL/Statement/

cpan

install SQL::Statement
reload cpan
quit

The fourth issue is very rare but the most frustrating. Perl is a collection of modules in which the individual modules have different developers. As such the different modules are published in different locations, many of which are Universities. So basically if you happen to be installing or upgrading Perl and it is the beginning of a new school year and a new student aid is being trained on publishing Perl modules, well you get the idea. The distribution list is published under the second half of /%perl root%/site_perl/%perl_version%/pod/perlmodlib.pod under CPAN Sites. You do not want to change any of the existing entries, but instead you will want to add additional entries as needed.

So for example lets say you are installing SOAP in CPAN and you are getting an error that the file can not be found under ftp://ftp.cise.ufl.edu/pub/mirrors/CPAN/ Then you do some more debugging and you find that the server is searching ftp://ftp.cise.ufl.edu/pub/mirrors/CPAN/l...by-module/SOAP/ as the path but when you go to the location in a web browser in the month of October and find it is actually in ftp://ftp.cise.ufl.edu/pub/mirrors/CPAN/m...by-module/SOAP/ then the server is not going to be able to find the location unless you change the publish point.

=item Florida

     ftp://ftp.cise.ufl.edu/pub/mirrors/CPAN/
     http://mirror.csit.fsu.edu/pub/CPAN/
     ftp://mirror.csit.fsu.edu/pub/CPAN/
     http://cpan.mirrors.nks.net/

change to

=item Florida

     ftp://ftp.cise.ufl.edu/pub/mirrors/CPAN/
     http://mirror.csit.fsu.edu/pub/CPAN/
     ftp://mirror.csit.fsu.edu/pub/CPAN/
     http://cpan.mirrors.nks.net/
     ftp://ftp.cise.ufl.edu/pub/mirrors/CPAN/modules/CPAN/

So now your server can find the module and install it even though it is published in the wrong location. As well once the student aid gets an F for the week and the error is corrected on their side, then the server will still be able to find it even if it is published in the correct location.

If the server is over a year old you will want to check the version. Perl on WHM is slightly customized and should be installed from the installer published on http://layer2.cpanel.net/ Currently with WHM version 11 it should be matched to Perl 5.8.8

perl -v

This is perl, v5.8.8 built for i686-linux

Copyright 1987-2006, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

If you need to reinstall Perl, it if pretty simple but has a lot of verbose output so don't let that scare you too much.

wget http://layer2.cpanel.net/perl588installer.tar.gz
tar -xzvf perl588installer.tar.gz
cd perl588installer
./install
cd ~
/scripts/checkperlmodules

There are some other instances of Perl having problems updating but this article is not intended to be all inclusive of repairing Perl. However the submodules are constantly being updated so most of the time the sub module update is not a severer issue and you simply wait a week of two for someone to notice the last update failed and try again.

And now you have a fresh new install of Perl that has not yet been upgraded. Congratulations, now start from the beginning of this post once again! Wash Rinse Repeat where does it all end?

Part 2 - System Update Service

This section is more particular to RedHat and CentOS systems, other distributions such as Ubuntu of BSD may vary as they have different subsystems used for system updates. In those cases you will want to refer to the system documentation for maintenance to the upgrade system.

Older systems will use up2date where as newer system will use yum. for the most part if these systems are operating correctly then you are good to go.

yum list

or

up2date -l

One of the more common errors as a server ages is that the registration will expire and needs to be re-registered. For this you would need to contact your software provider for support. If you are on ThePlanet.com network then you would contact their technical support.

With yum some times a simple cleaning will corrected an error.

yum clean all

With up2date and some versions of yum you may have to clear the RPM spool. This issue is more common with Plesk upgrades but will sometimes effect cPanel. Once you are certain this is cleared you can delete the old backup.

mkdir  /var/spool/up2date.backup
mv /var/spool/up2date/* /var/spool/up2date.backup/.

On newer versions of yum you can also check the RPM cache.

mkdir  /var/cache/yum.backup
mv /var/cache/yum/* /var/cache/yum.backup/.
yum clean all

As a server ages eventually it will no longer be supported by the Linux Distribution and no more updates will come through. At that time you should plan on retiring the server. For cPanel Maintenance to continue the server will still need to have the output of the update service to function properly even though no more updates are being published. The reason for this is that not all the updates come from the Linux distribution so it will parse the output of the update service to make decisions on what to upgrade from other sources. However eventually the service that the updater connects to will be abandon as well. Once this happens you should make final preparations to retire the server.

Part 3 - Snakeoil

The most common problems with cPanel stem from one single file which is /etc/cpupdate.conf. When you buy a new computer with Windows, you have to agree to the licensing and tell it that you want to have automatic updates, otherwise it will not run the system updates. cPanel is no different, it will not run the updates until you set them up. For WHM 11 this is found under "Main >> Server Configuration >> Update Preferences". In WHM 9 and 10 there were 3 options, where as in WHM 11 there are 11 options. All of these options will need to be set, otherwise the updates will not run correctly. To verify they have been set, look at /etc/cpupdate.conf and make sure there are 11 lines of code matching what was selected on the form. For operating systems that are still active it is suggested to use the following:

cPanel/WHM Updates - Automatic (either Stable or Release no real preference)
cPanel Package Updates - Automatic
bandmin - Inherit
courier - Inherit
dovecot - Inherit
exim - Inherit
ftp - Inherit
mysql - Inherit
nsd - Inherit
python - Inherit
spam assassin - Inherit
Security Package Updates - Automatic

However once the operating system has reached the End of Life, things can break because half the updates are built from source code using Perl. However these updates are going to want the dependencies from the operating system (RPM's from RedHat or CentOS for example). Since the Operating system is End of Life these dependencies will not be met and things will start breaking. So we then need to set everything to Never.

cPanel/WHM Updates - Never
cPanel Package Updates - Never
bandmin - Never
courier - Never
dovecot - Never
exim - Never
ftp - Never
mysql - Never
nsd - Never
python - Never
Security Package Updates - Never

First we will want to check to make certain the system updates are not in a locket state. So for Red Hat and CentOS Linux that would be the RPM system.

ls -la /var/lib/rpm/
total 46392
drwxr-xr-x  2 root root     4096 Oct 31  2009 ./
drwxr-xr-x 28 root root     4096 Oct  1  2009 ../
-rw-r--r--  1 root root  5505024 Jun  3 01:54 Basenames
-rw-r--r--  1 root root    12288 Jun  3 01:54 Conflictname
-rw-r--r--  1 root root        0 Oct 31  2009 __db.000
-rw-r--r--  1 root root    24576 Oct 31  2009 __db.001
-rw-r--r--  1 root root  1318912 Oct 31  2009 __db.002
-rw-r--r--  1 root root   450560 Oct 31  2009 __db.003
-rw-r--r--  1 root root  1945600 Jun  3 01:54 Dirnames
-rw-r--r--  1 root root  5341184 Jun  3 01:54 Filemd5s
-rw-r--r--  1 root root    24576 Jun  3 01:54 Group
-rw-r--r--  1 root root    16384 Jun  3 01:54 Installtid
-rw-r--r--  1 root root    45056 Jun  3 01:54 Name
-rw-r--r--  1 root root 34721792 Jun  3 01:54 Packages
-rw-r--r--  1 root root   172032 Jun  3 01:54 Providename
-rw-r--r--  1 root root    98304 Jun  3 01:54 Provideversion
-rw-r--r--  1 root root    12288 Oct 30  2009 Pubkeys
-rw-r--r--  1 root root   249856 Jun  3 01:54 Requirename
-rw-r--r--  1 root root   139264 Jun  3 01:54 Requireversion
-rw-r--r--  1 root root    94208 Jun  3 01:54 Sha1header
-rw-r--r--  1 root root    45056 Jun  3 01:54 Sigmd5
-rw-r--r--  1 root root    12288 Jun  3 01:54 Triggername

The /var/lib/rpm/__000.db file is a simple flag stating that the RPM is in a locked state. Either there is an application that has the RPM subsystem open or the last application forgot to unlock the system on exit. So we want to first verify if there are any applications that need to be shut down.

for i in $(ps -elf |egrep '(yum|up2date|rpm|upcp)' |grep -v 'grep'|awk '{print $4}'); do kill $i; done

Double check for the /var/lib/rpm/__000.db file, and it still exists then it should have a file size of zero. If so then manually delete it. Otherwise is the file size is not zero, there is something horribly wrong!

The WHM cPanel updates are handled in three sections controlled by /scripts/sysup /scripts/bandminup and /scripts/upcp. However if all three scripts are run in sequence, this will trigger a hidden 4th script to run called AUTOHEAL. While AUTOHEAL may sometimes run with /scripts/upcp, it appears that it always runs if these three scripts are called in sequence. Additionally it is just a generally good practice to start every thing and end everything with /scripts/checkperlmodules. Generally /scripts/upcp and /scripts/checkperlmodules are setup parallel rather than linear. That is to say that an update from either script can effect the dependencies on the other script. If a server has fallen far behind in many updates it may be common to run these two scripts multiple times. Finally there is an option for /scripts/upcp to bypass the MySQL checks which will save many hours for waiting on this to complete by using touch /etc/mysqlupdisable. Adding all this into one quick script nicknamed "Snake Oil" looks like the following:

!!! - NEW AS OF WHM 11.28.83 - !!!
Note: CPANEL=STABLE may vary from system to system.
Note: RPM SARULESUP SYSUP are now showing as "release" instead of "daily"

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
VERIFY THIS IN WHM UNDER - Main >> Server Configuration >> Update Preferences
THE DETAILS OF /etc/cpupdate.conf ARE CHANGING ON A WEEKLY BASIS

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

echo "BANDMINUP=inherit
COURIERUP=inherit
CPANEL=stable
DOVECOTUP=inherit
EXIMUP=inherit
FTPUP=inherit
MYSQLUP=inherit
NSDUP=inherit
PYTHONUP=inherit
RPMUP=release
SARULESUP=release
SYSUP=release" > /etc/cpupdate.conf

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


touch /etc/mysqlupdisable;\
/scripts/checkperlmodules;\
/scripts/sysup;\
/scripts/bandminup --force;\
/scripts/upcp --force;\
/scripts/checkperlmodules;\
rm -rf /etc/mysqlupdisable

If you are still having errors that are not related to Exim, Apache, PHP, or MYSQL then go back to step one and start again. Unfortunately Exim, Apache, PHP, or MYSQL errors are well beyond what can be posted here. Perhaps they will make future articles.

Part 4 - Planning

WHM cPanel is dependent upon Perl, and the one quirks about Perl is that it is maintained by computer science student aids across the world. Keeping this in mind, there are 3 times a year that you should not try to do anything such as trying to modify Perl, and avoid trying to rebuild Apache. The end of September has new student aids coming in. This one is not so bad, but has the occasional screw up every few years. Christmas through New Years can be bad since if they made a mistake they will be gone for a few weeks before someone will come in to fix it. Spring Break? Ok, this is really simple. Imagine drunken tanked up young twenty something year old's chasing scantily clad bikini girls across the beach. Are you certain you want to trust your server to these kids at this particular time? Hopefully the answer to that question is no!

I used to work for a small company. No I actually worked for a tiny company which started a policy they named "Cradle to Grave". We had less than 8 technicians at any one time, and most of the time we really only had 6 techs. Yet we boasted a combined history of over 200 years experience between all of the technicians. The concept was that if a technician installed a device, so long as they worked for the company they were the lead tech for that installation. So if I need assistance on one of my jobs, even if it was the foreman that was assisting me I was in charge. And if I was assisting the new hire who had graduated to doing his first install, I was the helper.

All this lead me to a very simple story. How many times have you heard someone complain their roof was leaking? They get the roof repaired every other year, blame the roofing company, and then after years of patching or even replacing the same roof, someone finally comes along and points out that the foundation is cracked and sinking. So long as the foundation was sinking, which pulled down the walls and made the roof unbalance, the roof will always continue to break apart and leak. And if you really thing about it, outside of patching the foundation you practically have to tear down the entire building and start from scratch if you really wanted it to be repaired correctly.

Computers are no different, planning our what the purpose of the computer will be used for, how it will be managed and maintained, all the way to how it will be replaced when it has reached the end if its lifespan, all these things should be planned out before ever building the system. Otherwise when your roof starts leaking, you can expect it to only get worse until you start over and build a new server.

So once you have figured out how much RAM you need, how much disk space for the operating system including disk space for upgrades over the next 3 to 6 years, how much disk space for /home/ according to expected end users, how much disk space for /var/ which is where /var/lib/mysql will be, how much disk space for /usr/ since Apache on cPanel logs there. Once you have this planned out, only then are you ready to order a server.

Once a server is on line, have plenty of time to setup the configuration and security, test out the security, and give it some time for a burn in. Take for example hard drives. Hard Drive manufactures boast level whatever air cleaners that allow for only 1 single particle of dust for so many cubic feet. Well the reality is that that 1 single particle of dust will eventually land an a hard drive while it is assembled, and might just make it past quality testing and gets installed in a computer somewhere. So you got a brand new computer or server and the odds are not very likely it will happen to you. But if you happen to be an IT company where you see many hard drives every year verses the typical home user, the odds of a hard drive going poof in the first few days grows exponentially. And that is simply one of many pieces of hardware that makes a server.

Last is that a computer ages very quickly, and eventually it will need to be replaced. What method have you planned to migrate the data to a newer server when the time comes? And just i case that method is broken, or even what if that technology is so outdated and no longer exists, what is you backup plan on migrating the data to a newer server? How many years do you plan on keeping a server? If you are a very large company boasting the highest level of product support, you might expect to get 3 years of service out of a server. Compare that to a company that advertises economy based hosting, then you might get 6 years out of a server. Anything more than that you are just getting lucky until something goes wrong.

Comments