FNDSM-Service Manager Tips and Tricks and Concepts
An E-Business Suite system depends on a variety of services like
Forms Listeners, HTTP Servers, Concurrent Managers, Workflow Mailers etc. Such services are composed of one or more processes that must be kept running for the proper functioning of the E-Business Suite. Until recently many of these processes had to be individually started and monitored by system administrators. Management of these processes was complicated by the fact that these services could be distributed across multiple host machines. The new Service Management feature for Release 11i helps to greatly simplify the management of these processes by providing a fault tolerant service framework and a central management console built into Oracle Applications Manager 11i.
Service Management is an extension of Concurrent Processing, which provides a powerful framework for managing processes on multiple host machines. Today, services such as the Oracle Forms Listener, Oracle Reports Server, Apache Web listener, and Oracle Workflow Mailer can be run under Service Management.
With Service Management, the Internal Concurrent Manager (ICM) manages the various service processes across multiple hosts. On each host, a Service Manager acts on behalf of the ICM, allowing the ICM to monitor and control service processes on that host. System administrators can then configure, monitor, and control services though a management console which communicates with the ICM.
Service Management provides a fault tolerant system. If a service process exits unexpectedly, the ICM will automatically attempt to restart the process. If a host fails, the ICM may start the affected service processes on a secondary host. The ICM itself is monitored and kept alive by Internal Monitor processes located on various hosts.
Service Management provides significant improvements in the manageability of the E-Business Suite. System administrators can now use the central console in Oracle Applications Manager 11i to manage a variety of services that formerly had to be managed independently on separate hosts. The entire set of system services may be started or stopped with a single action. Service Management also provides a great benefit by automatically compensating for certain system failures.
The GSM automates the management of Java processes.
If you are on Release 11.5.7 or higher and have already used AutoConfig to configure your Oracle Applications system, then GSM has already been configured , .
Configuration Notes
GSM and Multiple Nodes
GSM enables users to manage Applications services across multiple middle-tier nodes. This includes services on Web/Forms nodes that previously have had no concurrent processing footprint. Users configuring GSM in a multiple-node system should be sure to have followed the instructions for Parallel Concurrent Processing. This includes setting the environment variable APPLDCP=ON and assigning a primary node for all defined managers and services (if not already defined.)
Seeded GSM Services
Users that configure GSM using AutoConfig or by manually using an Applications Context File (section 2.2) will have the following GSM Services seeded automatically: Forms Listener, Metrics Server, Metrics Client, Reports Server, Apache Listener. These services, once seeded, may be managed under GSM and controlled via the Oracle Applications Manager. Note: LINUX users should not Activate the Reports Server under GSM.
2.1 Prerequisite patches
Apply the prequisite patches required for your installation.
For all installations,
Apply patch 2221688 for Concurrent Processing GSM configuration files, if this patch has not yet been applied.
If you are configuring you system using an Application Context file as described in Section 2.2 below, you should do the following:
If you are not on AD minipack F or later, apply the latest AD minipack from OracleMetalink.
Migrate to JDK 1.3.1 following the instructions in Note 130091.1 on OracleMetaLink.
Note: Section 2.2 refers to steps if you will be using an Applications Context file for your configuration. Section 2.3 refers to manual configuration steps. If you are not on 11.5.7 and higher and using AutoConfig, you should follow the instructions in either section 2.2 or 2.3, but not both.
2.2 Configuring your system using an Application Context file.
Refer to MetaLink Note 204090.1 for instructions on how to configure your system for GSM using an Applications Context file.
The configuration instructions referenced in this note will generate the configuration files required by GSM. As a result of the configuration steps described in this section, an Applications Context file will be created. Existence of this Applications Context file does not mean that your system has been converted to use AutoConfig.
2.3 Configuring your system manually.
If you are on UNIX, refer to MetaLink Note 210122.1 for instructions on how to configure your system manually.
If you are on Windows, refer to MetaLink Note 197361.1 for instructions on how to configure your system manually.
To verify GSM is working, start the concurrent managers. Once GSM is enabled, the ICM uses Service Managers to start all concurrent managers and activated services. If the ICM is successfully starting the managers, then GSM has been configured properly. If managers and/or services fail to start, errors should appear in the ICM logfile. See the Troubleshooting Guide below for help with these errors. Each Service Manager maintains its own logfile named FNDSMxxxx.mgr, located in the same directory as concurrent manager logfiles. It will also be useful to examine these logfiles when there are problems starting services. If you cannot locate the Service Manager logfile it is likely that the Service Managers are not starting properly and there is a configuration issue that needs troubleshooting.
This guide will help you locate common configuration problems.
If you start the concurrent managers and are experiencing problems with GSM, the ICM log file is a good place to start troubleshooting the problem.
Note: In PCP mode be sure there are no extraneous nodes registered. All inactive nodes should be marked as disabled as otherwise the ICM will take a long time trying to test the node.
If the log file has recorded errors we can begin to diagnose the problem:
CONC-SM TNS CONNECT ERROR: Could not start Service Manager &SM_NAME. The TNS alias...
CONC-SM INIT NODE ERROR: Could not initialize the Service Manager &SM_NAME. Verify that &NODE...
CONC-SM INIT ENV ERROR: Service Manager &SM_NAME could not initialize. The Service Manager...
CONC-SM TNS CONNECT ERROR
or
"Could not start Service Manager &SM_NAME. The TNS alias could not be located, the listener process on &NODE could not be contacted, or the listener failed to spawn the Service Manager process."
This means the ICM could not successfully contact and initiate a Service Manager process. There was an error in contacting the listener and the spawning the FNDSM process from the listener. This usually means one of the configuration steps needs to be reexamined. A good place to start is finding the listener logfile as
specified in the listener.ora file. (You did start the listener, right?) This log file should be able to tell you if the listener is getting contacted by the ICM and
what the ICM is trying to start.
If the listener log as a line like this:
01-JAN-2000 12:00:00 * (CONNECT_DATA=(SID=FNDSM_prod115)) *
(ADDRESS=(PROTOCOL=tcp)(HOST=1.2.3.4)(PORT=12345)) * establish * FNDSM_prod115 * 0
then it is being contacted by the ICM.
If the listener log file does not show contact from the ICM:
This probably indicates a problem with the tnsnames.ora file. Did you add the FNDSM entries appropriately? They should be of the form :
FNDSM_<node_name>_<dbname> = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=<node_name>)(PORT=<port_number>))
(CONNECT_DATA=(SID=FNDSM_<dbname>))
)
Note: If the &SM_NAME listed in the ICM logfile does not start with 'FNDSM_' this means you have the RRA prefix set in your system for FNDFS. Either unset the prefix and rename your FNDFS services or apply patch 2125967.
Be sure that node_name matches the node_name you have started the listener on and that it is the node you are trying to start managers and/or services on. A common and yet easy to overlook mistake is to switch the order of the db and node name in the service name (e.g. FNDSM_prod115_ap999sun instead of FNDSM_ap999sun_prod115). Also make sure you are dealing with the correct TNSNAMES.ora. The TNS_ADMIN environment variable will override the location otherwise used. One way to double check that an entry exists is to use sqlplus to try to make a connection:
sqlplus apps/apps@fndsm_mynode_mydb
will attempt to contact the fndsm service. If the error message returned is that TNS could not resolve the name, then there is still a problem with the tnsnames entry. If the entry exists, you will see different error messages as SQL*Plus and FNDSM are not designed to talk to each other (there still might be some problem with the entry (e.g. incorrect SID) but the entry is being found. This check is probably best done immediately after running startmgr, so you will be in the same environment as the ICM.
If the node_name and port_number are correct and the entry is specified correctly, the ICM will be able to contact the listener. If the listener log shows contact but you're not establishing a connection to FNDSM: check that the SID specified in the tnsnames.ora as above matches the SID described in the listener.ora file. When you start the listener it should tell you it is listening for the SID. The listener process will dump errors to the screen from where it was started as well as to its logfile, so it's a good idea when troubleshooting to bounce the listener from a local window you can observe. Double-check your listener.ora entry is correct. Test the APPSORA.env file you specified to make sure it has no errors. Make sure an FNDSM executable exists under $FND_TOP/$APPLBIN on UNIX, or %FND_TOP%\bin on Windows. If there is no FNDSM executable use adrelink to make one. Sometimes the APPSORA and other environment files expect certain commands to be available in their environment. ('sed and 'awk' are examples.) The environment files will error if they cannot find these. If this seems to be your problem, you should locate where these commands reside and add the directory to the PATH variable in your listener.ora file as follows:
In this example we need to add the '/bin' directory to the path:
[on UNIX platforms]
(SID_DESC=(SID_NAME=fndsm_prod115)
(ORACLE_HOME=/local/dB/8.0.6)
(program=/my_appl_top/fnd/11.5.0/bin/FNDSM)
(ENVS='MYAPPSORA=/myappltop/APPSORA.sh,PATH=/bin:$PATH,FNDSM_SCRIPT=/d1/APPS/11.5.3/prodappl/fnd/11.5.0/bin/gsmstart.sh'))
[on Windows platforms]
(SID_DESC=(SID_NAME=fndsm_prod115)
(ORACLE_HOME=/local/db/8.0.6)
(program=/my_appl_top/fnd/11.5.0/bin/FNDSM)
(ENVS='MYAPPSORA=/myappltop/APPSORA.sh,PATH=/bin:%PATH%,FNDSM_SCRIPT=/d1/APPS/11.5.3/prodappl/fnd/11.5.0/bin/gsmstart.sh'))
CONC-SM INIT NODE ERROR
or
"Could not initialize the Service Manager &SM_NAME. Verify that &NODE has been registered for concurrent processing."
This suggests a problem occurred in the initialization of the Service Manager. Did you register all nodes used in your system in FND_NODES? You must add entries to FND_NODES for all nodes you are using, even in single node environments. These entries are used to seed Service Manager entries. If you look at the Administer Concurrent Managers form and do not see any entries for Service Managers - one for each node, (and the profile option switch has been turned on) then you have not successfully registered your nodes.
Note:: Do NOT simply edit an already registered node to reflect your new node_name. For GSM you must register your node anew. If you have your node registered and there is a service manager listed and you still get this error, this would suggest a problem in the environment the FNDSM process is being spawned from. Test the environment script specified as 'MYAPPSORA' in the listener.ora entry and make sure it successfully creates an environment.
CONC-SM INIT ENV ERROR
or
"Service Manager &SM_NAME could not initialize. The Service Manager has been unable to verify its environment and create its log file."
This would suggest the Service Manager is able to start but has failed to open a log file. The most common cause is that the Service Manager does not have write
permissions for the directory it is trying to open a log file in. The Service Manager process is started by the same user that started the listener that spawns
it. It is recommended that you use applmgr to start the listener.
If you find errors not covered by these steps, please be prepared to provide examples of your ICM log, listener log, tnsnames.ora, listener.ora, as well as the
environment script used as MYAPPSORA and any FNDSM logfiles if they exist.