Concurrent Manager

The Master Concurrent Managers

There is a lot of talk about "the" concurrent manager in Oracle Applications. Actually, there are many Concurrent Managers, each governing flow within each Oracle Apps areas. In addition there are "super" Concurrent Managers whose job is to govern the behavior of the slave Concurrent Managers. The Oracle e-Business suite has three important master Concurrent Managers:

Internal Concurrent Manager — The master manager is called the Internal Concurrent Manager (ICM) because it controls the behavior of all of the other managers, and because the ICM is the boss, it must be running before any other managers can be activated. The main functions of the ICM are to start up and shutdown the individual concurrent managers, and reset the other managers after one them has a failure.
Standard Manager — Another important master Concurrent Manager is called the Standard Manager (SM). The SM functions to run any reports and batch jobs that have not been defined to run in any specific product manager. Examples of specific concurrent managers include the Inventory Manager, CRP Inquiry Manager, and the Receivables Tax Manager.
Conflict Resolution Manager — The Conflict Resolution Manager (CRM) functions to check concurrent program definitions for incompatibility rules. However, the ICM can be configured to take over the CRM's job to resolve incompatibilities.

Overview of Concurrent Processing

http://docs.oracle.com/cd/A60725_05/html/comnls/us/fnd/over.htm

FND Concurrent Manager FAQ's [ID 105133.1]

When would one be required to bounce (stop and restart) the Concurrent Manager?

When you modify the Printer Driver you have to restart the Manager which runs the request which is attached to that Printer Driver, however,if you do not know which manager then you have to restart the Internal manager because the printer driver can be used by multiple managers and multiple requests. If only a concurrent program definition is modified, running a verify on the Internal Manager will pick up the changes without the need for bouncing the manager.

Does the Internal manager schedule requests to be run or does it put requests into queues to be run by other managers?

This is a very common misconception. The ICM really does not have any such scheduling responsibilities. It has NOTHING to do with scheduling requests, or deciding which manager will run a particular request. Its function is only to run 'queue control' requests, which are

requests to startup or shutdown other managers. It is responsible for startup and shutdown of the whole concurrent processing facility, and it also monitors the other managers periodically, and restarts them if they should go down. It can also take over the Conflict Resolution manager's job, and resolve incompatibilities.

If the ICM itself should go down, requests will continue to run normally, except for 'queue control' requests. You can restart it with

'startmgr', you do not need to kill the other managers first.

How can I check to see if a concurrent manager is running?

One way to see if a manager is running is to use the 'Administer Concurrent Managers' form. Navigate to Concurrent->Managers->Administer. You will see two columns labeled 'Actual' and 'Target'. The Target column lists the number of processes that should be running for each manager for this particular workshift. The Actual column lists the number of processes that are actually running. If the Actual column is zero, there are no processes running for this manager. If the Target column is zero, then either a workshift has not been assigned to this manager, or the current workshift does not specify any target processes. If the target column is zero, then the manager processes have either failed to start up, or gone down. You should check the manager's logfile and the ICM logfile. You can also search for OS processes using the 'ps' command. It is possible for the form to be inaccurate, i.e. it may show actual processes even though they are not really running. When in doubt, check for processes at the OS level. On NT, you can check to see if the Concurrent Manager service is running using the Services control panel.

Where do concurrent request or manager logfiles and output files go?

The concurrent manager first looks for the environment variable $APPLCSF. If this is set, it creates a path using two other

environment variables: $APPLLOG and $APPLOUT .It places log files in $APPLCSF/$APPLLOG, output files go in

$APPLCSF/$APPLOUT

So for example, if you have this environment set:

$APPLCSF = /u01/appl/common

$APPLLOG = log

$APPLOUT = out

The concurrent manager will place log files in /u01/appl/common/log, and output files in /u01/appl/common/out

Note that $APPLCSF must be a full, absolute path, and the other two are directory names.

If $APPLCSF is not set, it places the files under the product top of the application associated with the request. For example, a PO report would go under $PO_TOP/$APPLLOG and $PO_TOP/$APPLOUT

Logfiles go to: /u01/appl/po/9.0/log

Output files to: /u01/appl/po/9.0/out

All these directories must exist and have the correct permissions.

Note that all concurrent requests produce a log file, but not necessarily an output file. Concurrent manager logfiles follow the same convention, and will be found in the $APPLLOG directory

What are the logfile and output file naming conventions?

Request logfiles: l.req

Output files: If $APPCPNAM is not set: .

If $APPCPNAM = REQID: o.out

If $APPCPNAM = USER: .out

Where: = The request id of the concurrent request

And: = The id of the user that submitted the request

Manager logfiles:

ICM logfile: Default is std.mgr, can be changed with the mgrname

startup parameter

Concurrent manager log: w.mgr

Transaction manager log: t.mgr

Conflict Resolution manager log: c.mgr

Where: is the concurrent process id of the manager

Can I delete a concurrent manager?

You can disable the manager by checking the 'Enabled' checkbox, or you can simply Terminate the manager and it will not run again unless you reactivate it. If it is really necessary, you can query the manager in the 'Define Manager' form, and delete the row. (It is recommended that you DO NOT do this)

What is the function of the 'Conflict Resolution Manager'?

Concurrent managers read requests to start concurrent programs running. The Conflict Resolution Manager checks concurrent program definitions for incompatibility rules.

If a program is identified as Run Alone, then the Conflict Resolution Manager prevents the concurrent managers from starting other programs in the same conflict domain.

When a program lists other programs as being incompatible with it, the Conflict Resolution Manager prevents the program from starting until any incompatible programs in the same domain have completed running.

What is the 'Internal Scheduler/Prereleaser' manager?

The short name for this manager is FNDSCH. It is also known as the Advanced Scheduler/Prereleaser Manager. This manager is intended to implement Advanced Schedules. Its job is to determine when a scheduled request is ready to run. Advanced Schedules were not fully implemented in Release 11.0, they are implemented in Release 11.5, but are not widely used by the various Apps products. General Ledger uses FNDSCH for financial schedules based on different calendars and period types. It is then possible to schedule AutoAllocation sets, Recurring Journals, MassAllocations, Budget Formulas, and MassBudgets to run according to the General Ledger schedules that have been defined.

If financial schedules in GL are not being used then it is not a problem to deactivate this manager.

What is the 'Internal Monitor' manager/service?

This manager/service is used to implement Distributed Concurrent Processing. It monitors whether the ICM is still running, and if the ICM crashes, it will restart it on another node. You do not need to run this manager/service unless you are using Distributed Concurrent Processing.

How do I check/set the PMON method?

To check the PMON method:

1) cd $FND_TOP/sql

2) sqlplus apps/apps @afimchk.sql

This will tell whether the internal manager is running, what the PMON

method is, and where the log file is.

To set the PMON method:

1) first shut the concurrent managers down

2) cd $FND_TOP/sql

3) sqlplus apps/apps @afimpmon.sql LOCK (or RDBMS)

How do I enable/disable the Conflict Resolution Manager?

Use the system profile option 'Concurrent: Use ICM'. Setting this to 'No' (which is the default) allows the CRM to be started.

Setting it to 'Yes' causes the CRM to be shutdown and the Internal Manager (ICM) will take over the conflict resolution duties. If the CRM will not start (it is started automatically by the ICM), check this profile option.

Note that using the ICM to resolve conflicts is not recommended. The CRM's sole purpose is to resolve conflicts, while the ICM has

other functions to perform as well. Only set this option to 'YES' if you have a good reason to do so.

How do I clean out the Concurrent Manager tables?

Cleaning out the tables is a useful method of making sure that there are no invalid statuses that can prevent the managers from starting. Previously, this has been done by truncating fnd_concurrent_processes and/or fnd_concurrent_requests. Truncation of the tables is a little drastic, and can cause problems later when trying to purge requests, not to mention losing all of the request information.

Run the script, cmclean.sql, article note 134007.1 CMCLEAN.SQL - Non Destructive Script to Clean Concurrent Manager Tables

It will make sure the relevant status codes are valid without deleting any information.

How do I tell concurrent manager processes apart at the OS level?

Use: pf -ef | grep FNDLIBR

This will produce output like:

vd11 13703 13660 0 May 11 ? 0:01 FNDLIBR FND Concurrent_Processor MANAGE OLOGIN="APPS/94A491A1000000000000000000

n1070161 24936 24927 0 Apr 29 ? 0:05 FNDLIBR FND Concurrent_Processor

MANAGE OLOGIN="APPS_APPDEMO/94C4B1C10000000000

n1070161 24938 24927 0 Apr 29 ? 0:06 FNDLIBR FND Concurrent_Processor

MANAGE OLOGIN="APPS_APPDEMO/94C4B1C10000000000

n1070161 24927 24922 0 Apr 29 ? 2:03 FNDLIBR FND CPMGR FNDCPMBR sysmgr

="" sleep=60 pmon=20 diag=N logfile=/u16/app

The last process, #24927, shows 'FNDLIBR FND CPMGR', this one is the Internal Manager (ICM). Notice that it gives some of the parameters it was started with, the other processes showing 'Concurrent_Processor' are Standard manager processes. Notice that the ICM process is the parent process of the Standard managers. (processes 24936 and 24938)

Other managers will have the name of the executable, like ARLIBR or INVLIBR:

$ ps -ef | grep ARLIBR

vd11 13683 13660 0 May 11 ? 0:20 ARLIBR APPS/82A2A4940000000000000

000000000000000000000000000000000000000 AR ART

The Conflict Resolution manager will look like:

$ ps -ef | grep FNDCRM

n1070161 24941 24927 0 Apr 29 ? 1:17 FNDCRM APPS_APPDEMO/84BFBEB900000

0000000000000000000000000000000000000000000000

What is the syntax for controlling the concurrent manager using startmgr and concsub in NT?

On NT, the concurrent manager is run as an NT service, you start and stop the managers using the Services control panel.

Why am I seeing pinging entries like this in the ICM logfile?

PING (0.0.0.0): 56 data bytes

64 bytes from 192.75.91.2: icmp_seq=0 ttl=255 time=0.705 ms

64 bytes from 192.75.91.2: icmp_seq=1 ttl=255 time=1.120 ms

Process monitor session ended : 29-FEB-2000 10:38:43

64 bytes from 192.75.91.2: icmp_seq=2 ttl=255 time=0.985 ms

64 bytes from 192.75.91.2: icmp_seq=3 ttl=255 time=1.006 ms

Pinging other machines is used in Distributed Concurrent Processing. This means you have DCP turned on, using the environment variable APPLDCP. Set APPLDCP to OFF and restart the managers.

I hit the Restart button to start the Standard manager, but it still did not start?

Telling a manager to restart just sets the status to Restart. The ICM will start it the next process monitor session or the next time the ICM starts. Use Activate to start a manager immediately.When a manager is deactivated manually, the ICM will not restart it, you will need to set it to Restart, or activate it manually.

How many rows are in FND_CONCURRENT_REQUESTS and FND_CONCURRENT_PROCESSES tables?

Depending on the specification of the system it has been seen that when tables reach above 3000-4000 rows, the performance begins to diminish, however, there could be 30000-40000 rows in the tale before the performance begins to degrade.

You may want to run the Purge Concurrent Request and/or Manager Data on regular basis, dependant on the amount of requests being run.

The Purge Concurrent Requests job can be used to purge:

Requests, Mgr logs, and All requests depending on what is chosen.

Use the following options: Enter = All, Mode = AGE, Mode Value = 15

The std.mgr log continuously grows where it may good to archive it regularly.

Any processes pending in Internal or Conflict Resolution Manager?

Best course of action before starting the Concurrent Managers is to cancel any "Deactivate" or "Verify" jobs pending in the Internal Manager and place ny other pending jobs on hold.

How do I turn on transaction manager diagnostics?

Set the profile option 'Concurrent:Debug Flags' to 'TCTM1' at the site level. This will cause transactions to make debug entries in the

FND_CONCURRENT_DEBUG_INFO table. Truncate this table before running a tranasction, then select the entries from the table.

Starting the managers with diag=Y will also produce more information in the transaction manager logfile.

How do transaction managers work?

Briefly:

(See the server documentation for details on the DBMS_PIPE package)

1) A tranasction manager is started on the concurrent processing server, and periodically reads the pipe for incoming transactions.

2) A client program (usually a form) calls the FND_TRANSACTION.SYNCHRONOUS function.

3) This function writes a message into the pipe containing the program to be run and its parameters.

4) FND_TRANSACTION.SYNCHRONOUS begins reading a return pipe for the return status.

5) The manager sees the message in the pipe, retrieves the program id and parameters.

6) The manager runs the program with the specified parameters. The program will be of type 'Immediate', so there will not be a

separate concurrent request run.

7) The program completes, and the manager packs its return status into the return pipe.

8) FND_TRANSACTION.SYNCHRONOUS reads the return value and passes it back to its caller.

Note that these events take place essentially simultaneously on the client and server. This is a synchronous transaction because the

client waits for the server to return, or times out waiting for it.

Problem....

When you try to submit a request like Active users or Active responsibilities, request gets submitted.

When we view the help requests, you find that it is inactive / nomanager. Within 12 to 15 seconds, you refresh-it gets completed.

Initially, you could find only inactive and we look at the diagnostic- the concurrent manager assigned is not picking up.

There is no specialization rules in any managers except the include program this source.

Solution....

Most often when this occurs where a request goes "inactive/no manager" and is then processed a short time later, the solution is to either increase the cache size for your Standard manger, or increase the actual number of Standard manager processes.

Cache Size is set on the CONCURRENT/MANAGER/DEFINE form. Basically, this regulates how many requests a manager will pick up for each

sleep cycle.

How do I process more concurrent requests concurrently?

The Concurrent Manager parameters, (Query the concurrent manager by

the relevant concurrent manager), should be modified to handle more concurrent requests concurrently, this can be done in two steps:

(i) Increase the Number of Target processes for the manager

(ii) Change the cache size of the concurrent manager as this determines

how many requests will be evaluated by a manager at a time and should match the target (process) value as set above.

FNDSM and FNDFS

The tnsnames.ora file of the 806 home has many entries starting with FNDFS_ and FNDSM_.

FNDFS or the Report Review Agent (RRA) is the default text viewer within Oracle Applications, which allows users to view report output and log files. Report Review Agent is also referred to by the executable FNDFS. The default viewer must be configured correctly before external editors or browsers are used for viewing requests. Check out metalink note 111383.1 which describes the Basics About Report Review Agent (FNDFS) on 11i.

FNDSM

FNDSM is executable & core component in GSM ( Generic Service Management Framework discussed above). You start FNDSM services via APPS listener on all Nodes in Application Tier in E-Business Suite.

Sometimes after cloning you may find that concurrent managers do not start up. Metalink Note 305986.1 describes the scenario:

Symptoms

You have followed instructions within Note 230672.1 - Cloning Oracle Applications Release 11i with Rapid Clone and completed the cloning successfully.

However, now the Concurrent Manager cannot be started. The following error is produced in the Internal Concurrent Manager log file:

Could not start Service Manager FNDSM_hostname.

The TNS alias could not be located, the listener process on hostname could not be contacted, or the listener failed to spawn the Service Manager process.

Cause

This is caused because FNDSM_hostname service is not present in the target instance.

FNDSM database trigger is used to create the FNDIM_ and FNDSM_ concurrent managers. This trigger is present on FND_NODE table. If triggers are disabled in source then issue may occur.

Solution

- Run this following select SQL statements on the target node only to check trigger status:

> SELECT trigger_name , status

FROM user_triggers

WHERE table_name = 'FND_NODES' ;

- If status of the triggers show as DISABLED, then enable these following triggers as follows:

> Connect apps/apps

> alter trigger UPNAME enable;

> alter trigger FNDSM enable;

- Take backup of FND_NODES table in Target.

> create table FND_NODE_bk as select * from FND_NODES ;

- Login into Oracle Application with System Administrator Responsibility.

- Delete the rows from FND_NODES table in target using install -> node -> Define form.

- Then re-Register TRAGET hostname.

- This should register FNDIM_ and FNDSM_ concurrent managers in FND_CONCURRENT_QUEUE Table.

For cluster configuration, when the cluster name is manually inserted in FND_NODES, you can do the following to ensure that FNDFS_ entries are created in tnsnames.ora :

From the Application tier:

1. Log in as applmgr

2. cd to $FND_TOP/patch/115/sql

3. Run the script: afdcm037.sql

4. This script will create libraries for FNDSM and create Managers for Preexisting Nodes.

Note: Service Manager "FNDSM" can not be created from form:

Concurrent> Manager> Define under Sysadmin Responsibility.

Reference Bug 6085070 FNDSM TRIGGER CAUSES SERVICE MANAGER NOT TO BE CREATED AFTER CLONING SINGLE NODE