Recently we upgraded our production environment to a shared application file system. The intention of posts like these isn't to list out step by step which tasks we followed while performing the upgrade but to provide you with an overview of our approach and issues you may encounter.
To start, it would probably be best to provide you with an overview of our architecture. Our production environment is by use by almost everyone in the company. If you have an expense, we use EBS, if you have to submit a timesheet, chances are you use EBS. Of course, its mostly used as a financials environment but by the end of this year more and more modules will be brought online.
As you can see from the diagram, we have a load balanced application tier, a single database node with a cold failover and most of the environment resides on a SAN.Details on how to implement a Shared Application filesystem can be found in Note: 233428.1Here are the basic steps we followed:
Merge existing APPL_TOPS. The document above doesn't say which node should be the primary in a shared application filesystem. I did encounter one note: 372758.1 which mentions WEB node must be primary. In the end, we did choose the admin node as primary since it would cut down on the implementation time.
When running perl adcfgclone.pl appsTier on the primary node after you copy the files from the secondary nodes, you have to say yes to all services.
Share the filesystem to the other Nodes. Make sure the filesystem is mounted in nolock mode if this is a unix based environment. Or you may encounter the problem described in Note: 390425.1 - Cannot Start Apache After Adding A Web Node On Shared APPL_TOP. For our environment, we shared a SAN mount point on the dbTier via NFS to the application tiers.
Add a node to the shared application filesystem. Note: you will need to create a location for each application tier to store configuration files. It needs to be created before you run txkSOHM.pl. In our environment, we used local disk to avoid the issue above.
Issues Encountered
Even tho we ran the adadmin option to check for missing files there were still some that were missing after the merge. Files that were customized on the secondary nodes but not the primary node (pre-merge) did not get updated. One example is the appslogo.gif file. I'm not sure what logic Oracle follows when it determines which files need to be brought over from the secondary servers, so i'm assuming that since appslogo.gif already existed on the primary server, Oracle didn't replace it.
We also noticed that there were quite a few files missing under the $OA_HTML subdirectories.
We have SSL setup in our environment, so we had to re-enable it via the txkrun.pl script. I believe the reason we did this was so we didn't have to manually edit all the files it touches. You will need to copy your SSL certificates to the location you created on each server for configuration files.
Since we have a loadbalanced environment we had to change the related context file values to the appropriate settings. It appears the clone conveniently resets those values. (ie, Web enty%, Active Web Port, Login page... etc)
If you receive a the infamous yellow security bar within forms after you regenerate the jar files and you didn't choose the web tier as your primary server, then you may need to copy your adcert.txt, adsign.txt and adappltop.cer to the primary node.
Maintenance mode via adadmin only works on the primary server. I hit this issue once and haven't double checked it to confirm. Since all patching is performed on the primary server this really isn't an issue.
Each time we run autoconfig a number of files have to be modified, such as adovars.env, some context variables.
As I have mentioned above, on our application tier we have forms/web and on the database tier concurrent/admin. The problem with our approach above is that all the services are enabled on the dbTier. So, when you execute adstrtal.sh all of the services, even apache start up. Obviously, this is not something you want. You can maintain your own start/stop scripts, remove the scripts for the services you don't need (and see some errors when running adstrtal.sh) or you can rerun adcfgclone.pl on the primary tier and disable the services you don't want.
When you execute adpatch it checks $APPL_TOP/admin/adconfig.txt file for which services are enabled on that tier. This file will still show that all services are available, so patching is not an issue. We tested this configuration by applying some patches in our sandbox and confirming that forms and web files have been updated properly.