Internal Error --- 500 1. Shutdown the http server, oc4j and opmn services using following scripts at $INST_TOP/admin/scripts adapcctl.sh stop adoacorectl.sh stop adformsctl.sh stop adoafmctl.sh stop adopmnctl.sh stop Note: If its during Production Down Time or a UAT/TEST Instance then all the middle tier services can be shutdown. 2. Backup/Delete existing log files in: $LOG_HOME/ora/10.1.3/Apache $LOG_HOME/ora/10.1.3/j2ee $LOG_HOME/ora/10.1.3/opmn Note: If the issue is easily reproducible it is safe to delete the log files (unless otherwise required) 3. Enable HTTP/OC4J/OPMN debug logging a) Enable http ODL logging, Edit httpd.conf file, add the following to the end of file $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf OraLogMode oracle OraLogSeverity TRACE:32 OraLogDir $LOG_HOME/ora/10.1.3/Apache/oracle Note: Please use the full path to $LOG_HOME e.g. OraLogDir /u01/inst/apps/JCB_atg/logs/ora/10.1.3/Apache/oracle b) Make the following directory which will be holding the created Apache ODL log files > mkdir $LOG_HOME/ora/10.1.3/Apache/oracle 4. Increase OC4J logging for oacore a) edit j2ee-logging.xml adjust the following in file: $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/config/j2ee-logging.xml <logger name='oracle' level='TRACE:32' useParentHandlers='false'> b) Increase OC4J logging for forms: edit j2ee-logging.xml adjust the following in file: $ORA_CONFIG_HOME/10.1.3/j2ee/forms/config/j2ee-logging.xml <logger name='oracle' level='TRACE:32' useParentHandlers='false'> c) edit orion-web.xml adjust the following in file: $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/application-deployments/oacore/html/orion-web.xml param-name>debug_mode</param-name> <param-value>true</param-value> 5. Increase OPMN Logging a) edit opmn.xml adjust the following in file: $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml <debug path="/u01/inst/apps/JB_atg/logs/ora/10.1.3/opmn/opmn.dbg" comp=" internal;ons;pm" rotation-size="1500000" rotation-hour="0" /> 6. Use the ojspcompile.pl perl script to perform a manual pre-compilation of the JSP pages. The following command will compile all the JSP pages and build up the JSP cache again. Unix: # $FND_TOP/patch/115/bin/perl ojspCompile.pl --compile --flush -p 2 Windows: C:> <FND_TOP>\patch\115\bin\perl -x ojspCompile.pl --compile --flush This utility is also used by the AD utilities to perform this action, for example after patches are applied which replaced 1 or more JSP pages. Since this a manual action which also requires understanding of parameters to be passed to have the right (set of) JSP compiled this option may be error-prone. 7. Startup the http server, oc4j and opmn (reverse the startup sequence above) 8. Reproduce the issue - Access the following urls and inform support if the url was successful or unsuccessful (with the resulting error): http://<server>.<domain>:<PORT>/OA_HTML/ServletPing http://<server>.<domain>:<PORT>/OA_HTML/OA.jsp?OAFunc=OAHOMEPAGE http://<server>.<domain>:<PORT> http://<server>.<domain>:<PORT>/OA_HTML/OAInfo.jsp http://<server>.<domain>:<PORT>/OA_HTML/jsp/fnd/aoljtest.jsp http://<server>.<domain>:<PORT>/OA_MEDIA/FNDLOGOL.gif http://<server>.<domain>:<PORT>/forms/frmservlet http://<hostname.domainname>:<port>/OA_HTML/fndvald.jsp?username=sysadmin&password=<sysadmin_password> 9. Upload the FRESH log files at: $LOG_HOME/ora/10.1.3/Apache $LOG_HOME/ora/10.1.3/j2ee $LOG_HOME/ora/10.1.3/opmn

Action plan-2

1.Choose another value for $CONTEXT_FILE variable: s_java_object_cache_port

2.Make sure its the same on all apps tiers

3.Run AutoConfig and retest

NOTE:issue is resolved

Gather Schema concurrent request is failed

+---------------------------------------------------------------------------+

Application Object Library: Version : 12.0.0

Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved.

FNDGSCST module: Gather Schema Statistics

+---------------------------------------------------------------------------+

Current system time is 26-JUN-2013 08:16:48

+---------------------------------------------------------------------------+

**Starts**26-JUN-2013 08:16:49

**Ends**26-JUN-2013 08:49:42

ORA-0000: normal, successful completion

+---------------------------------------------------------------------------+

Start of log messages from FND_FILE

+---------------------------------------------------------------------------+

In GATHER_SCHEMA_STATS , schema_name= ALL percent= 10 degree = 8 internal_flag= NOBACKUP

stats on table AQ$_WF_CONTROL_P is locked

stats on table FND_CP_GSM_IPC_AQTBL is locked

stats on table FND_SOA_JMS_IN is locked

stats on table FND_SOA_JMS_OUT is locked

stats on table GAT_REQ_QTBL is locked

Error #1: ERROR: While GATHER_TABLE_STATS:

object_name=GL.JE_BE_LINE_TYPE_MAP***ORA-20001: invalid column name or duplicate columns/column groups/expressions in method_opt***

Error #2: ERROR: While GATHER_TABLE_STATS:

object_name=GL.JE_BE_LOGS***ORA-20001: invalid column name or duplicate columns/column groups/expressions in method_opt***

Error #3: ERROR: While GATHER_TABLE_STATS:

object_name=GL.JE_BE_VAT_REP_RULES***ORA-20001: invalid column name or duplicate columns/column groups/expressions in method_opt***

+---------------------------------------------------------------------------+

End of log messages from FND_FILE

+---------------------------------------------------------------------------+

+---------------------------------------------------------------------------+

Executing request completion options...

Output file size:

0

Finished executing request completion options.

+---------------------------------------------------------------------------+

Concurrent request completed

Current system time is 26-JUN-2013 08:49:42

SOLUTION:

1.       Please also review "ORA-20005 WHEN GATHERING STATISTICS FOR EBS APPS 11i or R12 IN Database 10G or 11G [ID 375351.1]" for the messages

2.       Please review "Gather Schema Statistics fails with Ora-20001 errors after 11G database upgrade [ID 781813.1]" which appears to match your issue.

Cause

There are two reasons for that error message:

1 ) There are duplicate rows on FND_HISTOGRAM_COLS table for JE_BE_LINE_TYPE_MAP table.
Because of this problem, FND_STATS tries to gather histogram information using wrong command and
it fails with ora-20001 errors.

Following SQL should have returned one row , not two.

SQL>select column_name, nvl(hsize,254) hsize
from FND_HISTOGRAM_COLS 
where table_name = 'JE_BE_LINE_TYPE_MAP'
order by column_name;

COLUMN_NAME HSIZE
------------------------------ ----------
SOURCE 254
SOURCE 254

Since there are two rows in histograms table, FND_STATS creates following command to gather statistics on table 'JE_BE_LINE_TYPE_MAP' :

 dbms_Stats.gather_table_stats(OWNNAME => 'GL', TABNAME 
=>'JE_BE_LINE_TYPE_MAP',METHOD_OPT=>'FOR ALL COLUMNS SIZE 1 FOR COLUMNS SOURCE SIZE 254 FOR 
COLUMNS SOURCE SIZE 254');

 Above command will work on 9i and 10G databases but it will fail with ora-20001 errors on 11G.

2) Column does not exist on the table but still listed in FND_HISTOGRAMS_COL table.

You can use following SQL to identify. SQL will prompt for table name, use table name from the errors. In above examples you can use FII_FIN_ITEM_HIERARCHIES.

select hc.table_name, hc.column_name
from FND_HISTOGRAM_COLS hc , dba_tab_columns tc
where hc.table_name ='&TABLE_NAME'
and hc.table_name= tc.table_name (+)
and hc.column_name = tc.column_name (+)
and tc.column_name is null;

Solution

Find out all duplicates and/or obsolete rows in FND_HISTOGRAM_COLS and delete one of them.
Remember to take backup of the FND_HISTOGRAM_COLS table before deleting any data.

-- identify duplicate rows

select table_name, column_name, count(*)
from FND_HISTOGRAM_COLS
group by table_name, column_name
having count(*) > 1;

-- Use above results on the following SQL to delete duplicates

delete from FND_HISTOGRAM_COLS
where table_name = '&TABLE_NAME'
and  column_name = '&COLUMN_NAME'
and rownum=1;

-- Use following SQL to delete obsoleted rows

delete from FND_HISTOGRAM_COLS
where (table_name, column_name) in 
  (
   select hc.table_name, hc.column_name
   from FND_HISTOGRAM_COLS hc , dba_tab_columns tc
   where hc.table_name  ='&TABLE_NAME'
   and hc.table_name= tc.table_name (+)
   and hc.column_name = tc.column_name (+)
   and tc.column_name is null
  );

1-----à

Cause

This can happen with Advance Queue tables. In 10g, if a queue is created or recreated during the upgrade, automatic statistics gather is locked (disabled) on these queue.

The following statement can be used to check the tables which have statistics locked:

    select owner, table_name, stattype_locked
    from dba_tab_statistics
    where stattype_locked is not null; 

Solution

Unlock statistics gathering on those queues running the commands below. To unlock all the tables in a schema at once:

exec dbms_stats.unlock_schema_stats('schema_owner'); 

OR

To unlock individual tables (need to run for all tables individually):

exec dbms_stats.unlock_table_stats('table_owner','table_name');

Examples:

SQL> exec dbms_stats.unlock_schema_stats ('AR'); 
SQL> exec dbms_stats.unlock_table_stats('AR', 'AR_REV_REC_QT'); 

1. If CPU is maxing out (100%), then you may have no other choice than to kill off the sessions/processes that are consuming the high CPU.
This is normally due to spinning of the frmweb process due to returning rows from LOVs by end users that don't attempt to filter (perform blind query) in core forms.

When an end user login to forms and start working with LOV within core forms sometimes and according to the search criteria that the user will provide to filter the results in LOV, it may fetch huge numbers of records in which causes the frmweb process to grow very large, and in extreme cases this can even lock up the current process or even the whole machine.

So when executing a LOV query, every row is fetched into memory on the middle tier, the frmweb process can get extremely large, and the larger it gets the more likely it is to start paging.
Eventually it starts consuming excessive CPU just paging the process in and out of memory, which is probably what you can see here in this case as the amount of memory consumed when the LOV records are fetched into memory obviously depends on the amount of data in each record.

This has been mentioned in the following bug:
Bug 6519700 - ESC: CSE: R12SIP: 6513826 FRMWEB RUNAWAY PROCESS CONSUMING 100% CPU-MIDDLE TIER


Therefore, if the FRMWEB process is consuming near 100% CPU and sustaining this consumption, you may have to kill off the process to resume normal operation.


2. You may check the directory: $FORMS_TRACE_DIR on forms node. The FRMWEB process may create a coredump file here which may lead to clues of what caused the issue. The file is normally named: frmweb_dump_<PID>

Where PID is the process ID of the user running the form session.

Adrelink is exiting with status 1: for MSC Product
Issue:

adrelink is exiting with status 1: In 11i/R12  & RHEL 5.4 while patching

Fix:

STEP1:  cd $AD_TOP/bin

STEP2:  Take backup of the existing adrelinknew.sh

cp -rp  adrelinknew.sh  adrelinknew.sh_orig

STEP3: Edit the adrelinknew.sh

 vi adrelinknew.sh

Change CPP_LDFLAGS  as below

From:

CPP_LDFLAGS=' -L$(ORACLE_HOME)/lib -L$(ORACLE_HOME)/lib/stubs -lclntsh'

 To:

#CPP_LDFLAGS=' -L$(ORACLE_HOME)/lib -L$(ORACLE_HOME)/lib/stubs -lclntsh'
CPP_LDFLAGS=' -L$(ORACLE_HOME)/lib -L$(ORACLE_HOME)/lib/stubs -lclntsh -Wl,--noinhibit-exec'

:wq! --> save & exit

Symptoms:

Users unable to open the out files.

Concurrent requests failed with “Post-processing of request failed error message”

One or more post-processing actions failed. Consult the OPP service log for details.

No further attempts will be made to post-process this request.

Cause:

The concurrent manager process was able to successfully invoke the Output Post-Processor (OPP) but encountered a timeout as the OPP takes longer than the value assigned to complete the job.

 Solution:

Increase the value of profile Concurrent: OPP Response Timeout . Bounce Apache and retest.

If the issue still exists, perform the following steps.

Increase the number of Output Post Processors as follows:

Increase the number of processes for Output Post Processor.

Additionally, ensure there is a setting of  oracle.apps.fnd.cp.opp.OPPServiceThread:2:0:max_threads=5 under Parameters.

CONCURRENT REQUEST IS FAILED WITH ERROR 127:

/oracle/appltest/apps/tech_st/10.1.2/bin/rwrun: error while loading shared libraries: librw.so: cannot open shared object file: No such file or directory
++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1) unlink /usr/lib/libXtst.so.6
ln -s /usr/X11R6/lib/libXtst.so.6.1 /usr/lib/libXtst.so.6

2) cd 10.1.2 $ORACLE_HOME/reports/lib
make -f ins_reports.mk install

3) Relink application executables from the adadmin utility.

4) Bounce the services.

5) Retry failing concurrent requests.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++

Note: 551795.1 - How to change the default branding on the homepage which shown as "E-Business Suite" ?
Note: 395445.1 - Oracle Application Framework Profile Options Release 12
Note: 421636.1 - How to replace the default Oracle Logo with a Customized Logo?
Note: 459350.1 - R12 - Image Size Problem On Login Page