IBM Support

MustGather: For FileNet Process Engine 5.0

Question & Answer


Question

What information should be collected when I experience a problem with the IBM FileNet P8 Process Engine 5.0?

Answer

Must Gather for FileNet Process Engine 5.0
Collecting data before calling IBM support will help you understand the problem and save time analyzing the data.

Collecting data early, before opening the PMR, helps IBM® Support quickly determine if:

  • Symptoms match known problems (rediscovery).
  • There is a non-defect problem that can be identified and resolved.
  • There is a defect that identifies a workaround to reduce severity.

Locating the root cause of a problem can speed development of a code fix.

Note that there are many significant differences between the new PE 5.x software and the older PE 4.x software. This technote describes how to collect information for the PE 5.0 release. The Must Gather for PE 4.5.1 and older releases is here:
http://www.ibm.com/support/docview.wss?uid=swg21327304.

This document describes the information that should be collected for Process Engine problems. You can collect this information "by hand," using the information described here. Or, you can use the automated IBM FileNet Process Engine 5.0 Data Collector. Using the Process Engine 5.0 ISA Data Collector is significantly easier than gathering the information by hand.

The use of the automated IBM FileNet Process Engine 5.0 Data Collector is highly encouraged. If you use the IBM FileNet Process Engine 5.0 data collector, you would only have to be concerned about the steps described under the "Collecting Information Specific to the Problem" heading. For example, if you can reproduce the problem, you might enable the appropriate traces and replicate the problem and then run the automated data collector. The data collector would then gather those traces along with all of the other basic information described below.

The IBM FileNet Process Engine 5.0 Data Collector is described here:
http://www.ibm.com/support/docview.wss?uid=swg21454222.

Gathering General Information

This information is needed, but is generally added to the PMR or the defect:

  • How often does this problem occur?
  • Can the problem be reproduced?
  • How many users are affected by this problem?
  • What is the business impact of this problem?
  • Is there a workaround?

Gathering Operating System Versions, Settings and Logs

All of the information in this section is collected automatically if you use the IBM FileNet Process Engine 5.0 Data Collector.

These commands collect:
  • OS Version
  • OS Environment Variables
  • OS Kernel parameters
  • OS running processes
  • OS networking information
  • OS locale information
  • OS log files

Run the following commands on the PE Server and collect the output:

AIX Servers
CommandCollects
uname -a or oslevel -sOS Version
/usr/sbin/instfix -ivq OS Patches or Maintenance level
env OS Environment Variables
errpt -aOS Configuration & Error Reporting
/usr/sbin/lsattr -E -l sys0 OS Kernel parameters
ps -efcurrently running processes
netstat -ansocket information
no -a | grep udp_ephemeralephemeral port info
locale
locale -a
locale information
/var/adm/syslog/syslog.loginclude system log files
ls -RalF of the ProcessEngine directorydirectory listing of Process Engine files

HP Servers
CommandCollects
uname -aOS Version
/usr/sbin/swlist -l product OS Patches or Maintenance level
env OS Environment Variables
/usr/sbin/dmesg OS Configuration & Error Reporting
/usr/sbin/sysdef OS Kernel parameters
ps -efcurrently running processes
netstat -ansocket information
ndd /dev/udp udp_smallest_anon_port
ndd /dev/udp udp_largest_anon_port
ephemeral port information
locale
locale -a
locale information
/var/adm/syslog/syslog.loginclude system log files
ls -RalF of the ProcessEngine directorydirectory listing of Process Engine files

Solaris Servers
CommandCollects
uname -aOS Version
/usr/bin/showrev -a OS Patches or Maintenance level
env OS Environment Variables
/usr/sbin/prtdiag -v OS Configuration & Error Reporting
/usr/sbin/sysdef -D
/usr/bin/getconf -a
OS Kernel parameters
ps -efcurrently running processes
netstat -ansocket information
/usr/sbin/ndd /dev/udp udp_smallest_anon_port
/usr/sbin/ndd /dev/udp udp_largest_anon_port
/usr/contrib/bin/nettune -l
ephemeral port information
locale
locale -a
locale information
/var/adm/messages & /var/log/sysloginclude system log files
ls -RalF of the ProcessEngine directorydirectory listing of Process Engine files

Windows Servers
CommandCollects
systeminfoOS Version, Patches and Maintenance level
tasklistcurrently running processes
WMIC /OUTPUT:C:\ProcessList.txt PROCESS get Caption,Processid,Usermodetime,Commandlinecurrent running processes including the command line
setOS Environment Variables
Windows 2003:
cscript C:\windows\system32\eventquery.vbs
/L application /R 1000 /V
cscript C:\windows\system32\eventquery.vbs
/L system /R 1000 /V

Windows 2008:
wevtutil epl System Systemlog.evtx
wevtutil epl Application Applicationlog.evtx
OS Event Logs
netstat -ansocket information
chcpdisplay code page
dir /s of the ProcessEngine directorydirectory listing of Process Engine files
the following registry keys from SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\
MaxUserPort
TcpMaxConnectTransmissions
TcpMaxConnectRetransmissions
networking registry settings

Linux Servers
CommandCollects
uname -aOS Version
/usr/bin/showrev -a OS Patches or Maintenance level
env OS Environment Variables
/usr/sbin/prtdiag -v OS Configuration & Error Reporting
/usr/sbin/sysdef -D
/usr/bin/getconf -a
OS Kernel parameters
cat /proc/sys/net/ipv4/ip_local_port_rangeephemeral port information
/usr/bin/getconf -aconfiguration info
/bin/dmesgmessages
ps -efcurrently running processes
netstat -ansocket information
locale
locale -a
locale information
/var/log/messages.*include system log files
ls -RalF of the ProcessEngine directorydirectory listing of Process Engine files

Collecting PE Server Version and Configuration Information

All of the information in this section is collected automatically if you use the IBM FileNet Process Engine 5.0 Data Collector.

For the following commands, it is assumed that you're collecting information for the Default instance of the Process Engine. If you have multiple instances of the PE on the same server, you will need to specify the particular instance in place of the Default instance. That is, you would replace the word Default with the name of the desired PE virtual server in the commands below.

PE's CE Connection Information:

PEInit Default -sc

PE Server Version and Build information:

java -classpath pesvr.jar filenet.jpe.general.JPEVersion


java -classpath pe.jar filenet.vw.toolkit.utils.VWBuildStamp

You may need to fully specify the path to java and you may need to fully specify the path to pe.jar and pesvr.jar.

Run vwtool and collect the output from these vwtool commands:

Specify the appropriate Process Engine virtual server name when you run vwtool. If you are using the Default server, you do not need to specify the virtual server name when running vwtool. Otherwise run vwtool <server name> in the command prompt window. Then set the region appropriately, using vwtool's region command and run these commands, collecting the resulting output:

threaddump
configdetails
schemastatus
sysconfig b
appspace
dump f c s
config
queueconfig *
rosterconfig *
logconfig *
threaddump
dump s
regionConfig
dbviews
views
loadstatus
count *
threaddump

Note that the threaddump command appears three times, spread out between the other commands. This gives us three separate threaddumps separated by 30 or so seconds.


PE Database information

Database Type and Version: For example, Oracle 11g 11.1.0.6 or SQL Server 2008 or DB2 LUW 9.5 FP2. Note that vwtool's config command (collected above) will include this information in its output.

Is the database on the same server as the PE or on a remote database server?

Include the OS type and version information for database server if the database is remote.


PE's configured connection to the Content Engine

In a web browser on the PE Server, go to your CE Server's basic info pages:
http://ceserver:9080/wsi/FNCEWS40MTOM

http://ceserver:9080/FileNet/Engine

In the above URLs, replace "ceserver:9080" with the name and port for your CE Server.

Include the returned HTML pages.

If you don't have a web browser on the PE Server, you can do this in a command prompt to get an HTML page:

java -classpath pe.jar;pe3pt.jar filenet.vw.server.ConfigUtils /url http://ceserver:9080/wsi/FNCEWS40MTOM/

NOTE: You may need to fully specify the path to java and to the pe.jar and pe3pt.jar files
NOTE: If your PE Server is running on Unix or Linux, you will need to separate the jar file names with a colon rather than a semi-colon.


PE Configuration

Include the returned HTML page for these URLs as seen in a browser on the PE Server:

http://localhost:32776/IOR/ping?farm=true&user=Admin&pw=xxx
http://localhost:32776/IOR/ping?farm=true&acminfo=1&user=Admin&pw=xxx
http://localhost:32776/IOR/FileNet.PE.vworbbroker

You must substitute a valid PE administrator user and password for the example user name (Admin) and password (xxx) in the two IOR/ping URLs.

You must specify the an appropriate region number to replace the "1" in the "acminfo=" clause in the second URL.

NOTE: If you have changed the IOR port from 32776, you must use that new port in the above URLs.

If you don't have a web browser on your PE Server, you can use the same filenet.vw.server.ConfigUtils utility to acquire the HTML pages (as described above).


Process Engine Logs and Traces

Include the appropriate number of Process Engine log and trace files. There are two folders from which logs and traces should be collected. Both folders are under the ProcessEngine install directory. The first folder is .../ProcessEngine/data/logs/ and the second folder is .../ProcessEngine/data/<pe server>/logs/ folder. For example, on Windows, these might be
c:\Program Files\IBM\FileNet\ProcessEngine\data\logs\ and
c:\Program Files\IBM\FileNet\ProcessEngine\data\pesvr.default\logs\.
Zip up an appropriate number of logs and traces that are in those folders.


Collecting Information Specific to the Problem



The following sections describe extra information that can be collected for particular types of problems.


Farm Issues - Load Balancer Configuration

If the PE is being used in a farm configuration, read through the PE 5.0 Farm Troubleshooter:
http://www.ibm.com/support/docview.wss?uid=swg21570520

Then, include the following information:

  • Brand and model of load balancer
  • Load Balancer settings and configuration information
  • The results of what was found when working through the Farm Troubleshooter document.
  • The "hosts" file from each PE Servers in the farm.

The data collector does not gather this information automatically.

Tracing In General

The data collector does not automatically enable or create any tracing. But it will collect any traces that have already been created at the time the data collector is run.

If you will be collecting traces, it is easiest to generate the traces before running the data collector so that the traces are automatically included as part of the data collection.

If you have collected any traces, be sure to state the date and time when the problem has occurred.

Screenshots

The data collector does not automatically take any screenshots. It may be appropriate to include screenshots of errors or other symptoms that will help us understand the problem.

PE Server Fault

If the problem is reproducible, it can be invaluable to enable the appropriate PE traces and re-created the problem.  For example, in vwtool, you can use the trace command to enable server tracing. The exact traces you enable would vary, depending upon the type of server problem. And after enabling the traces, reproduce the problem.

Include the trace output file from a period where the problem has occurred. Be sure to state the date and time when the problem has occurred.

Include any dumps or cores that may be produced.


Reproducible Problem

If the problem is reproducible, it may be helpful to turn on "ORB RPC", "ORB RPC INPUT", and "Database Access" tracing. Then, reproduce the problem and include trace output. Be sure to state the date and time when the problem has occurred.


PE Server Hung or Performance Problem

If the server is hung or there is a performance issue, run vwtool on the PE Server (or on each server in the farm) and issue a "threaddump" command 3 times, separated by 30 second intervals. Include the thread dump files.

Also note the CPU load when the server was hung. Was the load at zero (indicating everything was blocked)? Or was it at some other number, indicating the server was possibly looping?

If the server is hung and the CPU load is not at zero, turn on "ORB RPC", "ORB RPC Input", and "Database access" tracing, and include the trace output. Be sure to state the date and time when the problem has occurred.


Security / Connection to CE / Logon / Group Membership issues

Enable the appropriate PE traces using vwtool's trace command. For security issues, it is generally good to enable the tracing for "Security calls" and "Environment cache".

Then, reproduce the problem and include the trace output. Be sure to state the date and time when the problem has occurred.


PE API Issues

Enable the appropriate java trace options in the fnlog4j.properties file.

Put that fnlog4j.properties file in the appropriate JVM’s LIB folder. Note that the appropriate JVM may be within an application server.

See the ...\ProcessEngine\data\fnlog4j.properties.sample file for an example fnlog4j.properties file.

Replicate the problem and submit the resulting java trace file. Be sure to state the date and time when the problem has occurred.

[{"Product":{"code":"SSTHRT","label":"IBM Case Foundation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Process Engine","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"},{"code":"PF016","label":"Linux"}],"Version":"5.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
17 June 2018

UID

swg21446755