Being a Java developer or Java based Application server administrator like WebSphere or WebLogic , i am sure you should have come across this exception " Error: java.lang.ClassNotFoundException " which basically implies that the JVM is unable to find a definition for your class. It is usually resolved by adding the jar file that contains the class to the classpath or packaging the jar file along with your EAR or WAR file or placing the jar file in the right locations so that your Application Server classloader can find and load the class . So inorder to indentify the .jar file where the class file is present , the following simple UNIX command will help us to indentify the jar file and take appropiate actions to resolve the issue .
LINUX:
find PATH -name '*.jar' -o -name '*.war' -o -name '*.ear' -type f |xargs -i bash -c "jar -tvf {}| tr / . | grep CLASSNAME && echo {}"
(e.g)
find /usr/IBM/WebSphere/AppServer/ -name '*.jar' -o -name '*.war' -o -name '*.ear' -type f | xargs -i bash -c "jar -tvf {}| tr / . | grep org.apache.xerces.dom.DocumentImpl.class && echo {}"
Output:
11160 Fri Feb 20 14:00:34 PST 2004 org.apache.xerces.dom.DocumentImpl.class
/usr/IBM/WebSphere/AppServer/archive_lib/xerces-2.6.2.jar
11160 Fri Oct 06 10:12:24 PDT 2006 org.apache.xerces.dom.DocumentImpl.class
/usr/IBM/WebSphere/AppServer/updateinstaller/update.jar
The above command will recursively search all the jar , EAR and WAR files under the /usr/IBM/WebSphere/AppServer/ for the class name org.apache.xerces.dom.DocumentImpl.class and output the location of the .jar file that contains the class.
Although this command is useful only in unix environment but there are other java based utilities to find the same like Jar Class Finder , Java class finder, etc , but i find this command simple to use without having to copy addtional jar files to your systems.
Useful UNIX commands for WebSphere & Portal Administrator
These are some useful commands that i came across when administrating WebSphere Application Server or WebSphere Portal Server. Note these commands are specifically written for LINUX but most of them would for other UNIX platforms like Solaris, AIX and other flavours of LINUX like SUSE or REDHAT.
Performance related Commands to monitor system performance
-------------------------------------------------------------------------
1) Find the process that uses most CPU
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -2
(eg) output of the above command when websphere server process is utilizing the most of the CPU
%CPU PID USER COMMAND
71.7 31237 root /cust/IBM/WebSphere/AppServer/java/bin/java
-Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere
/AppServer/java/jre/lib/ext/ibmext.jar
-Dwas.status.socket=54859 -classpath
/cust/IBM/WebSphere/AppServer/profiles/AppSrv02/properties:/cust/IBM/WebSphere/AppSer
2) Find the last 10 process that use the most CPU
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10
%CPU PID USER COMMAND
2.1 6375 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=2132 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ
0.3 13886 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=25042 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppSer
0.2 17271 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9606 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ
0.2 17871 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9606 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ
0.2 17713 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9140 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ
0.1 18499 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=11309 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppSer
0.1 17504 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=8969 -classpath /cust/IBM/WebSphere/AppServer/profiles/Dmgr01/properties:/cust/IBM/WebSphere/AppServer
3)Find cpu usage
mpstat -P ALL or mpstat
Linux 2.4.21-27.ELsmp (machine1.www.com) 02/02/2008
11:55:00 AM CPU %user %nice %system %iowait %irq %soft %idle intr/s
11:55:00 AM all 1.51 0.00 0.41 0.04 0.01 0.09 97.94 184.92
11:55:00 AM 0 1.50 0.00 0.41 0.04 0.01 0.11 97.93 123.76
11:55:00 AM 1 1.51 0.00 0.42 0.04 0.00 0.07 97.95 61.16
4)Find CPU usage with the process listing in runtime and much more statistics in terms of memory, virtual memory, etc.
top
12:00:20 up 118 days, 15:16, 2 users, load average: 0.07, 0.02, 0.00
572 processes: 570 sleeping, 2 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 2.7% 0.0% 1.1% 0.0% 0.0% 0.0% 95.9%
cpu00 1.9% 0.0% 0.3% 0.0% 0.0% 0.0% 97.6%
cpu01 3.5% 0.0% 1.9% 0.1% 0.0% 0.0% 94.2%
Mem: 11819592k av, 5088092k used, 6731500k free, 0k shrd, 295304k buff
3998132k active, 682312k inactive
Swap: 2048276k av, 0k used, 2048276k free 1258592k cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
17719 root 16 0 306M 306M 3992 S 0.7 2.6 22:14 1 java
19358 root 16 0 2532 2532 896 R 0.5 0.0 0:00 1 top
17771 root 15 0 306M 306M 3992 S 0.4 2.6 3:48 0 java
18058 root 16 0 458M 458M 4032 S 0.1 3.9 12:55 0 java
1 root 15 0 512 512 452 S 0.0 0.0 1:19 0 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
5)Continuous CPU usage monitoring and find the average CPU used.
a) sar -u 2 5
b) sar -o output.file 12 8 >/dev/null 2>&1 &
c) nohup sar -o output.file 12 8 >/dev/null 2>&1 &
Linux 2.4.21-27.ELsmp (machine1.web.com) 02/02/2008
12:04:34 PM CPU %user %nice %system %iowait %idle
12:04:36 PM all 0.25 0.00 0.00 0.00 99.75
12:04:38 PM all 0.00 0.00 0.50 0.25 99.25
12:04:40 PM all 7.00 0.00 7.25 0.00 85.75
12:04:42 PM all 21.05 0.00 1.50 0.00 77.44
12:04:44 PM all 3.00 0.00 0.50 0.00 96.50
Average: all 6.26 0.00 1.95 0.05 91.74
Disk Usage related Commands to monitor Disk space
--------------------------------------------------------
1)Disk Usage on all mounts
df -h -T
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext3 2.9G 785M 2.0G 29% /
/dev/sda7 ext3 57G 9.3G 45G 18% /cust
/dev/sda6 ext3 2.0G 82M 1.8G 5% /lc
none tmpfs 5.7G 0 5.7G 0% /dev/shm
/dev/sda5 ext3 2.0G 33M 1.8G 2% /tmp
/dev/sda3 ext3 2.0G 64M 1.8G 4% /var
2)prints just one line with the total size of the directory, the below example show the size of the /usr/IBM/Websphere WebSphere Portal Server directory.
du -ch | grep total
bash-2.05b# du -ch | grep total
6.3G total
Network related Commands
-------------------------------------
1)
a) dig (domain information groper) - is a flexible tool for interrogating DNS name servers.
b) nslookup - Queries a name server for a host or domain lookup.
c) host - DNS lookup uility.
The below example shows the information like CNAME , Aliases, ipaddres , etc about www.google.com
a) dig
bash-2.05b# dig www.google.com
; <<>> DiG 9.2.4 <<>> www.google.com
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1088
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 7, ADDITIONAL: 0
;; QUESTION SECTION:
;www.google.com. IN A
;; ANSWER SECTION:
www.google.com. 585119 IN CNAME www.l.google.com.
www.l.google.com. 69 IN A 216.239.51.99
www.l.google.com. 69 IN A 216.239.51.104
;; AUTHORITY SECTION:
l.google.com. 60734 IN NS g.l.google.com.
l.google.com. 60734 IN NS b.l.google.com.
l.google.com. 60734 IN NS f.l.google.com.
l.google.com. 60734 IN NS c.l.google.com.
l.google.com. 60734 IN NS a.l.google.com.
l.google.com. 60734 IN NS e.l.google.com.
l.google.com. 60734 IN NS d.l.google.com.
;; Query time: 2 msec
;; SERVER: 10.5.50.184#53(10.5.50.184)
;; WHEN: Sun Feb 3 09:51:46 2008
;; MSG SIZE rcvd: 196
b) host
bash-2.05b# host www.google.com
www.google.com is an alias for www.l.google.com.
www.l.google.com has address 216.239.51.104
www.l.google.com has address 216.239.51.99
c) nslookup
bash-2.05b# nslookup www.google.com
Server: 10.5.50.184
Address: 10.5.50.184#53
Non-authoritative answer:
www.google.com canonical name = www.l.google.com.
Name: www.l.google.com
Address: 216.239.51.99
Name: www.l.google.com
Address: 216.239.51.104
2) List all network interfaces, the below example list two interfaces eth0 and eth1 and a loopback.
infconfig -a
bash-2.05b# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:11:43:32:49:99
inet addr:10.2.150.74 Bcast:10.5.50.127 Mask:255.255.255.192
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:357854361 errors:0 dropped:0 overruns:0 frame:0
TX packets:383621806 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3168219417 (3021.4 Mb) TX bytes:3724305382 (3551.7 Mb)
Interrupt:17
eth1 Link encap:Ethernet HWaddr 00:11:43:32:49:9A
inet addr:10.5.154.138 Bcast:10.5.54.127 Mask:255.255.255.128
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:18
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:92809439 errors:0 dropped:0 overruns:0 frame:0
TX packets:92809439 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1454141744 (1386.7 Mb) TX bytes:1454141744 (1386.7 Mb)
3) To find more information about a network interface, the below example shows detail information about interface eth0.
bash-2.05b# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: Not reported
Advertised auto-negotiation: No
Speed: 100Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Link detected: yes
File related commands
----------------------
1) find text in the matching files and show the filename with the pattern. The below example shows the list of filenames and the location of the hostname in those files, which will be helpful when changing hostnames for your websphere installation.
find . -name '*.xml' -print | xargs grep 'hostName' /dev/null
bash-2.05b# find /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/ -name 'server*.xml' -print | xargs grep 'hostName' /dev/null | more
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/portalserverNode01/serverindex.xml:
.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="portalserver.example.com" endPointRefs="NamedEndPoint_1130375480667 NamedE
ndPoint_1130375480668 NamedEndPoint_1130375480669 NamedEndPoint_1130375480670 NamedEndPoint_1130375480671 NamedEndPoint_113037548067
2 NamedEndPoint_1130375480673">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2034indchn2.example.com/serverindex.xml:<
serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserv
er/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1130386859873" hostName="n2034indchn2.example.com">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2033indchn2.example.com/serverindex.xml:<
serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserv
er/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1130361145826" hostName="n2033indchn2.example.com">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/portalserverCellManager01/serverindex.xml:
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2011indchn2Node01/serverindex.xml:
.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="n2011indchn2.example.com" endPointRefs="NamedEndPoint_1130375717654 NamedE
ndPoint_1130375717655 NamedEndPoint_1130375717656 NamedEndPoint_1130375717657 NamedEndPoint_1130375717658 NamedEndPoint_113037571765
9 NamedEndPoint_1130375717660">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2033indchn2UnNode01/serverindex.xml:
/5.0/serverindex.xmi" xmi:id="ServerIndex_1173388344070"hostName="n2033indchn2.example.com">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2034indchn2UnNode01/serverindex.xml:
/5.0/serverindex.xmi" xmi:id="ServerIndex_1173388421900" hostName="n2034indchn2.example.com">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/templates/servertypes/APPLICATION_SERVER/serverindex.xml:
erindex.xmi" xmi:id="ServerIndex_1" hostName="$(node.host.name)">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/backup/base/cells/portalserverNode01Cell/nodes/portalserverNode01/serverindex.x
ml:
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/backup/base/templates/servertypes/APPLICATION_SERVER/serverindex.xml:
mas/5.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="$(node.host.name)">
2) Soemtimes you might run out of diskspace and want to clean up some files, so this command will help you find all large files. In this example it will list all the files greater than 10 mb in size.
find / -xdev -size +1024 -exec ls -al {} \; | sort -r -k 5
bash-2.05b# find /usr/IBM/WebSphere -xdev -size +1024 -exec ls -al {} \; | sort -r -k 5-rw-r--r--
1 root root 1047553 Oct 26 2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node01/n2110indch2_server1/transaction/tranlog/log1
-rw-r--r-- 1 root root 1047553 Oct 26 2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node01/n2110indch2_server1/transaction/partnerlog/log2
-rw-r--r-- 1 root root 1047553 Oct 26 2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node01/n2110indch2_server1/transaction/partnerlog/log1
3) When sending SystemOut.log or SystemErr.log or trace.log to your developers or to IBM support Please use Tar and gzip on the fly to reduce the size of the file and the transfer time to FTP,email or http upload.
To pack on the fly...
tar -cvf - FILE-LIST | gzip -c > FILE.tar.gz
To unpack on the fly..
gunzip < FILE.tar.gz | tar -xvf -
To pack on the fly and email the file to IBM Support
tar -cvf - SystemOut.log | gzip -c | uuencode SystemOut.tar.gz | mail -s "PMR 09921,227,000" -b "SystemOut.log for server crash " WebSphere_Support@mainz.ibm.com
To find selective files and tar it up on the fly...
find /opt/IBM/WebSphere/PortalServer/log -name 'System*.log'| xargs tar -rvf System_logs.tar
4) Sometimes you might encounter a situation for example when running xmlaccess or wsadmin where you might want to write stdout to the file as well as to the console so that you have a copy even when the standard output disappears because of the screen buffer size.
ls -l | tee > filename
xmlaccess.sh -in ExportRelease.xml -username wpsadmin -password wpsadmin -url http://localhost:9080/wps/config | tee filename
./wsadmin.sh | tee wsadmin.log
5) find files that is recently updated which might be useful to see what changes are made to the system recently (e.g) below command shows the files that are modified within the last 20 minutes(-mmin) and 20 days (-mtime)
find /usr/IBM -mmin -20
find /usr/IBM -mtime -20
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/dmgr/SystemOut.log
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/ffdc/dmgr_exception.log
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/activity.log
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer0.ser
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer1.ser
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer2.ser
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/logs
6) Find a memory usage of the system.
free
cat /proc/meminfo
bash-2.05b# free
total used free shared buffers cached
Mem: 11819592 4938416 6881176 0 298148 1440200
-/+ buffers/cache: 3200068 8619524
Swap: 2048276 0 2048276
bash-2.05b# cat /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 12103262208 5056954368 7046307840 0 305303552 1474797568
Swap: 2097434624 0 2097434624
MemTotal: 11819592 kB
MemFree: 6881160 kB
MemShared: 0 kB
Buffers: 298148 kB
Cached: 1440232 kB
SwapCached: 0 kB
Active: 3718616 kB
ActiveAnon: 2968776 kB
ActiveCache: 749840 kB
Inact_dirty: 789328 kB
Inact_laundry: 237596 kB
Inact_clean: 0 kB
Inact_target: 949108 kB
HighTotal: 11140992 kB
HighFree: 6681628 kB
LowTotal: 678600 kB
LowFree: 199532 kB
SwapTotal: 2048276 kB
SwapFree: 2048276 kB
Committed_AS: 6039100 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB
4) To repeat a command contimuously.
watch -d=10 -n 1 "netstat -a"
Every 1s: netstat -a| grep ESTABLISHED Sun Feb 3 21:48:04 2008
tcp 0 0 n1212indchn2.www.com:9900 n1312indchn2.www.com:11530 ESTABLISHED
tcp 0 0 n1212indchn2.www.com:9353 n1312indchn2.www.com:11549 ESTABLISHED
tcp 0 0 n1212indchn2.www.com:9354 n1312indchn2.www.com:11550 ESTABLISHED
tcp 0 0 n1212indchn2.www.com:9201 n1312indchn2.www.com:11544 ESTABLISHED
tcp 0 0 n1212indchn2.www.com:9352 n1312indchn2.www.com:11545 ESTABLISHED
SSL Certificates expiration monitoring for WebSphere or any java based application server using java keystore
If you are a WebSphere Administrator or any Application Server (WebLogic, Tomcat) administrator you might already know that managing the ssl certificates in a large complex environments becomes hectic and troublesome because of the different expiration dates of the certificates that websphere uses and also the ssl certificates of the external systems (like SAP , Siebel) that websphere applicaton server might connect to using a secure connection, multiple administrators in an organization renewing it and not keeping tracking of the expiration dates. The problem is , SSL certificate might expire resulting in servers will not initialize and your running servers will stop operating, becomes unresposive if not properly renewed on time. Hence this article will explain how you can monitor the expiration of the ssl certificates using a simple command and proactively monitor the expiration dates, setup your calendar and renew it on time to prevent any downtimes.
If you are using the latest version (as on Dec 2007) of WebSphere Application Server 6.1 and using default self signed there are provisions in the application server itself to effectively monitor the certificate expiration, notifying you and renewing it automatically. If you are using a perosnal self signed certificate or a personal certificate signed by a certificate authority (CA) like VeriSign or Thawte , which is most likely in all of your production environments then you might want to use the below command to find the expiration date and renew it.
If you are using a java keystore ( DummyServerKeyFile.jks , DummyServerTrustFile.jks , jssecerts , cacerts) use the below find command along with keytool or websphere's ikeyman command line option to find the expiration of the keys.
LINUX:
Using Ikeyman cmdline utility (ikeyman.sh):
find PATH -name *.jks | xargs -i bash -c '$WAS_HOME/java/bin/java -classpath $WAS_HOME/java/jre/lib/ext/ibmjceprovider.jar:$WAS_HOME/AppServer/java/jre/lib/ext/ibmjcefw.jar:$WAS_HOME/AppServer/java/jre/lib/ext/US_export_policy.jar:$WAS_HOME/AppServer/java/jre/lib/ext/local_policy.jar:$WAS_HOME/AppServer/java/jre/lib/ext/ibmpkcs.jar:$WAS_HOME/AppServer/java/jre/lib/ext com.ibm.gsk.ikeyman.ikeycmd -cert -list all -expiry -db {} -type jks -pw
(e.g)
find /usr/IBM/WebSphere/AppServer/profiles/ -name *.jks | xargs -i bash -c '/usr/IBM/WebSphere/AppServer/java/bin/java -classpath /usr/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmjceprovider.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmjcefw.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext/US_export_policy.jar:/usr/IBM/WebSphere/AppServer//java/jre/lib/ext/local_policy.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmpkcs.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext com.ibm.gsk.ikeyman.ikeycmd -cert -list all -expiry -db {} -type jks -pw WebAS | grep "Not After:.*200[7,8]" -B 3 && echo {}'
Using Java Keytool utility:
find $WAS_HOME -name '*.jks' | xargs -i bash -c "echo {}; $WAS_HOME/java/jre/bin/keytool -list -v -keystore {} -storepass password | grep 'until: .*/.*/Y[Y,Y]' -A 3 -B 7"
(e.g)
find /usr/IBM/WebSphere/AppServer -name '*.jks' | xargs -i bash -c "echo {};/usr/IBM/WebSphere/AppServer/java/jre/bin/keytool -list -v -keystore {} -storepass WebAS | grep 'until: .*/.*/0[7,8]' -A 3 -B 7"
Output:
/usr/IBM/WebSphere/AppServer/profiles/etc//WWWStageClientKey.jks
Alias name: verisign class 1 ca individual subscriber-persona not validated
Creation date: Dec 18, 2006
Entry type: trustedCertEntry
Owner: CN=VeriSign Class 1 CA Individual Subscriber-Persona Not Validated, OU="www.verisign.com/repository/RPA Incorp. By Ref.,LIAB.LTD(c)98", OU=VeriSign Trust Network, O="VeriSign, Inc."
Issuer: OU=Class 1 Public Primary Certification Authority, O="VeriSign, Inc.", C=US
Serial number: d8b4feeaad218df5bf4756a9d29e17ffb
Valid from: 5/12/98 12:00 AM until: 5/12/08 11:59 PM
Certificate fingerprints:
MD5: CA:66:3C:FC:71:2B:BA:41:92:71:DD:72:AD:E5:65:65
SHA1: 12:51:9A:E9:CD:77:7A:56:01:84:F1:FB:D5:42:15:22:2E:95:E7:1F
--
Creation date: Dec 18, 2006
Entry type: keyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=wadm.stage.WWW.com, OU=IT, O=WWW, L=Fremont, ST=CA, POSTALCODE=94089, C=US
Issuer: CN=wadm.stage.WWW.com, OU=IT, O=WWW, L=Fremont, ST=CA, POSTALCODE=94089, C=US
Serial number: 458709ef
Valid from: 12/18/06 9:36 PM until: 12/18/07 9:36 PM
Certificate fingerprints:
MD5: C2:4F:CC:SS:19:DC:E4:88:B0:2A:78:76:69:4D:DC:EF:47
SHA1: 53:AB:D0:13:45:45:12:59:64:F6:C9:38:41:F2:C8:E3:37:05:73:95:F3:83
/usr/IBM/WebSphere/AppServer/profiles/etc/wwwStageServerKey.jks
Alias name: www stage websphere server ca
Creation date: Dec 8, 2006
Entry type: keyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=wadm.stage.www.com, OU=IT, O="www, Inc.", L=Fremont , ST=California, C=US
Issuer: OU=Secure Server Certification Authority, O="RSA Data Security, Inc.", C=US
Serial number: 1ea5f21b33ca65e551440d662399dffdfga382
Valid from: 12/5/06 12:00 AM until: 12/15/07 11:59 PM
Certificate fingerprints:
MD5: C2:22:E4:12:12:6D:3F:22:67:4E:37:48:12:3F:F3:1B:FE:26
SHA1: CA:81:37:A7:E9:12:22:19:27:D9:3F:C5:79:E9:25:C8:1A:3C:1E:5C:17:02
As you can see in the above output the command will list the file name (/usr/IBM/WebSphere/AppServer/profiles/etc/wwwStageServerKey.jks) , certificate label (www stage websphere server ca) , Entry Type (keyEntry - personal cert or trustedCertEntry - Signer Cert), and the expiration date (12/5/06 12:00 AM until: 12/15/07 11:59 PM) of
all the java keystore (*.jks) in the given directory that expired in 2007 (07) or 2008 (08). By which you shoule be able to indentify the certs, setup your calendat atleast 10 days before the expiration and renew it to proactively to preveny any problems.
Note you can also change the above command by passing differnt filename like cacerts and jssecerts and also pass different year infomration to find expirations during those years.
Sometimes you might get no output which means either there are no expiring ceritificates for the years that you passed or if you are expecting any expiring certificate to show and if not then the password for the keystore might be wrong, so run without the grep portion in the above command which will list all the certificates irrespective of the expiration date or it will complain if the password is wrong as below.
find /usr/IBM/WebSphere/AppServer -name '*.jks' | xargs -i bash -c "echo {};/usr/IBM/WebSphere/AppServer/java/jre/bin/keytool -list -v -keystore {} -storepass WebAS
/usr/IBM/WebSphere/AppServer/profiles/etc/wwwStageServerKey.jks
java.io.IOException: Keystore was tampered with, or password was incorrect
Also you can use the same command by modifying the -type parameter to "-type cms" for the webserver such as IBM HttpServer where it's using cms database ( kdb file).
find /usr/IBM/HttpServer -name *.kdb | xargs -i bash -c '/usr/IBM/WebSphere/Plugins/java/bin/java -classpath /usr/IBM/WebSphere/Plugins/java/jre/lib/ext/ibmjceprovider.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext/ibmjcefw.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext/US_export_policy.jar:/usr/IBM/WebSphere/Plugins//java/jre/lib/ext/local_policy.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext/ibmpkcs.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext com.ibm.gsk.ikeyman.ikeycmd -cert -list all -expiry -db {} -type cms -pw WebAS | grep "Not After:.*200[7,8]" -B 3 && echo {}'
Posted by onewebclick at 10:23 AM 1 comments
Labels: certificates, DummyServerKeyFile.jks, DummyServerTrustFile.jks, expiration, httpserver, ibm, javax.net.ssl.SSLHandshakeException | certificate expired, renewal, self signed certificate, ssl, websphere
Friday, December 7, 2007
How to identify running WebSphere process as Dmgr or Nodeagent or Appserver process ?
WebSphere processes are nothing but java application processes started with a long cmdline as below, which contains classpath , command line parameters, the Main java class and the arguments passed to it.
(e.g) Example of a websphere dmgr process in Linux
root 20308 0.2 2.6 450200 331496 ? S Dec06 3:39 /usr/IBM/WebSphere/AppServer/java/bin/java
-Xbootclasspath/p:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/usr/IBM/WebSph
ere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=62306 -classpath /usr/IBM/WebS
phere/AppServer/profiles/Dmgr02/properties:/usr/IBM/WebSphere/AppServer/properties:/usr/
IBM/WebSphere/AppServer/lib/bootstrap.jar:/usr/IBM/WebSphere/AppServer/lib/j2ee.jar:/usr/IB
M/WebSphere/AppServer/lib/lmproxy.jar:/usr/IBM/WebSphere/AppServer/lib/urlprotocols.jar -X
ms50m -Xmx256m -Dws.ext.dirs=/usr/IBM/WebSphere/AppServer/CEI/lib:/usr/IBM/WebSphere/A
ppServer/CEI/client:/usr/IBM/WebSphere/AppServer/java/lib:/usr/IBM/WebSphere/AppServer/pr
ofiles/Dmgr02/classes:/usr/IBM/WebSphere/AppServer/classes:/usr/IBM/WebSphere/AppServer
/lib:/usr/IBM/WebSphere/AppServer/installedChannels:/usr/IBM/WebSphere/AppServer/lib/ext:/u
sr/IBM/WebSphere/AppServer/web/help:/usr/IBM/WebSphere/AppServer/deploytool/itp/plugins/c
om.ibm.etools.ejbdeploy/runtime -Dderby.system.home=/usr/IBM/WebSphere/AppServer/derby -
Dcom.ibm.itp.location=/usr/IBM/WebSphere/AppServer/bin -Djava.util.logging.configureByServer
=true -Dibm.websphere.preload.classes=true -Duser.install.root=/usr/IBM/WebSphere/AppServer/
profiles/Dmgr02 -Dwas.install.root=/usr/IBM/WebSphere/AppServer -Djava.util.logging.manage
r=com.ibm.ws.bootstrap.WsLogManager -Ddb2j.system.home=/usr/IBM/WebSphere/AppServer
/cloudscape -Dserver.root=/usr/IBM/WebSphere/AppServer/profiles/Dmgr02 -Djava.awt.headless
=true -Djava.security.auth.login.config=/usr/IBM/WebSphere/AppServer/profiles/Dmgr02/properti
es/wsjaas.conf -Djava.security.policy=/usr/IBM/WebSphere/AppServer/profiles/Dmgr02/propertie
s/server.policy com.ibm.ws.bootstrap.WSLauncher com.ibm.ws.runtime.WsServer /usr/IBM/WebS
phere/AppServer/profiles/Dmgr02/config xx181uspln2Cell01 xx181uspln2CellManager01 dmgr
The normal process to list a process in Unix environment is to run ps -ef and grep the name of the process, if you now the name.(e.g) ps -ef | grep java. The issue is that by default the OS will show 80 characters of the process string or to the widhth of the terminal which is anyway 80 by default in most of the terminals, hence if you try to list the websphere java process where different process types like dmgr, nodeagent, appserver all running in the same box you would pretty much get the same string as there won't be any noticable difference because the classpath, etc would be pretty much same for the all process. Hence we need to use the following ps command with addtional parameters for to indentify the respective websphere process.
To determine if Deployment Manager (dmgr) is running ,
AIX : ps -auxww | grep dmgr
LINUX : ps -auxww | grep dmgr
SOLARIS: /usr/ucb/ps -auxww | grep dmgr (Note you need to run specifically from this directory , otherwise you will invoke ps from /usr/bin which will not like the arguments )
(e.g)
root 20308 0.2 2.6 450200 331496 ? S Dec06 3:39 /usr/IBM/WebSphere/AppServer/java/bin/java
-Xbootclasspath/p:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/usr/IBM/WebSph
ere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=62306 -classpath /usr/IBM/WebS
phere/AppServer/profiles/Dmgr02/properties:/usr/IBM/WebSphere/AppServer/properties:/usr/
IBM/WebSphere/AppServer/lib/bootstrap.jar:/usr/IBM/WebSphere/AppServer/lib/j2ee.jar:/usr/IB
M/WebSphere/AppServer/lib/lmproxy.jar:/usr/IBM/WebSphere/AppServer/lib/urlprotocols.jar -X
ms50m -Xmx256m -Dws.ext.dirs=/usr/IBM/WebSphere/AppServer/CEI/lib:/usr/IBM/WebSphere/A
ppServer/CEI/client:/usr/IBM/WebSphere/AppServer/java/lib:/usr/IBM/WebSphere/AppServer/pr
ofiles/Dmgr02/classes:/usr/IBM/WebSphere/AppServer/classes:/usr/IBM/WebSphere/AppServer
/lib:/usr/IBM/WebSphere/AppServer/installedChannels:/usr/IBM/WebSphere/AppServer/lib/ext:/u
sr/IBM/WebSphere/AppServer/web/help:/usr/IBM/WebSphere/AppServer/deploytool/itp/plugins/c
om.ibm.etools.ejbdeploy/runtime -Dderby.system.home=/usr/IBM/WebSphere/AppServer/derby -
Dcom.ibm.itp.location=/usr/IBM/WebSphere/AppServer/bin -Djava.util.logging.configureByServer
=true -Dibm.websphere.preload.classes=true -Duser.install.root=/usr/IBM/WebSphere/AppServer/
profiles/Dmgr02 -Dwas.install.root=/usr/IBM/WebSphere/AppServer -Djava.util.logging.manage
r=com.ibm.ws.bootstrap.WsLogManager -Ddb2j.system.home=/usr/IBM/WebSphere/AppServer
/cloudscape -Dserver.root=/usr/IBM/WebSphere/AppServer/profiles/Dmgr02 -Djava.awt.headless
=true -Djava.security.auth.login.config=/usr/IBM/WebSphere/AppServer/profiles/Dmgr02/properti
es/wsjaas.conf -Djava.security.policy=/usr/IBM/WebSphere/AppServer/profiles/Dmgr02/propertie
s/server.policy com.ibm.ws.bootstrap.WSLauncher com.ibm.ws.runtime.WsServer /usr/IBM/WebS
phere/AppServer/profiles/Dmgr02/config xx181uspln2Cell01 xx181uspln2CellManager01 dmgr
Simlarly you can find for other types of WebSphere processes:
To determine if Nodeagent is running ,
AIX : ps -auxww | grep nodeagent
LINUX : ps -auxww | grep nodeagent
SOLARIS : /usr/ucb/ps -auxww | grep nodegent (Note you need to run specifically from this directory , otherwise you will invoke ps from /usr/bin which will not like the arguments )
To determine if AppServer is running ,
AIX : ps -auxww | grep
LINUX : ps -auxww | grep
SOLARIS: /usr/ucb/ps -auxww | grep
(e.g) ps -auxww | grep server1 - to list the process of websphere Applicatoin Server named server1.
(e.g) ps -auxww | grep WsServer - to list all the process of the websphere installation.
Steps involved in moving WebSphere dmgr from one host (machine1) to another host (machine2) with the hostname change :
1) As a caution make please make a backup of the following so that it's easy to restore when something goes wrong,
a) Backup the configuration of all the profiles (DMGR) and (AppSrv) in machine1 that is involved in the cell .
(eg) basically run WAS_ROOT/bin/backupconfig.sh which will create WebSphereConfig_2007-11-16.zip
b) (Optional) Also do a filesystem backup of the directory if possible to avoid any surprises.
2) Install WebSphere ND in the new box and create a new Dmgr profile with machine2_dmgr_profile .
3) Extract the WebSphereConfig_2007-11-16.zip to
(eg) jar -xvf WebSphereConfig_2007-11-16.zip
4) if
(eg)
5) Change the following properties in
(eg) WAS_CELL=machine1Cell01
WAS_NODE=machine1CellManager01
6) Copy the custom keyfiles (*.jks) from
7) Follow the instructions mentioned in http://www-1.ibm.com/support/docview.wss?rs=180&context=SSEQTP&q1=best+practices&uid=swg27007419&loc=en_US&cs=utf-8&lang=en from page 4-6 on section 2 and 2.1
8) check if syncs works for all the nodes and you were able to see all the configurations from the previous dmgr.
Forcing HTTP Download
To force HTTP download of a dynamically generated content, I usually set the HTTP header Content-Type to application/octet-stream. This forces the browser to display the Save dialog box. But this has the limitation of sending the wrong content-type even when we know the correct one. Recently I discovered another HTTP header which solves this problem. This is the Content-Disposition header. This can take following two vales:
1. inline: This will render the content inline in the browser.
2. attachment: This will force the browser to display the Save dialog.
When generating dynamic content, it is also recommended to specify proper filename. This file name can also be specified as a parameter to Content-Disposition header. An example:
Content-Disposition: attachment;filename=document.pdf
Content-Disposition is covered in RFC 2183.
