Rotate Access logs in Apache

Rotatelogs is a simple program for use in conjunction with Apache's piped log file feature. It supports rotation based on a time interval or maximum size of the log.

Using the module mod_log_rotate, the log rotation is handled by the server process so you save on the process count and file descriptors.

Download the mod_log_rotate.dll  or  mod_log_rotate.so files  in the following URL if your apache not having the modules.

http://www.sitebuddy.com/files/mod_log_rotate2055_08_2006.dll 

Rename it as mod_log_rotate.dll  and place the file in Modules Directory (D:\IBMHTTPServer\modules)

To load this module, modify your httpd.conf:
LoadModule log_rotate_module modules/mod_log_rotate.so
#or if using the DLL version
LoadModule log_rotate_module modules/mod_log_rotate.dll
#Add the following line to use the module

RotateLogs On

RotateLogsLocalTime On

RotateInterval 86400

CustomLog logs/access_%Y-%m-%d-%H_%M_%S.log combined

The directive for your individual sites custom logs, will not change:
CustomLog " access_%Y-%m-%d-%H_%M_%S.log "  combined
Your log file will be in the format: ex.log.1140048000
The numeral is the number of seconds since: 01-01-1970

To translate that number into a date, the equation is: 01-01-1970 + 1140048000 seconds.

mod_log_rotate directives

RotateLogs On|Off
Enable / disable automatic log rotation.
 Note: On Apache 2, once enabled mod_log_rotate takes responsibility for all log output server wide even if RotateLogs Off is subsequently used. That means that the BufferedLogs directive that is implemented by mod_log_config will be ignored. As BufferedLogs isn’t document and is flagged as an experimental feature this shouldn’t be a problem in production environments. This doesn’t apply to the Apache 1.3 version of the module.
RotateLogsLocalTime On|Off
Normally the log rotation interval is based on UTC. For example an interval of 86400 (one day) will cause the logs to rotate at UTC 00:00. When this option is on, log rotation is timed relative to the local time.
RotateInterval <interval> [<offset>]
Set the interval in seconds for log rotation. The default is 86400 (one day). The shortest interval that can be specified is 60 seconds. An optional second argument specifies an offset in minutes which is applied to UTC (or local time if RotateLogsLocalTime is on). For example RotateInterval 86400 60 will cause logs to be rotated at 23:00 UTC.

Manually Replacing SSL Certificates

This document describes the steps necessary to replace the certificates in IBM WebSphere Application Server V6.1 when the certificates have expired or if the nodes are out of sync.

NOTE: This document assumes that you are using a default configuration. If you have made modifications to your SSL configurations you will need to take these changes into account. For example, additional steps will be required if you have enabled client authentication on the application servers.
1. Run backupConfig on the Deployment Manager.
2. Stop all of the nodeagents and application servers in the cell. Stop the Web server(s). Start the Deployment Manager.
3. Replace the Deployment Manager certificate.
i. In the Administrative Console, go to Security > SSL certificate and key management > Key stores and certificates > CellDefaultKeyStore > Personal certificates > Create a self-signed certificate
ii. Enter the required attributes.
Alias : cell_default Common name : <hostname> Validity period : <number of days> <-- this can be set greater than 365 Organization : <company> Click OK and Save the changes. iii. Return to Security > SSL certificate and key management > Key stores and certificates > CellDefaultKeyStore > Personal certificates iv. Select the old certificate and click Replace. v. On the next screen, you are able to choose which certificate will replace the old certificate. Accept your new certificate. Do not select either Delete old certificate after replacement or Delete old signers. Accept your new certificate and any browser prompts. vi. On the next screen, select the old certificate and click Delete. Click OK and Save the changes. At this point the Deployment Manager has its certificate replaced.
4. Add the Deployment Manager signer certificate to the CellDefaultTruststore.
i. Go to SSL certificate and key management > Key stores and certificates. ii. Select CellDefaultKeyStore and CellDefaultTrustStore and click Exchange signers.
iii. Select the certificate in CellDefaultKeyStore personal certificates created in previous step and click Add. Click OK and Save the changes.
5. Replace the certificate on the node(s). This step will need to be done for each node in the cell.
i. Go to Security > SSL certificate and key management > Manage endpoint security configurations. ii. Under Inbound, click the link for the node, node_name(NodeDefaultSSLSettings,null). iii. Click the Manage certificates button.
iv. Click Create a self-signed certificate. v. Enter the required attributes. Alias : nodeX_default <-- where X is the node number Common name : <hostname> Validity period : <number of days> <-- this can be set greater than 365 Organization : <company> Click OK and Save the changes. vi. Return to Security > SSL certificate and key management > Manage endpoint security configurations, click node_name(NodeDefaultSSLSettings,null), click Manage certificates. vii. Select the old certificate and click Replace. viii. On the next screen, you are able to choose which certificate will replace the old certificate. Accept your new certificate. Do not select either Delete old certificate after replacement or Delete old signers. ix. On the next screen, select the old certificate and click Delete. Click OK and save the changes.
6. Add the Node signer certificate to the CellDefaultTruststore. This step will need to be done for each node in the cell.
i. Go to Security > SSL certificate and key management > Manage endpoint security configurations. ii. Under Inbound, click the link for the node, node_name(NodeDefaultSSLSettings,null) and select Key stores and certificates. iii. Select NodeDefaultKeyStore and CellDefaultTrustStore and then Click Exchange signers. iv. Select the certificate in NodeDefaultKeyStore personal certificates created in previous step and click Add.
Click OK and Save the changes.
7. Repeat steps 5 and 6 for each node in the cell. 8. Delete the old signer certificates and extract the new ones.
i. Go to SSL certificate and key management > Key stores and certificates > CellDefaultTrustStore > Signer certificates ii. Select all of the old signer certificates and click Delete. If you are not sure, you can compare the Fingerprint and/or the Expiration dates with the personal certificate in the keystores. iii. Select one of the new certificates. Click Extract. iv. Enter a File Name that corresponds to the certificate. For example, node1.arm. Click Ok. v. Repeat iii. and iv. for each of the new certificates making sure you have done this for the cell signer and all of the node signers. These files are saved to the profile_root/Dmgr/etc directory.
9. Manually copy the trust store to each of the /etc directories.
i. Backup the trust.p12 in profile_root\Dmgr\etc ii. Copy the profile_root\Dmgr\config\cells\cell-name\trust.p12 to profile_root\Dmgr\etc iii. Backup the trust.p12 on each of the nodes profile_root\Appsrv\etc directories. iv. Copy the profile_root\Dmgr\config\cells\cell-name\trust.p12 to profile_root\Appsrv\etc v. Repeat the previous step for each node in the cell.
10. Sync and Start the node(s).
i. Restart the Deployment Manager. ii. Run a command line syncNode from each of the nodes. iii. Start the nodeagents and application servers. They should now be fully synchronized with the new certificates in place.
11. Propagate the signer certificate(s) to plug-in(s).
i. Go to Servers > Web servers. Click webserver_name, then under Additional Properties click Plug-in properties.
IMPORTANT NOTE: Depending on your configuration you may or may not be able to perform the next 3 steps with the console. If the fields are greyed out and you are unable to manage your plugin-key.kdb from the console you will need to use IKEYMAN to manually add the certificates from step 8. iv. to the Web server plugin-key.kdb file and then continue at step 11 v. ii. Click Manage keys and certificates under Additional Properties, click Signer certificates and then click Add. iii. Enter a unique Alias Name and then specify the File Name that you created in step 8. iv. iv. Repeat this for each of the new certificates making sure you have done this for the cell signer and all of the node signers. v. Manually copy the plugin-key.kdb from the local configuration to the Web server. Default local configuration location: profile_root\Dmgr\config\cells\cell-name\nodes\node-name\servers\web-server-name\plugin-key.kdb Default Web server location: Web-server-root\Plugins\config\web-server-name\plugin-key.kdb Note: You can also determine the location from the Plug-in properties page in step i. vi. Repeat steps i. to v. for each Web server if you have more than one. vii. Start the Web server(s).

Hung threads

WebSphere Application Server monitors thread activity and performs diagnostic actions if one has become inactive. When WebSphere detects that a thread has been active longer than the time defined by the thread monitor threshold, the application server takes the following actions:

Logs a warning in the SystemOut log that indicates the name of the thread that is hung and how long it has already been active. The following message is written to the log:

WSVR0605W: Thread threadname has been active for hangtime and may be hung.  There are totalthreads  threads in total in the server that may be hung.

Example:

WSVR0605W: Thread "WebContainer : 4" (0000003b) has been active for 758692 milliseconds and may be hung.  There is/are 1 thread(s) in total in the server that may be hung.

If the work actually completes, a second set of messages, notifications and PMI events is produced to identify the false alarm. The following message is written to the log:

WSVR0606W: Thread threadname was previously reported to be hung but has completed. It was active for approximately hangtime. There are totalthreads threads in total in the server that still may be hung.

Example:

WSVR0606W: Thread "WebContainer : 5" (000000ca) was previously reported to be hung but has completed.  It was active for approximately 3599854 milliseconds.  There is/are 0 thread(s) in total in the server that still may be hung.

If the thread monitor determines that too many false alarms are issued (determined by the number of pairs of hang and clear messages), it can automatically adjust the threshold. When this adjustment occurs, the following message is written to the log:

WSVR0607W: Too many thread hangs have been falsely reported.  The hang

threshold is now being set to thresholdtime.

We can prevent WebSphere Application Server from automatically adjusting the hang time threshold.

Hang thread detection option is enabled by default. You can adjust the values of the detection policy or disable it by using the below procedure.

Navigate to Servers --> Applicaiton Servers --> server_name --> administration --> custom properties

Add the following 4 custom properties

#	Property	Description	Default
A	com.ibm.websphere.threadmonitor.interval	How frequently thread monitor should check all the managed threads for hung threads	180
B	com.ibm.websphere.threadmonitor.threshold	After how many seconds a thread can be considered as hung	600
C	com.ibm.websphere.threadmonitor.false.alarm.threshold	The number of times that false alarms can occur before automatically increasing the threshold (T)	100
D	com.ibm.websphere.threadmonitor.dump.java	when set to 'true', creates a java core when a hung thread is detected	false

        
Click OK and save the changes.  
Sync the changes and restart the servers for the changes to take effect.

Generating Heap dumps and Thread dumps

Automated heap dump generation

1. Click Servers > Application servers in the administrative console navigation tree.
2. Click server_name >Performance and Diagnostic Advisor Configuration.
3. Click the Runtime tab.
4. Select the Enable automatic heap dump collection check box.
5. Click OK.

Generating Heap dump Manually

set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]

Where server1 is the name of application server for which we want heapdump.

For generating heap dump:

$AdminControl invoke $jvm generateHeapDump

Generating Thread dump Manually

set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]

Where server1 is the name of application server for which we want heapdump.

For generating Thread dump:

$AdminControl invoke $jvm dumpThreads

OR

use kill -3 PID on unix/linux machines.

Automatically

There are some conditions where a thread dump is created automatically for your Java Virtual Machine (JVM), such as when WebSphere Application Server is stopped by some means other than a normal stop server request

 Thread dumps can also be triggered by issuing a signal to the WebSphere Application Server process. For example, to produce a thread dump in a UNIX environment, you can run this command:

kill -3 process_id

Heap Dumps and Java Dumps

Heap Dump: 
Heap dump is a text file which keeps record of all objects in the java heap. It contains the information like size and address of all objects, as well as the addresses of all objects that it references.
Heap dump is nothing but a snap shot of our JVM’s Heap memory,contains all active java objects and their activities which object doing what and how much memory it is occupying.
whenever OutOfMemoryException occurs Websphere application Server creates heap dump in profile's home directory.

Thread Dump: 

A Java dump, also known as a Java core, Java thread dump, or a thread dump is a file that contains the information about thread like active thread, hung thread, dead lock, runnable and inactive thread info.
The thread dump is a snapshot of exactly what's executing at a moment in time.


Thread dumps, or core files, are generated with names in this format:

javacore.date.time.id.txt

For example: javacore.20070919.204717.27050.txt