How To Resolve Server Hang, Memory Leakage and Out Of Memory Issues

Below are the steps and tips that a production support engineer keep in mind :

• Gather memory leakage data by enabling GC verbose on deployment servers.

• If Application is in JAVA, then you make sure JSP page are with pre-compilation.

• For Web application, Check the deployment descriptor entries.

• HTTP session duration settings, Check for Auto Time out settings.

• Check JDBC Connection Pool Setting like initial and max no of connection with database with no by which it can increase.

• Look for code how developer is handing JDBC connection.

• Optimize Heap Size as per application load.

• High CPU utilization - This is the condition when a process is taking high portion of CPU. You may check that by top command of unix. you may kill it, if that is abnormal and not required to run.

• Monitor Garbage Collection

• Define Separate files for different logs

• Use Application log frameworks like log4j

• Use separate log files for Garbage collection per JVM.

• OS Chunk pool size

• OS Connection backlog buffering

• Use OS performance packs

• Setting OS TCP-IP parameters

• tcp_time_wait_interval

• tcp_conn_req_max_q

Application Production Support Issues Part -1

We are trying to list down all possible reasons that can be the root cause of below issues listed for Application Production Support domain. These are the basics that will be asked in each interview as well as a production support engineer needs to be familiar interact with these issues.


Thread Dump

• On Linux/UNIX system just the below command to get JAVA thread dump

                    $     kill -3   <PID of JVM>

•  In Case IBM WAS server -

            Use DMGR console  >> Troubleshooting >> Select JVM and click core

• Oracle Weblogic Server -

          Using wlsadmin.sh   script   or JSTACK utility  

JAVA Out Of Memory

• Low JAVA Heap Size on server
• Object Slicing too long, like HTTP sessions
• Memory leakage in application code
• Full garbage collection is not happening due to JVM bug.

Server Hang issue 

Various reason can attract the server hang condition that may be called a deadlock condition where system will behave wrong or even does not respond to user queries at all.

• Memory leakage in application code
• Database query is taking long to time to return the output 
• Deadlock Condition
• Network latency
• Out Of Memory Error

_________________________________________________________________________________

IBM WebSphere Introduction

What is IBM WebSphere ?


This is Application server built on Java and provides the Middleware and MQ support in enterprise application development in IT software field. It is large app server in market that is used in Enterprise service bus or service oriented architecture in various organization. It is called as WAS in short.



Below are the some info to remember about WebSphere


WAS Editions

• WAS Network Deployment
• WAS Developers
• WAS Z/OS
• WAS Hypervisor 
• WAS Express
• WAS Base
• WAS Liberty Core

WAS default Ports ?

HTTP /HTTPS     -    9080 / 9443

How To Check WAS Version

Go to profile and bin folder of WAS installation on server box and execute below scripts

$    ./versioninfo.sh




More info at IBM WebSphere Home
_________________________________________________________________________________

Oracle Weblogic Server Starting Status in logs

Different status message When a user starts a weblogic server instance 


1. STARTING

2. STANDBY

3. STARTING

4. ADMIN

5. RESUMING

6. RUNNING



When Weblogic server admin starts the weblogic server from domain home, he may check the status of admin server in AdminServer.log file.

Start service on Operating system on boot time - chkconfig/systemctl command

These are Linux commands to deal with services running for different apps on operating system. By the use of systemctl and chkconfig command, we can start and stop the services as well as configure them for auto start/stop in unix.


Basically above commands or tools allows you to configure automatic service start stop in the /etc/init.d scripts available on operating systems. Until RHEL 6.x, "chkconfig " command was available and later RHEL 7, the command "systemctl" introduced for auto service start/stop. although all legacy commands may support in new versions of RHEL and CentOS.


List All Services

  By Chkconfig command, list all the services on

[root@aps ~]# chkconfig  --list
httpd           0:off   1:off   2:on    3:on    4:on    5:on    6:off
rhnsd           0:off   1:off   2:off   3:on    4:off   5:on    6:off
netconsole      0:off   1:off   2:off   3:on    4:off   5:on    6:off
...

Above snippet shows different run level by numbering and post fixes.

By Systemctl, all active services and all enabled/disabled services

[root@aps ~]# systemctl list-units --type service

[root@aps ~]# systemctl list-unit-files --type service

Check Specific Service Status

[root@aps ~]# chkconfig --list | grep httpd
httpd           0:off   1:off   2:off   3:on   4:on   5:on   6:off

[root@aps ~]# systemctl list-units --type service | grep httpd

Stop Specific Service on Run Levels

Below command will stop httpd service at run level 2,3,4,5.

[root@aps ~]# chkconfig --level 2345 httpd off

Start/Stop Specific Service by systemctl

[root@aps ~]# systemctl stop [service_name]

[root@aps ~]# systemctl start [service_name]

Check Status of a Service by systemctl

[root@aps ~]# systemctl status [service_name]

How to Auto Start a Service at Boot of Operating System

By chkconfig command, below will stop httpd service at run level 4,5.

[root@aps ~]# chkconfig --level 45 httpd off

By systemctl command, you can enable any of service to start on boot time as below

[root@aps ~]# systemctl enable [service_name]

[root@aps ~]# systemctl disable [service_name]

_________________________________________________________________________________

Remove 7 days older log files from Unix server

This is the basic task in application support that requires log files to remove, back up to other location that are older than "N" days or let suppose 7 days from system date.

Sometimes requirement may ask you to run this task automatically on specified day, date and time to check on system file system, then you may use below commands in cron schedule on UNIX by making a script and set with schedule on crontab.

• find the log files that are older than "n" days and remove all ? write the command plz....

           $ find   <path-to-check>  -name *.log -mtime  +n   -exec rm -rf {}\;

          Here <path-to-check> is server location or path to check and "+n" shows days.   * is wild card            to check all log files having extension .log

          Example :

          $ find   /tmp/logs  -name *.log -mtime  +7   -exec rm -rf {}\;

          Above will remove all log files older than 7 days.

Apache Web Server basic support Questions

• Start & Stop of Apache web Server

             $  ./apachectl stop

             $  ./apachectl start 

              you may use scripts located at /etc/init.d/ mostly it will be "apache" or "httpd"

             $  /etc/init.d/apache   -start

             $  /etc/init.d/apache   -stop

                                   

                                                     "OR"

             $  /etc/init.d/httpd   -start

             $  /etc/init.d/httpd   -stop

• Start/stop Apache web server as a OS service

             $ httpd stop

             $ service httpd start

• Default Ports for Apache

              HTTP/HTTPS   -    80/443

• How to check version of Apache web server

            $  ./httpd  -v          (in bin folder of installation)

            OR -    rpm -qa | grep httpd

• How to check listen address of Apache 

          Open httpd.conf file and search for "Listen" string it will tell you the listen IP address and port

          Example :        Listen   10.10.10.10:80

• Apache Web Server main configuration file

            httpd.conf        

• How to check Apache web server is running ?

           by checking running process on Unix,    $ ps -aef | grep "httpd"

________________________________________________________________________________