Monday, February 18, 2019

How To Resolve Server Hang, Memory Leakage and Out Of Memory Issues





Below are the steps and tips that a production support engineer keep in mind :


  • Gather memory leakage data by enabling GC verbose on deployment servers.
  • If Application is in JAVA, then you make sure JSP page are with pre-compilation.
  • For Web application, Check the deployment descriptor entries.
  • HTTP session duration settings, Check for Auto Time out settings.
  • Check JDBC Connection Pool Setting like initial and max no of connection with database with no by which it can increase.
  • Look for code how developer is handing JDBC connection.
  • Optimize Heap Size as per application load.
  • High CPU utilization - This is the condition when a process is taking high portion of CPU. You may check that by top command of unix. you may kill it, if that is abnormal and not required to run.
  • Monitor Garbage Collection
  • Define Separate files for different logs
  • Use Application log frameworks like log4j
  • Use separate log files for Garbage collection per JVM.
  • OS Chunk pool size
  • OS Connection backlog buffering
  • Use OS performance packs
  • Setting OS TCP-IP parameters
  • tcp_time_wait_interval
  • tcp_conn_req_max_q





Application Production Support Issues Part -1



We are trying to list down all possible reasons that can be the root cause of below issues listed for Application Production Support domain. These are the basics that will be asked in each interview as well as a production support engineer needs to be familiar interact with these issues.


Thread Dump


  • On Linux/UNIX system just the below command to get JAVA thread dump
                    $     kill -3   <PID of JVM>

  •  In Case IBM WAS server -
            Use DMGR console  >> Troubleshooting >> Select JVM and click core

  • Oracle Weblogic Server -
          Using wlsadmin.sh   script   or JSTACK utility  



JAVA Out Of Memory

  • Low JAVA Heap Size on server
  • Object Slicing too long, like HTTP sessions
  • Memory leakage in application code
  • Full garbage collection is not happening due to JVM bug.

Server Hang issue 

Various reason can attract the server hang condition that may be called a deadlock condition where system will behave wrong or even does not respond to user queries at all.

  • Memory leakage in application code
  • Database query is taking long to time to return the output 
  • Deadlock Condition
  • Network latency
  • Out Of Memory Error






_________________________________________________________________________________

IBM WebSphere Introduction


What is IBM WebSphere ?


This is Application server built on Java and provides the Middleware and MQ support in enterprise application development in IT software field. It is large app server in market that is used in Enterprise service bus or service oriented architecture in various organization. It is called as WAS in short.



Below are the some info to remember about WebSphere


WAS Editions

  • WAS Network Deployment
  • WAS Developers
  • WAS Z/OS
  • WAS Hypervisor 
  • WAS Express
  • WAS Base
  • WAS Liberty Core

WAS default Ports ?

HTTP /HTTPS     -    9080 / 9443



How To Check WAS Version

Go to profile and bin folder of WAS installation on server box and execute below scripts

$    ./versioninfo.sh




More info at IBM WebSphere Home
_________________________________________________________________________________


Oracle Weblogic Server Starting Status in logs



Different status message When a user starts a weblogic server instance 


1. STARTING

2. STANDBY

3. STARTING

4. ADMIN

5. RESUMING

6. RUNNING



When Weblogic server admin starts the weblogic server from domain home, he may check the status of admin server in AdminServer.log file.