Application Production Server Environment Setup

Application Production Server

Any software application is said to be in production when it is directly dealing with business either in Business to business or business to customer and even for internal business uses. And the application critical or responsible for business operation to work for profitability.


Any application before going to production server/environment goes out from various environments and levels like development, configuration/testing, SIT, Pre-Production. Once a application reaches to Pre-production or SIT environment, it will move to Production server after completing the user acceptance testing (UAT).



Strategically, Production environment setup can be done in various ways that depends on your requirement like organizational, application, geography, business nature and cost. we are discussing various approach here for production servers.



Standalone Setup

As the name depicts, in this setup application software has only one instance at all time means application logic is available at one place or hosted on one physical server. This approach generally used in small desktop application with low load of user traffic.

Standalone setup can have one hardware machine that can host operating system, database, application server and code logic.

The main disadvantage of this setup is, it does not provide you the facility of high availability and load balance. If the hardware failure occurs, application will not be available to serve.




Clustering Setup

The Cluster is a group of host servers on which your application logic is deployed. In this setup, the whole setup will be visible to client/user as a single or standalone setup where he will have advantages like High Availability and Load Balance.


As a cluster that is group of servers where if any hardware or network failure occurs due to any reason, then user is automatically transferred to other available server in group and the user will use the system as usual and will not face error or downtime during this switch over. This is called High Availability of servers.

How does a cluster work as a set-up point of view ?

• HOT - HOT Setup                     -    Both the Data Center takes traffic simultaneously.

• HOT - COLD Setup                  -    Only one Data Center takes traffic. if HOT DC goes down,                                                            the Cold DC will be manually made active to take the                                                                    traffic. 

• HOT - WARM Setup                 -    One DC takes traffic. if HOT DC goes down, WARM DC                                                             automatically comes up and starts taking traffic.

                              

_________________________________________________________________________________

How To Resolve Server Hang, Memory Leakage and Out Of Memory Issues

Below are the steps and tips that a production support engineer keep in mind :

• Gather memory leakage data by enabling GC verbose on deployment servers.

• If Application is in JAVA, then you make sure JSP page are with pre-compilation.

• For Web application, Check the deployment descriptor entries.

• HTTP session duration settings, Check for Auto Time out settings.

• Check JDBC Connection Pool Setting like initial and max no of connection with database with no by which it can increase.

• Look for code how developer is handing JDBC connection.

• Optimize Heap Size as per application load.

• High CPU utilization - This is the condition when a process is taking high portion of CPU. You may check that by top command of unix. you may kill it, if that is abnormal and not required to run.

• Monitor Garbage Collection

• Define Separate files for different logs

• Use Application log frameworks like log4j

• Use separate log files for Garbage collection per JVM.

• OS Chunk pool size

• OS Connection backlog buffering

• Use OS performance packs

• Setting OS TCP-IP parameters

• tcp_time_wait_interval

• tcp_conn_req_max_q

Application Production Support Issues Part -1

We are trying to list down all possible reasons that can be the root cause of below issues listed for Application Production Support domain. These are the basics that will be asked in each interview as well as a production support engineer needs to be familiar interact with these issues.


Thread Dump

• On Linux/UNIX system just the below command to get JAVA thread dump

                    $     kill -3   <PID of JVM>

•  In Case IBM WAS server -

            Use DMGR console  >> Troubleshooting >> Select JVM and click core

• Oracle Weblogic Server -

          Using wlsadmin.sh   script   or JSTACK utility  

JAVA Out Of Memory

• Low JAVA Heap Size on server
• Object Slicing too long, like HTTP sessions
• Memory leakage in application code
• Full garbage collection is not happening due to JVM bug.

Server Hang issue 

Various reason can attract the server hang condition that may be called a deadlock condition where system will behave wrong or even does not respond to user queries at all.

• Memory leakage in application code
• Database query is taking long to time to return the output 
• Deadlock Condition
• Network latency
• Out Of Memory Error

_________________________________________________________________________________

IBM WebSphere Introduction

What is IBM WebSphere ?


This is Application server built on Java and provides the Middleware and MQ support in enterprise application development in IT software field. It is large app server in market that is used in Enterprise service bus or service oriented architecture in various organization. It is called as WAS in short.



Below are the some info to remember about WebSphere


WAS Editions

• WAS Network Deployment
• WAS Developers
• WAS Z/OS
• WAS Hypervisor 
• WAS Express
• WAS Base
• WAS Liberty Core

WAS default Ports ?

HTTP /HTTPS     -    9080 / 9443

How To Check WAS Version

Go to profile and bin folder of WAS installation on server box and execute below scripts

$    ./versioninfo.sh




More info at IBM WebSphere Home
_________________________________________________________________________________

Oracle Weblogic Server Starting Status in logs

Different status message When a user starts a weblogic server instance 


1. STARTING

2. STANDBY

3. STARTING

4. ADMIN

5. RESUMING

6. RUNNING



When Weblogic server admin starts the weblogic server from domain home, he may check the status of admin server in AdminServer.log file.

Start service on Operating system on boot time - chkconfig/systemctl command

These are Linux commands to deal with services running for different apps on operating system. By the use of systemctl and chkconfig command, we can start and stop the services as well as configure them for auto start/stop in unix.


Basically above commands or tools allows you to configure automatic service start stop in the /etc/init.d scripts available on operating systems. Until RHEL 6.x, "chkconfig " command was available and later RHEL 7, the command "systemctl" introduced for auto service start/stop. although all legacy commands may support in new versions of RHEL and CentOS.


List All Services

  By Chkconfig command, list all the services on

[root@aps ~]# chkconfig  --list
httpd           0:off   1:off   2:on    3:on    4:on    5:on    6:off
rhnsd           0:off   1:off   2:off   3:on    4:off   5:on    6:off
netconsole      0:off   1:off   2:off   3:on    4:off   5:on    6:off
...

Above snippet shows different run level by numbering and post fixes.

By Systemctl, all active services and all enabled/disabled services

[root@aps ~]# systemctl list-units --type service

[root@aps ~]# systemctl list-unit-files --type service

Check Specific Service Status

[root@aps ~]# chkconfig --list | grep httpd
httpd           0:off   1:off   2:off   3:on   4:on   5:on   6:off

[root@aps ~]# systemctl list-units --type service | grep httpd

Stop Specific Service on Run Levels

Below command will stop httpd service at run level 2,3,4,5.

[root@aps ~]# chkconfig --level 2345 httpd off

Start/Stop Specific Service by systemctl

[root@aps ~]# systemctl stop [service_name]

[root@aps ~]# systemctl start [service_name]

Check Status of a Service by systemctl

[root@aps ~]# systemctl status [service_name]

How to Auto Start a Service at Boot of Operating System

By chkconfig command, below will stop httpd service at run level 4,5.

[root@aps ~]# chkconfig --level 45 httpd off

By systemctl command, you can enable any of service to start on boot time as below

[root@aps ~]# systemctl enable [service_name]

[root@aps ~]# systemctl disable [service_name]

_________________________________________________________________________________

Remove 7 days older log files from Unix server

This is the basic task in application support that requires log files to remove, back up to other location that are older than "N" days or let suppose 7 days from system date.

Sometimes requirement may ask you to run this task automatically on specified day, date and time to check on system file system, then you may use below commands in cron schedule on UNIX by making a script and set with schedule on crontab.

• find the log files that are older than "n" days and remove all ? write the command plz....

           $ find   <path-to-check>  -name *.log -mtime  +n   -exec rm -rf {}\;

          Here <path-to-check> is server location or path to check and "+n" shows days.   * is wild card            to check all log files having extension .log

          Example :

          $ find   /tmp/logs  -name *.log -mtime  +7   -exec rm -rf {}\;

          Above will remove all log files older than 7 days.