Wednesday, January 14

Thread Dump



Thread Dump:

  1. What is Thread dump?
  2. When we will take Thread dump? (Scenarios)
  3. How  Many  ways take Thread Dumps
  4. Thread Dump Generating Procedure
  5. What can I Analysis with Thread Dump?
  6. How can I analysis thread dump?
  7. Actions taken for Issue resolving
  8. References

 Coming to step by step learning:
--------------------------------

What is Thread dump?

Thread Dump is a textual dump of all active threads and monitors of Java apps running in a Virtual Machine.

When we will take Thread dump? (Scenarios)

1.  Scenario 1: when server is hang Position, i.e. that time server will not respond to coming requests.

2.  Scenario 2: While sever is taking more time to restart

3.  Scenario 3: When we are Getting exception like “java.lang.OutOfMemoryException”

4.  Scenario 4: Process running out of File descriptors. Server cannot accept further requests because sockets cannot be created

5.  Scenario 5: Infinite Looping in the code


How many ways take Thread Dumps?

Many types we have to take a Thread dumps. As per your flexibility you can choose one Procedure. For analyzing take dumps some Intervals (like every 10mins, 10mins etc.).



Generating Dump Talking Procedures

1. Take Thread dump from Console by Using of below command
    $kill -3 PID
   (For Getting PID, Use this Command ps –ef | grep “java”)
Here The Output of the Thread Dump will be generated in the Server STDOUT.
(Note: If a process is not responding to kill -3 then it’s a JVM bug.)


2. Generation Thread Dump via Admin Console

a.  login to Admin Console(with Admin Username/Password)
b.  Click on Server, after choose your server
c.  Goto Monitoring TAB
d.  Goto Threads TAB, after click on “Dump Thread Stack” Button
e.  Now you can view the all the Threads in Same page
f.  Copy and paste in a txt file.

3.  We can Collect Thread Dump Using “WebLogic.Admin” which is deprecated, but still available or may be available in near future as well As i think because it is one of the best debugging utility for Admins.

  java WebLogic.Admin -url t3://hostname: port -username Weblogic -password Weblogic THREAD_DUMP


This Thread Dumps will be generated in Servers STDOUT file


4. Getting Thread Dumps by using Jstack Utility

    a.jstack –m (to connect to a live java process)

    b. jstack –m [server_id@]
                (to connect to a remote debug server)
    (-m Means print both java and native frames (mixed mode)) 

5. By Using WLST Script, can contain extension of (.py)

connect(‘weblogic’,'weblogic’,'t3://hostname:port′) cd (”Servers’) ls()cd (‘AdminServer’) ls() threadDump()

 Execute this Script in console. 

What can I Analysis with Thread Dumps?
We need to analyze the thread dumps for analyzing running threads and their states to identifying.

How can I analysis thread dumps?

For analyze thread dumps we have lots of tools to understand easily thread states

1.  samurai tool :

    In this tool you can identify all the Thread states by     identifying colors. We need to take care about Deadlocks and waiting state threads.

   More Details:

    $ java -jar samurai.jar

     After running we will get a Screen like below

      Goto Thread dump tab
    When Samurai detects a thread dump in your log, a tab named "Thread Dump" will appear.

 You can just click "Thread dumps" tab to see the analysis result. Samurai colors idle threads in gray, blocked threds in red and running threds in green. There are three result views and Samurai shows "Table view" by default. In many case you are just interested in the table view and the sequence view. Use the table view to decide which thread needs be inspected, the sequence view to understand the thread's behavior. You should takecare especially threds always in red.


2.  TDA Tool :


Actions taken for Issue resolving

1.  Classic Dead Locks : Look for the threads waiting for monitor entry

For Example :

"ExecuteThread: '95' for queue: 'default'" daemon prio=5 tid=0x411cf8 nid=0x6c waiting for monitor entry [0xd0f80000..0xd0f819d8]
    at weblogic.common.internal.ResourceAllocator.release(ResourceAllocator.java:766)
    at weblogic.jdbc.common.internal.ConnectionEnv.destroy(ConnectionEnv.java:590)
Reason: The above thread is waiting to acquire lock on Resource Allocator object. The next step is to identify the thread that is holding the Resource Allocator object
"ExecuteThread: '0' for queue: '__weblogic_admin_rmi_queue'" daemon prio=5 tid=0x41b978 nid=0x77 waiting for monitor entry [0xd0480000..0xd04819d8]
    at weblogic.jdbc.common.internal.ConnectionEnv.getPrepStmtCacheHits(ConnectionEnv.java:174)
    at weblogic.common.internal.ResourceAllocator.getPrepStmtCacheHitCount   (ResourceAllocator.java:1525)
Reason: This thread is holding lock on source Allocator object, but is waiting for Connection Env object. This is a classic deadlock.

     

2.  Threads in wait() state:
   A sample dump:

"ExecuteThread: '10' for queue: 'SERV_EJB_QUEUE'" daemon prio=5 tid=0x005607f0 nid=0x30 in Object.wait() [83300000..83301998]
  at java.lang.Object.wait(Native Method)
  - waiting on (a weblogic.ejb20.pool.StatelessSessionPool)
at weblogic.ejb20.pool.StatelessSessionPool.waitForBean(StatelessSessionPool.java:222)

Reason: The above thread would come out of wait() under two conditions
 (Depending on application logic)
1) One of the thread available in the execute queue pool would call notify() on this object when an instance is available. (If the wait() is indefinite).
  This can cause the thread to hang for ever if server never does a notify() to this object.
2) If the timeout exceeds, the thread would throw an exception and back to execute queue thread pool.

References

Thanks to Bodepudi Kotesh ..

Thursday, January 8

What is lok file ?how many types of lok files are there?


  • Lubomír Maliník lok files are for locking some action which should be run or made only by one user or holding process. You can find config.lok invoked for serial update of config.xml, then each server has its own lok file for server and lok file for embedLDAP to avoid start same manged server in time twice. Then you can find edit.lok invoked by Edit actions for one user editing domain configuration in time. I am not sure if other exist.
  • Lubomír Maliník It is good to know it, because sometimes when server crash you cant start servers and it writes something like "Instance is already running" Or second issue server begin to start and then after line in log with text "....IIOP..." next step is load embedLDAP but server stuck for long time in this step because ldap is locked or corrupted. Remove of these lok files usually can help you to start server perfectly again

Persistent Store


  • Moqthayar Ali Shaik It is used to store weblogic resources and weblogic subsystems
  • Raju Sagi persistent store stores all you JMS messages into either file store (which will be saved in .dat format) or else in database store, so even though our weblogic went crash ,our JMS messages still persist in the file or database, in other hand if it is non-persistent our Weblogic intance keep messages in its memory but if instance went down all its memory along with messages will be wiped out.
  • Maadhavamadhausudhan Reddy K it is a physical repository for storing jms server data like queue and topic. There are two types. file and jdbc store

  • Chandra Varagani If the server crashes when the requests/messages are processing.. those requests will save in one place which calls persistent store. it depends on you that you can configure it or not. As the guys said above, there are 2 types. first, file persistent store can be defined in JMS till 8x version.. guess till 9.1.. From 9.2 onwards, JDBC persistent store have been introduced. so, you can configure it under JDBC.

Hi, Suppose we have apache webserver and weblogic manage servers in an environment. Client can access an application only from external URL..if so do we need SSL only for Apache or both apache and Weblogic manage servers.?


  • Sri Vas What is the need of SSL, keystore sections in weblogic console? if SSL configuration for apache is enough for the environment to protect data?
  • Sambi Reddy Ssl is of multiple types like one way two way.. Pls go through it
  • Anfraz Shaikh Enabling SSL on Weblogic would be an extra level of security. But if you dont have option then it would be enough to enable SSL on Apache as long as application is accessed via Apache URL.
  • Madhu Katuri It depends on your Infrastructure requirements. 1) You can terminate SSL at Apache or 2) Extend till WebLogic Managed Servers. You need to configure SSL only on Apache in case 1 and on both Apache and WebLogic in case 2 above. Case 1 (Browser to Apache) is one way SSL and Case 2 is Two-Way SSL . Generally between servers is Two-Way SSL