Adding GC logs to hadoop child processes and analyzing the GC logs

Add the following to the mapred-site.xml file:

mapred.child.java.opts : Java opts for the task tracker child processes. The following symbol, if present, will be interpolated: @taskid@ is replaced by current TaskID. Any other occurrences of '@' will go unchanged. For example, to enable verbose gc logging to a file named for the taskid in /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of: -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc.

Additional options: -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC

Reference: http://hadoop.apache.org/docs/r1.0.0/mapred-default.html


Analyzing GC logs:
-------------------------

Meaning of the [GC [PSYoungGen: 230400K->19135K(268800K)] line is:

• Around 256MB (268800K) is the Young Generation Size, 
• Before Garbage Collection in young generation the heap utilization in Young Generation area was around 255MB (230400K) and 
• After garbage collection it reduced up to 18MB (19135K).

Reference: http://middlewaremagic.com/weblogic/?p=5131
https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=22d56091-3a7b-4497-b36e-634b51838e11