×
文章路径: 开发运维环境

Weblogic JRockit dump

发表于3年前(Dec 24, 2014 11:04:48 AM)  阅读 787  评论 0

分类: 开发运维环境

标签: Weblogic -Xnoopt dump Illegal memory access jrockit optimized 代码优化 服务器挂起

前天生产环境又挂了,幸好是下班时间,用的人不多,现场实施马上重启了。最近不忙,于是仔细分析了下日志文件,以前都没认真处理过,都是重启完事。生产环境用的redhat5.6,weblogic10.3,现在认真回查以前的日志文件,发现原来生产环境长的时候100多天,短的时候30多天就会挂一次,汗,原来这个问题这么严重,更加要重视。

查看日志发现有这么一段,[JRockit] ERROR: The JVM has crashed. Writing crash information to /bea/weblogic/user_projects/domains/top_domain/jrockit.23034.dump.
java虚拟机崩溃了,这是jrockit.23034.dump报错的日志内容,很长,不过关键性的就几句,应该说能看懂的就那么几句:

===== BEGIN DUMP =============================================================
JRockit dump produced after 48 days, 09:18:20 on Tue Mar 26 18:14:33 2013

    **************************************************************************
    *  If you see this dump, please go to                                    *
    *  http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html  *
    *  for troubleshooting information.                                      *
    **************************************************************************

Additional information is available in:
  /bea/weblogic/user_projects/domains/top_domain/jrockit.23034.dump
No snapshot file (core dump) will be created because core dumps have been
disabled. To enable core dumping, try “ulimit -c unlimited”
before starting JRockit again.

Error Message: Illegal memory access. [54]
Signal info  : si_signo=11, si_code=1 si_addr=0x58
Version      : Oracle JRockit(R) R28.1.3-11-141760-1.6.0_24-20110301-1432-linux-x86_64
CPU          : Intel Core 2 SSE SSE2 SSE3 SSSE3 SSE4.1 Core Intel64
Number CPUs  : 8
Tot Phys Mem : 8371331072 (7983 MB)
OS version   : Red Hat Enterprise Linux Server release 5.5 (Tikanga)
Linux version 2.6.18-194.el5 (mockbuild@x86-005.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Tue Mar 16 21:52:39 EDT 2010 (x86_64)
Hypervisor   : Unknown (but probably running on a hypervisor)
Thread System: Linux NPTL
LibC release : 2.5-stable
Java locking : Lazy unlocking enabled (class banning) (transfer banning)
State        : JVM is running
Command Line : -Xms3072m -Xmx3072m -Dweblogic.Name=AdminServer -Djava.security.policy=/bea/weblogic/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -da -Dplatform.home=/bea/weblogic/wlserver_10.3 -Dwls.home=/bea/weblogic/wlserver_10.3/server -Dweblogic.home=/bea/weblogic/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/bea/weblogic/patch_wls1033/profiles/default/sysext_manifest_classpath:/bea/weblogic/patch_oepe1033/profiles/default/sysext_manifest_classpath:/bea/weblogic/patch_ocp353/profiles/default/sysext_manifest_classpath -Dsun.java.launcher=SUN_STANDARD weblogic.Server
Repository   : /tmp/2013_02_06_08_56_20_23034
java.home    : /bea/jrockit/jre
j.class.path : /bea/weblogic/patch_wls1033/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/bea/weblogic/patch_oepe1033/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/bea/weblogic/patch_ocp353/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/bea/jrockit/lib/tools.jar:/bea/weblogic/wlserver_10.3/server/lib/weblogic_sp.jar:/bea/weblogic/wlserver_10.3/server/lib/weblogic.jar:/bea/weblogic/modules/features/weblogic.server.modules_10.3.3.0.jar:/bea/weblogic/wlserver_10.3/server/lib/webservices.jar:/bea/weblogic/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/bea/weblogic/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/bea/weblogic/wlserver_10.3/common/derby/lib/derbyclient.jar:/bea/weblogic/wlserver_10.3/server/lib/xqrl.jar
j.lib.path   : /bea/jrockit/jre/lib/amd64/jrockit:/bea/jrockit/jre/lib/amd64:/bea/jrockit/jre/../lib/amd64:/bea/weblogic/patch_wls1033/profiles/default/native:/bea/weblogic/patch_oepe1033/profiles/default/native:/bea/weblogic/patch_ocp353/profiles/default/native:/bea/weblogic/wlserver_10.3/server/native/linux/i686:/bea/weblogic/wlserver_10.3/server/native/linux/i686/oci920_8
JAVA_HOME    : /bea/jrockit
_JAVA_OPTIONS: <not set>
LD_LIBRARY_PATH: /bea/jrockit/jre/lib/amd64/jrockit:/bea/jrockit/jre/lib/amd64:/bea/jrockit/jre/../lib/amd64:/bea/weblogic/patch_wls1033/profiles/default/native:/bea/weblogic/patch_oepe1033/profiles/default/native:/bea/weblogic/patch_ocp353/profiles/default/native:/bea/weblogic/wlserver_10.3/server/native/linux/i686:/bea/weblogic/wlserver_10.3/server/native/linux/i686/oci920_8
LD_ASSUME_KERNEL: <not set>
LD_PRELOAD   : <not set>
StackOverFlow: 0 StackOverFlowErrors have occured
OutOfMemory  : 0 OutOfMemoryErrors have occured
C Heap       : Good; no memory allocations have failed
GC Strategy  : Mode: throughput, with strategy: genparpar (basic strategy: genparpar)
GC Status    : OC is not running. Last finished OC was OC#13786.
             : YC is not running. Last finished YC was YC#52604.
YC Promotion : Last YC successfully promoted all objects
YC History   : Ran 6 YCs before OC#13782.
             : Ran 7 YCs before OC#13783.
             : Ran 5 YCs before OC#13784.
             : Ran 5 YCs before OC#13785.
             : Ran 7 YCs before OC#13786.
             : Ran 6 YCs since last OC.
Heap Holes   : 2 hole(s), at 0x410dd000 – 0x41ade000, 0x7fffe000 – 0x80000000
Heap         : 0x3f5fd000 – 0x100000000  (Size: 3072 MB)
Compaction   : (no compaction area)
Allocation   : TLA-min: 2048, TLA-preferred: 65536 TLA-waste limit: 2048
NurseryList  : 0x49656278 – 0xa5473898
KeepArea     : 0x8eb766c0 – 0xa5473898
KA Markers   : [ 0x7822cc48,  0x8eb766c0 , 0xa5473898 ]
Forbidden A  : (none)
Previous KA  : 0x7822cc48 – 0x8eb766c0
Previous FA  : (none)
CompRefs     : References are compressed, with heap base 0x0 and shift 0.

Registers (from ThreadContext: 0x2aaaf5755940:
  rax = 00002aab212231a8   rcx = 000000000000041f
  rdx = 0000000000004000   rbx = 0000000000000000
  rsp = 00002aaaf5755d88   rbp = 00002aaaf5756000
  rsi = 00002aab212231a8   rdi = 0000000000000000
   r8 = 0000000000000000    r9 = 0000000000000000
  r10 = 0000000000010000   r11 = 0000000000000010
  r12 = 0000000010000023   r13 = 00002aab212231a8
  r14 = 00002aab212104e0   r15 = 00000000ffffffff
   cs = 0000000000000033    fs = 0000000400000000
   gs = 0004000000000000
  rip = 00002b5c254b55f0 flags = 0000000000000202

Loaded modules:
(* denotes the module where the exception occured)
0000000000400000-0000000000412703  /bea/jrockit/bin/java
0000003e13400000-0000003e13401f93  /lib64/libdl.so.2
0000003e13c00000-0000003e13c15a6f  /lib64/libpthread.so.0
0000003e13000000-0000003e1314d067  /lib64/libc.so.6
0000003e12c00000-0000003e12c1bb7f  /lib64/ld-linux-x86-64.so.2
00002b5c253e0000-00002b5c256e5343 */bea/jrockit/jre/lib/amd64/jrockit/libjvm.so
00002b5c259bf000-00002b5c259dd9eb  /bea/jrockit/jre/lib/amd64/libosal.so
00002b5c25be0000-00002b5c25bec4fb  /bea/jrockit/jre/lib/amd64/libutil.so
0000003e13800000-0000003e1388122f  /lib64/libm.so.6
0000003e17c00000-0000003e17c06e8b  /lib64/librt.so.1
00002aaaaabd0000-00002aaaaabd964b  /lib64/libnss_files.so.2
00002aaaaaddb000-00002aaaaade6d03  /bea/jrockit/jre/lib/amd64/libjfr.so
00002aaaf1813000-00002aaaf181f337  /bea/jrockit/jre/lib/amd64/libverify.so
00002aaaf1922000-00002aaaf194a19b  /bea/jrockit/jre/lib/amd64/libjava.so
0000003e1ac00000-0000003e1ac140c7  /lib64/libnsl.so.1
00002aaaf1ad1000-00002aaaf1ad75bf  /bea/jrockit/jre/lib/amd64/native_threads/libhpi.so
00002aaaf5500000-00002aaaf550dbc3  /bea/jrockit/jre/lib/amd64/libzip.so
00002aaaf5c22000-00002aaaf5c34d1f  /bea/jrockit/jre/lib/amd64/libnet.so
00002aaaf60fb000-00002aaaf6100f43  /bea/jrockit/jre/lib/amd64/liborii.so
00002aaaf63c2000-00002aaaf63c897b  /bea/jrockit/jre/lib/amd64/libnio.so
00002aaaf728e000-00002aaaf729347b  /bea/jrockit/jre/lib/amd64/libmanagement.so
00002aaafdf88000-00002aaafdf91bbb  /bea/jrockit/jre/lib/amd64/libjmapi.so
00002aab01813000-00002aab01816d73  /lib64/libnss_dns.so.2
0000003e1b800000-0000003e1b810823  /lib64/libresolv.so.2
00002aab07504000-00002aab0759586b  /bea/jrockit/jre/lib/amd64/libawt.so
00002aab076d3000-00002aab076d799f  /bea/jrockit/jre/lib/amd64/headless/libmawt.so
00002aab14350000-00002aab143c821f  /bea/jrockit/jre/lib/amd64/libfontmanager.so
00002aab15742000-00002aab157428bb  /bea/jrockit/jre/lib/amd64/librmi.so

Stack:
(* marks the word pointed to by the stack pointer)
00002aaaf5755d88: 00002b5c25426f32* 000000000000000e  00002aab212240d0  00002aaaf5755f50 
00002aaaf5755da8: 00002aaaf5755e50  00002aaaf5755fc8  00002aaaf5755fa0  00002aaaf5756170 
00002aaaf5755dc8: 000000002561d52b  0000000000000094  00002aab21220c28  00002aab212214c0 
00002aaaf5755de8: 0000000000000000  0000000100000000  00002aab21224030  00002aab21224c38 
00002aaaf5755e08: 00002aab212219c0  00002aab2121fb80  000000000000000d  00002aab21223f20 
00002aaaf5755e28: 00002aab212212b8  00002aaaf5755e60  00002aab212104e0  000000000000005c 

Code:
(* marks the word pointed to by the instruction pointer)
00002b5c254b5590: 478b486842894868  89480474c0854868  394858578b486050  487a39483474407a 
00002b5c254b55b0: 00005847c7481e74  00006847c7480000  00006047c7480000  00401f0fc3c90000 
00002b5c254b55d0: 4842894860478b48  0000441f0f66d8eb  58578b4840428948  0000441f0f66c2eb 
00002b5c254b55f0: e589485558578b48* 39484a74407a3948  60478b485c74487a  7460468948c08548 
00002b5c254b5610: 478b486870894804  68468948c0854868  8948607089480474  00006847c7485856 
00002b5c254b5630: 00005847c7480000  00006047c7480000  00401f0fc3c90000  58578b4840728948 

Generating org/jaxen/saxpath/helpers/XPathReaderFactory.createReader(Ljava/lang/String;)Lorg/jaxen/saxpath/XPathReader; (NORMAL).
Directives (#1):
 strategy:quick_regalloc=0
 strategy:debug=0
 strategy:quick_misc=0
 strategy:fusion_regalloc=1
 strategy:jit_inline=1
 strategy:opt_inline=1
 strategy:callprofiling=0
 strategy:remove_redundant_store=1
 strategy:loop_peeling=1
 strategy:string_append=1
 strategy:remove_cast=1
 strategy:virtual_to_fixed=1
 strategy:string_maker=1
 strategy:expand_acopy=1
 strategy:inline_leafs=1
 strategy:pushup_cmpclass=1
 strategy:straighten=1
 strategy:sparse_constprop=1
 strategy:par_flow=1
 strategy:remove_casts=1
 strategy:dead_assignment_removal=1
 strategy:aggr_deadcode=1
 strategy:strength_reduction=1
 strategy:hoist_loopinv=1
 strategy:clean_memaccess=1
 strategy:opt_boxing=1
 strategy:push_boxops=1
 strategy:merge_boxops=1
 strategy:join_boxops=1
 strategy:copyprop_boxops=1
 strategy:explode_arraycopy=1
 strategy:opt_memset=1
 strategy:straighten_cmps=1
 strategy:straighten_ccmps=1
 strategy:remove_checks=1
 strategy:switch_conv=1
 strategy:dead_branch=1
 strategy:compact_blocks=1
 strategy:guard_acopy=1
 strategy:remove_aclear=1
 strategy:opt_divrem=1
 strategy:convert_virtual=1
 strategy:split_unlocks=1
 strategy:pair_locks=1
 strategy:lock_removal=1
 strategy:lock_copyprop=1
 strategy:lock_fusion=1
 strategy:group_getters=1
 strategy:ldst_ordering=1
 strategy:opt_prefetch=1
 strategy:inline_boxops=1
 strategy:copy_prop=1
 strategy:remove_rcmps=1
 strategy:opt_abcd=1
 strategy:remove_abc=1
 strategy:lcopy_prop=1
 strategy:gcopy_prop=1
 strategy:dead_code=1
 strategy:lcopy_rprop=1
 strategy:gcopy_rprop=1
 strategy:sign_elim=1
 strategy:lir_hints=1
 strategy:opt_fp=1
 strategy:insert_sp=1
 strategy:lir_modifystore=1
 strategy:opt_coalesce=1
 strategy:lir_peep=1
 strategy:nir_peep=1
 strategy:nir_copyprop=1
 strategy:inline_redudantstore=1
 strategy:inline_looppeel=1
 strategy:opt_prefetch_io=1
 strategy:opt_inline_rewrite=1
 strategy:repeated_constprop=1
 flags:strict_fp=0
 flags:schedule=1
 flags:trace_schedule=0
 flags:check_bounds=1
 flags:check_stacks=0
 flags:fill_delayslot=1
 flags:lazy_stacktraces=1
 flags:add_codeinfo=0
 flags:record=0

Last optimized methods:
  #6801 com/gdtopway/topologyweb/autolayout/vo/CheckInfoVO.<init>()V  4176765.436-4176765.439 0x2aaab4a36a40-0x2aaab4a36b39 3.18 ms 256KB 
  #6802 flex/messaging/io/amf/MessageBody.setReplyMethod(Ljava/lang/String;)V  4177265.951-4177265.989 0x2aaab4a36b40-0x2aaab4a37038 38.52 ms 768KB 
  #6803 org/hibernate/hql/ast/tree/Node.initialize(Lantlr/collections/AST;)V  4177679.510-4177679.513 0x2aaab4a37040-0x2aaab4a370ed 3.34 ms 256KB 
  #6804 $Proxy361.findDeviceVOsByGraphId(J)Ljava/util/List;  4177759.601-4177759.609 0x2aaab4a37100-0x2aaab4a3731a 7.68 ms 256KB 
  #6805 org/hibernate/hql/ast/HqlSqlWalker.prepareLogicOperator(Lantlr/collections/AST;)V  4179040.567-4179040.569 0x2aaab4a37320-0x2aaab4a37372 1.79 ms 256KB 
  #6806 org/hibernate/hql/ast/tree/SelectClause.getFirstSelectExpression()Lantlr/collections/AST;  4179158.544-4179158.547 0x2aaab4a37380-0x2aaab4a373f5 3.33 ms 256KB 
  #6807 sun/reflect/GeneratedConstructorAccessor2239.newInstance([Ljava/lang/Object;)Ljava/lang/Object;  4179313.486-4179313.489 0x2aaab4a37400-0x2aaab4a3759a 2.59 ms 256KB 
  #6808 org/hibernate/hql/antlr/HqlBaseParser.relationalExpression()V  4179525.716-4179526.257 0x2aaab4a375a0-0x2aaab4a39748 541.87 ms 3840KB 
  #6809 java/lang/String.substring(I)Ljava/lang/String;  4179649.125-4179649.130 0x2aaab4a39760-0x2aaab4a39897 4.40 ms 256KB 
  #6810 java/util/Date.normalize()Lsun/util/calendar/BaseCalendar$Date;  4179716.147-4179716.888 0x2aaab4a398a0-0x2aaab4a3bdea 741.20 ms 4352KB 
  #6811 sun/reflect/GeneratedConstructorAccessor2247.newInstance([Ljava/lang/Object;)Ljava/lang/Object;  4180074.254-4180074.260 0x2aaab4a3be00-0x2aaab4a3bf9a 6.29 ms 256KB 
  #6812 sun/reflect/GeneratedConstructorAccessor2246.newInstance([Ljava/lang/Object;)Ljava/lang/Object;  4180086.765-4180086.771 0x2aaab4a3bfa0-0x2aaab4a3c13a 6.05 ms 256KB 
 *#6813 $Proxy361.findTongGanVOByGraphId(Ljava/lang/Long;)Lcom/gdtopway/topologyweb/autolayout/vo/TongGanVO;  4180309.654-4180309.657 0x2aaab4a3c140-0x2aaab4a3c291 3.53 ms 256KB 
  #6798 java/lang/StringBuffer.setLength(I)V  4176307.515-4176307.527 0x2aaab4a36240-0x2aaab4a365c9 12.62 ms 512KB 
  #6799 java/lang/AbstractStringBuilder.setLength(I)V  4176307.527-4176307.536 0x2aaab4a365e0-0x2aaab4a36899 8.03 ms 512KB 
  #6800 sun/reflect/GeneratedConstructorAccessor2238.newInstance([Ljava/lang/Object;)Ljava/lang/Object;  4176707.426-4176707.428 0x2aaab4a368a0-0x2aaab4a36a3a 2.51 ms 256KB 

Thread:
“(Code Optimization Thread 1)” id=5 idx=0x34 tid=23047 lastJavaFrame=0xfffffffffffffffc
Stack 0: start=0x2aaaf5716000, end=0x2aaaf5758000, guards=0x2aaaf571b000 (ok), forbidden=0x2aaaf5719000

Thread Stack Trace:
    at irReplaceOp+0(ir.c:720)@0x2b5c254b55f0
    at optSSAStringAppend+95(append.c:763)@0x2b5c25427610
    at optmanOptimizeMIR+7432(optimizationmanager.c:241)@0x2b5c255b48f9
    at generate_stage+324(codemanager.c:340)@0x2b5c2544d995
    at cmgrGenerateMethodFromPhase+141(codemanager.c:404)@0x2b5c2544dace
    at cmgrGenerateMethod+386(codemanager.c:222)@0x2b5c2544de33
    at generate_code+1488(compilerfrontend.c:455)@0x2b5c25464031
    at cbrGenerateNormalMethod+41(compilerfrontend.c:668)@0x2b5c25464a3a
    at compileit+613(compilerbroker.c:453)@0x2b5c25459496
    at compileit+338(compilerbroker.c:379)@0x2b5c25459383
    at cbrCompileRequest+12(compilerbroker.c:537)@0x2b5c25459c2d
    at cg_thread+996(compilerqueue.c:223)@0x2b5c25465cc5
    at thread_stub+170(lifecycle.c:808)@0x2b5c2555d78b
    at start_thread+220()@0x3e13c0673d
    — Java stack —

Memory usage report:
Total mapped                  5427780KB           (reserved=1305792KB)
–              Java heap      3145728KB           (reserved=0KB)
–              GC tables       105572KB         
–          Thread stacks        31628KB           (#threads=82)
–          Compiled code      1048576KB           (used=47485KB)
–               Internal         1416KB         
–                     OS       311244KB         
–                  Other       455680KB         
– Compiling createReader          256KB           (malloced=83KB #310)
                                                  Not tracing sites.

–            Classblocks        17152KB           (malloced=16794KB #44704)
                                                  Not tracing sites.

–        Java class data       309504KB           (malloced=308550KB #206455 in 44704 classes)
                                                  Not tracing sites.

– Native memory tracking         1024KB           (malloced=170KB #12)
                                                  Not tracing sites.
Set the env variable TRACE_ALLOC_SITES=1 or use the print_memusage switch
trace_alloc_sites=true to enable alloc site tracing.

    **************************************************************************
    *  If you see this dump, please go to                                    *
    *  http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html  *
    *  for troubleshooting information.                                      *
    **************************************************************************

===== END DUMP ===============================================================

第一个地方:Error Message: Illegal memory access. [54]。翻译过来很好理解非法的内存读取,但是为什么会有这样的错误产生呢。

第二个地方:Last optimized methods:
  #6801 com/gdtopway/topologyweb/autolayout/vo/CheckInfoVO.<init>()V  4176765.436-4176765.439 0x2aaab4a36a40-0x2aaab4a36b39 3.18 ms 256KB 
  #6802 flex/messaging/io/amf/MessageBody.setReplyMethod(Ljava/lang/String;)V  4177265.951-4177265.989 0x2aaab4a36b40-0x2aaab4a37038 38.52 ms 768KB 
Last optimized methods翻译过来最后优化的方法。看了一下,就一个是我们项目的方法。检查了一下这个类,发现这是个很普通的hibernate实体类,没有什么特殊的地方。回过头看以往的服务器挂起时的日志,基本一下,就是最后优化的方法不同而已,基本上排除是代码的问题,应该是服务器或者项目本身配置的问题。

开始网上查找资料,翻阅书籍。在《Weblogic 企业级运维实战》一书中找到了相关有用的信息。在常规的服务器挂起中有一项是“代码优化中服务器挂起”,见P164原文“ 这个模式比较简单,顾名思义,当JVM在进行代码优化时,比如JIT编译时,会占用大量的cpu资源,有时对外表现为暂时没有响应。一般情况下,优化做完了服务器即能自动恢复正常,所以这种情况顺其自然就可以,没有特别刻意地去处理。”而实际上,每次我们的代码优化都导致了JVM崩溃,无法自动恢复服务。既然这样,那么能不能禁用代码优化呢。答案是肯定的,书上也给出了答案。p216本地内存OOM解决方法:JVM如果使用本地内存,可能会发生运行时类加载和代码优化。如果使用的是JRockit,可使用-Xnoopt参数来禁用,如果是Sun的Hotspot,可使用-Xint标志在解释模式中运行JVM(不会出现代码生成)。

我们使用的是Jrockit,找到域里面的setDomainEnv.cmd/setDomainEnv.sh文件,
if “%debugFlag%”==”true” (
 set JAVA_DEBUG=-Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,address=%DEBUG_PORT%,server=y,suspend=n -Djava.compiler=NONE
 set JAVA_OPTIONS=%JAVA_OPTIONS% %enableHotswapFlag% -Xnoopt -ea -da:com.bea… -da:javelin… -da:weblogic… -ea:com.bea.wli… –

ea:com.bea.broker… -ea:com.bea.sbconsole…
) else (
 set JAVA_OPTIONS=%JAVA_OPTIONS% %enableHotswapFlag% -Xnoopt -da
)

加入参数正常启动,功能一切正常,但还会不会出现此类错误有待时间证明,理论上问题处理是对头的。如有问题以后再补充。

发表评论