在系统访问高峰期出现无法获取数据库连接的异常
最近遇到一个棘手的问题,请教各位能否查明报出异常的原因：
1：正常情况下系统运行没有问题，但是系统访问高峰情况下会出现，日志如下，这段日志出现一段时间之后，应用就会假死，持续1分钟左右之后恢复。
org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.exceptions.PersistenceException:
Error querying database. Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException
: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/PnrDS
The error may exist in com/huifu/muser/common/dal/dao/MerUsrMapMapper.xml
The error may involve com.huifu.muser.common.dal.dao.MerUsrMapMapper.queryMerUsrMap
The error occurred while executing a query
Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceE
xception: IJ000453: Unable to get managed connection for java:jboss/datasources/PnrDS
at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:73)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:365)
at com.sun.proxy.$Proxy157.selectList(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:195)
at org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:124)
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:90)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:40)
at com.sun.proxy.$Proxy158.queryMerUsrMap(Unknown Source)
at com.huifu.muser.service.core.cash.impl.CashReqServiceImpl.checkMerUsrMap(CashReqServiceImpl.java:267)
at com.huifu.muser.biz.cash.impl.CashReqManager2Impl.cashConfirmReq(CashReqManager2Impl.java:211)
at com.huifu.muser.webapp.controller.cash.CashController.cashConfirm2(CashController.java:200)
at sun.reflect.GeneratedMethodAccessor2127.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:426)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:414)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:560)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
2：我的应用框架是 spring3.0.5+mybatis3.1.1+JBoss7.1+oracle11+jdk1.6
3：负载均衡 netscaler,两台虚拟机，四核，内存8G,jvm启动参数配置如下：
-XX:+UseCompressedOops -XX:+TieredCompilation -Xms4096m -Xmx4096m -Xmn1024m -XX:PermSize=256M -XX:MaxPermSize=512M -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:SurvivorRatio=8 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Xnoclassgc -XX:+CMSClassUnloadingEnabled
4：我的数据库链接池是配置在jboss的配置文件standalone-ha.xml中，连接池配置为18-300，高峰期能达到300，即应用确实与数据库建立300链接，但是活跃链接在15左右，配置如下
Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection;
看下你的jdbc的最大连接数是多少？
jdbc最大连接数多少？提供片段读取写入的代码，看一下你重用连接的代码。
查看一下数据库的日志 里面一般有描述当前数据库连接的问题等 看是连接数太多还是数据处理问题
" 连接池配置为18-300，高峰期能达到300 "
说明高峰期瞬间连接数确实可能超过300，那你分别把配置文件中max-pool-size调大一点，比如到400或500，然后确认Oracle最大进程数够用；再试试就可以了。
https://developer.jboss.org/thread/211940?start=0
直接原因就是 CannotGetJdbcConnectionException
当数据库连接到了设定最大值时,再想获得连接,就出错了
一方面,要检查是否有连接用完没关闭的情况
如果没问题,那么就要加大最大连接数了
你可以监控一下数据库本身,如果忙得时候很慢,会导致很多连接无法及时完成而无法释放,此时加大连接数只能更糟糕
这时你只能优化数据的软硬件和应用本身了
楼主 我也遇到这样的问题 
我的解决思路是：1 首先排查代码 是否 connection 没释放 2 模拟一个http请求直接访问服务器 但是效果都不明显 楼主有啥好建议吗？求交流
11:05:27,825 INFO Transaction [xwadmin]-queryPspGuarTaskList rollback current transaction...
11:05:27,826 INFO MVC [xwadmin]-queryPspGuarTaskList --------END--------VIEW=[psp/PspGuarTask/queryPspGuarTaskList.jsp]
11:05:27,828 INFO JMX Emmit notify from [CMISMvcServlet] as: Response too low in execute /cmis-main/queryPspGuarTaskList.do used time as:30060
11:05:44,904 ERROR stderr com.ecc.emp.jdbc.GetConnectionFailedException
11:05:44,906 ERROR MVC [113111633]-queryPubWbAllTaskList Failed to execute operation com.yucheng.cmis.pub.op.core.pubwbitem.QueryPubWbAllTaskListOp ! : com.ecc.emp.jdbc.GetConnectionFailedException: Failed to get connection from :com.ecc.emp.jdbc.JNDIDataSource@73220fd7 caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/WFDataSource cause: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/WFDataSource
at com.ecc.emp.jdbc.ConnectionManager.getConnection(ConnectionManager.java:57) [emp2.2-101210.jar:]
at com.yucheng.cmis.operation.CMISOperation.getConnection(CMISOperation.java:528) [classes:]
at com.yucheng.cmis.operation.CMISOperation.getConnection(CMISOperation.java:497) [classes:]
at com.yucheng.cmis.pub.op.core.pubwbitem.QueryPubWbAllTaskListOp.doExecute(QueryPubWbAllTaskListOp.java:24) [classes:]
at com.yucheng.cmis.operation.CMISOperation.execute(CMISOperation.java:182) [classes:]
at com.yucheng.cmis.ad.mvc.ADRequestController.executeEMPLogic(ADRequestController.java:113) [classes:]
at com.ecc.emp.web.servlet.mvc.EMPRequestController.executeTheFlow(EMPRequestController.java:487) [emp2.2-101210.jar:]
at com.ecc.emp.web.servlet.mvc.EMPRequestController.doRequest(EMPRequestController.java:307) [emp2.2-101210.jar:]
at com.ecc.emp.web.servlet.EMPRequestServlet.service(EMPRequestServlet.java:427) [emp2.2-101210.jar:]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.jasig.cas.client.util.AssertionThreadLocalFilter.doFilter(AssertionThreadLocalFilter.java:54) [cas-client-core-extension-3.2.1.jar:3.2.1]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.jasig.cas.client.util.HttpServletRequestWrapperFilter.doFilter(HttpServletRequestWrapperFilter.java:75) [cas-client-core-extension-3.2.1.jar:3.2.1]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.jasig.cas.client.authentication.AuthenticationExtensionFilter.doFilter(AuthenticationExtensionFilter.java:106) [cas-client-core-extension-3.2.1.jar:3.2.1]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.jasig.cas.client.session.SingleSignOutFilter.doFilter(SingleSignOutFilter.java:76) [cas-client-core-extension-3.2.1.jar:3.2.1]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at com.ecc.emp.web.jsptags.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:44) [emp2.2-101210.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at com.yucheng.cmis.base.CMISZipFilter.doFilter(CMISZipFilter.java:39) [classes:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at com.ecc.emp.log.filter.Log4jFilter.doFilter(Log4jFilter.java:33) [classes:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final-RECOMPILE.jar:7.1.1.Final]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/WFDataSource
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:137)
at com.ecc.emp.jdbc.JNDIDataSource.getConnection(JNDIDataSource.java:97) [classes:]
at com.ecc.emp.jdbc.ConnectionManager.getConnection(ConnectionManager.java:44) [emp2.2-101210.jar:]
... 43 more
Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/WFDataSource
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:390)
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:129)
... 45 more
Caused by: javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (30000 [ms])
at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:377)
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:397)
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:365)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329)
... 48 more
11:05:44,966 INFO Transaction [113111633]-queryPubWbAllTaskList rollback current transaction...
11:05:44,967 INFO MVC [113111633]-queryPubWbAllTaskList --------END--------VIEW=[core/PubWbItem/queryPubWbAllTaskList.jsp]
11:05:44,969 INFO JMX Emmit notify from [CMISMvcServlet] as: Response too low in execute /cmis-main/queryPubWbAllTaskList.do used time as:30069
11:05:45,662 INFO stdout chenmingping
11:05:45,662 INFO MVC Accept request: /cmis-main/userSignOn.do
11:05:45,664 INFO Transaction [not session created]-userSignOn Start new transaction...
11:05:45,665 INFO [Transaction] (http--