hi,
we are using QB 13.0.10,
and we launch nodes on K8s with cloud profile,
we facing with sporadic issue that few agents from same cloud profile with same docker image failed on connectivity issue to QB.
for example 10 Pods start at the same time on our cluster, trigger from QB Cloude profile, 5 successes to connect QB and move to active status, and 5 failed on connectivity and stay in inactive status until we terminated the manually.
in QB agent logs we see those issue:
ERROR Error connecting server.
jvm 1 | com.pmease.quickbuild.RemotingException: com.caucho.hessian.client.HessianRuntimeException: Error connecting 'http://10.184.71.46:8811/service/node'
jvm 1 | at com.caucho.hessian.client.HessianProxy.sendRequest(HessianProxy.java:285)
jvm 1 | at com.caucho.hessian.client.HessianProxy.invoke(HessianProxy.java:171)
jvm 1 | at com.sun.proxy.$Proxy94.setUserAttributes(Unknown Source)
jvm 1 | at com.pmease.quickbuild.grid.GridNode.setUserAttributes(GridNode.java:117)
jvm 1 | at com.pmease.quickbuild.grid.DefaultAgentManager.useCloudProfile(DefaultAgentManager.java:295)
jvm 1 | at com.pmease.quickbuild.grid.DefaultAgentManager.buildAgentActive(DefaultAgentManager.java:252)
jvm 1 | at com.pmease.quickbuild.grid.ConnectServlet.connect(ConnectServlet.java:196)
jvm 1 | at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
jvm 1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
jvm 1 | at java.lang.reflect.Method.invoke(Method.java:498)
jvm 1 | at com.caucho.hessian.server.HessianSkeleton.invoke(HessianSkeleton.java:302)
jvm 1 | at com.caucho.hessian.server.HessianSkeleton.invoke(HessianSkeleton.java:198)
jvm 1 | at com.caucho.hessian.server.HessianServlet.invoke(HessianServlet.java:399)
jvm 1 | at com.caucho.hessian.server.HessianServlet.service(HessianServlet.java:379)
jvm 1 | at com.pmease.quickbuild.grid.ConnectServlet.service(ConnectServlet.java:64)
jvm 1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
jvm 1 | at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)
jvm 1 | at com.pmease.quickbuild.Quickbuild$DisableTraceFilter.doFilter(Quickbuild.java:1278)
jvm 1 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
jvm 1 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
jvm 1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
jvm 1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
jvm 1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
jvm 1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
jvm 1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
jvm 1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
jvm 1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
jvm 1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
jvm 1 | at org.eclipse.jetty.server.Server.handle(Server.java:516)
jvm 1 | at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
jvm 1 | at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
jvm 1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
jvm 1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
jvm 1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
jvm 1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
jvm 1 | at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
jvm 1 | at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
jvm 1 | at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
jvm 1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
jvm 1 | at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
jvm 1 | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
jvm 1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
jvm 1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
jvm 1 | at java.lang.Thread.run(Thread.java:748)
we checked our nodes in cluster but we not saw something special, also we saw that same node run success Pods and Pod with connectivity issue at same time.