Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

cloud profile connectivity issue to QB #4553

idoharel ·

hi,
we are using QB 13.0.10,
and we launch nodes on K8s with cloud profile,
we facing with sporadic issue that few agents from same cloud profile with same docker image failed on connectivity issue to QB.
for example 10 Pods start at the same time on our cluster, trigger from QB Cloude profile, 5 successes to connect QB and move to active status, and 5 failed on connectivity and stay in inactive status until we terminated the manually.

in QB agent logs we see those issue:

ERROR Error connecting server.
jvm 1 | com.pmease.quickbuild.RemotingException: com.caucho.hessian.client.HessianRuntimeException: Error connecting 'http://10.184.71.46:8811/service/node'
jvm 1 | at com.caucho.hessian.client.HessianProxy.sendRequest(HessianProxy.java:285)
jvm 1 | at com.caucho.hessian.client.HessianProxy.invoke(HessianProxy.java:171)
jvm 1 | at com.sun.proxy.$Proxy94.setUserAttributes(Unknown Source)
jvm 1 | at com.pmease.quickbuild.grid.GridNode.setUserAttributes(GridNode.java:117)
jvm 1 | at com.pmease.quickbuild.grid.DefaultAgentManager.useCloudProfile(DefaultAgentManager.java:295)
jvm 1 | at com.pmease.quickbuild.grid.DefaultAgentManager.buildAgentActive(DefaultAgentManager.java:252)
jvm 1 | at com.pmease.quickbuild.grid.ConnectServlet.connect(ConnectServlet.java:196)
jvm 1 | at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
jvm 1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
jvm 1 | at java.lang.reflect.Method.invoke(Method.java:498)
jvm 1 | at com.caucho.hessian.server.HessianSkeleton.invoke(HessianSkeleton.java:302)
jvm 1 | at com.caucho.hessian.server.HessianSkeleton.invoke(HessianSkeleton.java:198)
jvm 1 | at com.caucho.hessian.server.HessianServlet.invoke(HessianServlet.java:399)
jvm 1 | at com.caucho.hessian.server.HessianServlet.service(HessianServlet.java:379)
jvm 1 | at com.pmease.quickbuild.grid.ConnectServlet.service(ConnectServlet.java:64)
jvm 1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
jvm 1 | at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)
jvm 1 | at com.pmease.quickbuild.Quickbuild$DisableTraceFilter.doFilter(Quickbuild.java:1278)
jvm 1 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
jvm 1 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
jvm 1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
jvm 1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
jvm 1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
jvm 1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
jvm 1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
jvm 1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
jvm 1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
jvm 1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
jvm 1 | at org.eclipse.jetty.server.Server.handle(Server.java:516)
jvm 1 | at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
jvm 1 | at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
jvm 1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
jvm 1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
jvm 1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
jvm 1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
jvm 1 | at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
jvm 1 | at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
jvm 1 | at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
jvm 1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
jvm 1 | at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
jvm 1 | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
jvm 1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
jvm 1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
jvm 1 | at java.lang.Thread.run(Thread.java:748)

we checked our nodes in cluster but we not saw something special, also we saw that same node run success Pods and Pod with connectivity issue at same time.

  • replies 3
  • views 165
  • stars 0
robinshen ADMIN ·

QB server will connect back to agent during agent start up. In this case, server will connect to agent via ip "10.184.71.46" and port "8811". When this issue happens, please run below on server terminal to see if it works:

telnet 10.184.71.46 8811

idoharel ·

I get:
telnet 10.167.22.134 8811
Trying 10.167.22.134...
telnet: connect to address 10.167.22.134: Connection refused

I take the IP from sporadic problematic POD

robinshen ADMIN ·

Seems like a network issue to me. Please present this issue to your k8s admin to see if they have some ideas.