Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

Sporadic git connection failures #4310

SameOldSong ·

In our build we fetch data from AzureDevops Git and sporadic connectivity failures are quite a common issue. In all the relevant steps we configured multiple "Retry on failure", and that has solved the problem completely on the step level.

However, we still sometimes see this error (s. screenshot) on the Configuration overview. If I delete the error manually and start the build - everything works fine.

The problem is that the builds on these configurations are often triggered by an overall build automatically during the night. This error is completely unpredictable and if it appears on the Confgiuration overview - this configuration's build is never triggered. So our long running overall nightly builds fail if only just one of the triggered configurations has this utterly sporadic failure.

It doesn't happen every day but really sets us several hours back because we can only fix this in the morning. Is there any workaround? Any advice would be highly appreciated

2021-02-12_09h47_01.png
  • replies 11
  • views 341
  • stars 0
robinshen ADMIN ·

Can you please show me full stack trace of this error? It can be displayed by clicking the icon besides title "Error Message"

SameOldSong ·

well, unfortunately, this log is already gone, because in such a case we can only trigger the build if we remove this error. I will post it once it happens again, or if we find it in the old logs.

This is what shows in the log of the overall configuration that attempted and failed to trigger this build:

Can not resolve requested build for configuration ( ). Check server log for details.

SameOldSong ·

hello again. We came across the above mentioned problem again yesterday. This time we have the logs

2021-04-06 21:16:35,929 [pool-2-thread-8842] INFO com.pmease.quickbuild.DefaultBuildEngine - Processing build request (configuration:root/<path_to_config>, request id:48c0dcd2-d18a-4b3f-accf-0d976f6784cb)
2021-04-06 21:16:35,944 [pool-2-thread-8842] INFO com.pmease.quickbuild.DefaultBuildEngine - Checking build condition on node (address: <node_name>:<port>, ip: <ip address>)...
2021-04-06 21:16:36,047 [pool-2-thread-8734] INFO com.pmease.quickbuild.plugin.report.buildstats.StepStatisticsGenerator - Deleting step records of current build 3689848
2021-04-06 21:16:36,123 [pool-2-thread-8734] INFO com.pmease.quickbuild.plugin.report.engine.contribution.publisher.StatisticsReportPublisher - Finished collecting statistics data for report 'Build Stats' [configuration: 43704, build: 3689848]
2021-04-06 21:16:36,123 [pool-2-thread-8734] INFO com.pmease.quickbuild.DefaultBuildEngine - Collecting statistics 'Busted Report'...
2021-04-06 21:16:36,125 [pool-2-thread-8734] INFO com.pmease.quickbuild.DefaultBuildEngine - Collecting statistics 'JIRA'...
2021-04-06 21:16:36,131 [pool-2-thread-8734] INFO com.pmease.quickbuild.DefaultBuildEngine - Populating configuration data...
2021-04-06 21:16:36,135 [pool-2-thread-8734] INFO com.pmease.quickbuild.DefaultBuildEngine - Sending build notifications...
2021-04-06 21:16:36,172 [pool-2-thread-8734] INFO com.pmease.quickbuild.DefaultBuildEngine - Build request has been processed.
2021-04-06 21:17:00,215 [qtp1473118702-62] ERROR org.apache.wicket.Component - Error while getting default model object for Component: [RunningStepsPanel [Component id = panel, page = com.pmease.quickbuild.web.page.grid.GridPage, path = 0:tabs:panel:content:tabs:panel.RunningStepsPanel, isVisible = true, isVersioned = true], markup = [markup = jar:file:/C:/opt/quickbuild/plugins/com.pmease.quickbuild_10.0.31.jar!/com/pmease/quickbuild/web/component/tabbedpanel/TabbedPanel.html
 <div wicket:id="panel" class="panel"></div>, index = 0, current =  '<div wicket:id="panel" class="panel">' (line 0, column 0)], children = [ [Component id = steps]]]
2021-04-06 21:17:00,215 [qtp1473118702-62] ERROR org.apache.wicket.Component - Error while getting default model object for Component: [ [Component id = steps, page = com.pmease.quickbuild.web.page.grid.GridPage, path = 0:tabs:panel:content:tabs:panel:steps.RunningStepsPanel$3, isVisible = true, isVersioned = true], markup = [markup = jar:file:/<server_installation_path> com.pmease.quickbuild_10.0.31.jar!/com/pmease/quickbuild/web/page/grid/RunningStepsPanel.html
 <tr wicket:id="steps">
 					<td wicket:id="stepPath"></td>
 					<td wicket:id="resources"></td>
 					<td><span wicket:id="buildLink"></span></td>
 					<td><a wicket:id="configurationLink"></a></td>
 				</tr>, index = 0, current =  '<tr wicket:id="steps">' (line 0, column 0)]]
2021-04-06 21:17:00,215 [qtp1473118702-62] ERROR com.pmease.quickbuild.web.WicketConfig - Error handling wicket request.
 org.apache.wicket.WicketRuntimeException: Error attaching this container for rendering: [RunningStepsPanel [Component id = panel]]
 	at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1797)
 	at org.apache.wicket.Component.onBeforeRender(Component.java:3798)
 	at org.apache.wicket.Component.internalBeforeRender(Component.java:986)
 	at org.apache.wicket.Component.beforeRender(Component.java:1022)
 	at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1785)
 	at org.apache.wicket.Component.onBeforeRender(Component.java:3798)
 	at org.apache.wicket.Component.internalBeforeRender(Component.java:986)
 	at org.apache.wicket.Component.beforeRender(Component.java:1022)
 	at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1785)
 	at org.apache.wicket.Component.onBeforeRender(Component.java:3798)
 	at org.apache.wicket.Component.internalBeforeRender(Component.java:986)
 	at org.apache.wicket.Component.beforeRender(Component.java:1022)
 	at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1785)
 	at org.apache.wicket.Component.onBeforeRender(Component.java:3798)
 	at org.apache.wicket.Component.internalBeforeRender(Component.java:986)
 	at org.apache.wicket.Component.beforeRender(Component.java:1022)
 	at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1785)
 	at org.apache.wicket.Component.onBeforeRender(Component.java:3798)
 	at org.apache.wicket.Component.internalBeforeRender(Component.java:986)
 	at org.apache.wicket.Component.beforeRender(Component.java:1022)
 	at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1785)
 	at org.apache.wicket.Component.onBeforeRender(Component.java:3798)
 	at org.apache.wicket.Page.onBeforeRender(Page.java:823)
 	at com.pmease.quickbuild.web.page.BasePage.onBeforeRender(BasePage.java:360)
 	at org.apache.wicket.Component.internalBeforeRender(Component.java:986)
 	at org.apache.wicket.Component.beforeRender(Component.java:1022)
 	at org.apache.wicket.Component.internalPrepareForRender(Component.java:2191)
 	at org.apache.wicket.Page.internalPrepareForRender(Page.java:280)
 	at org.apache.wicket.Component.render(Component.java:2280)
 	at org.apache.wicket.Page.renderPage(Page.java:1035)
 	at org.apache.wicket.request.handler.render.WebPageRenderer.renderPage(WebPageRenderer.java:105)
 	at org.apache.wicket.request.handler.render.WebPageRenderer.respond(WebPageRenderer.java:182)
 	at org.apache.wicket.request.handler.RenderPageRequestHandler.respond(RenderPageRequestHandler.java:147)
 	at org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:719)
 	at org.apache.wicket.request.RequestHandlerStack.execute(RequestHandlerStack.java:63)
 	at org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:210)
 	at org.apache.wicket.request.cycle.RequestCycle.processRequestAndDetach(RequestCycle.java:253)
 	at org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:162)
 	at org.apache.wicket.protocol.http.WicketServlet.doGet(WicketServlet.java:137)
 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
 	at com.pmease.quickbuild.web.MainServlet.service(MainServlet.java:135)
 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
 	at org.eclipse.equinox.http.helper.FilterServletAdaptor$FilterChainImpl.doFilter(FilterServletAdaptor.java:56)
 	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
 	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:365)
 	at org.eclipse.equinox.http.helper.FilterServletAdaptor.service(FilterServletAdaptor.java:37)
 	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
 	at com.pmease.quickbuild.Quickbuild$DisableTraceFilter.doFilter(Quickbuild.java:1186)
 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
 	at org.eclipse.jetty.server.Server.handle(Server.java:499)
 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258)
 	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
 	at java.lang.Thread.run(Unknown Source)
2021-04-06 21:17:05,855 [pool-2-thread-8842] ERROR com.pmease.quickbuild.DefaultBuildEngine - Error processing build request.
 java.lang.RuntimeException: Error executing check condition job.
 	at com.pmease.quickbuild.CheckConditionTask.reduce(CheckConditionTask.java:39)
 	at com.pmease.quickbuild.CheckConditionTask.reduce(CheckConditionTask.java:16)
 	at com.pmease.quickbuild.grid.GridTaskFuture.get(GridTaskFuture.java:168)
 	at com.pmease.quickbuild.DefaultBuildEngine.process(DefaultBuildEngine.java:402)
 	at com.pmease.quickbuild.DefaultBuildEngine.access$000(DefaultBuildEngine.java:148)
 	at com.pmease.quickbuild.DefaultBuildEngine$2.run(DefaultBuildEngine.java:1275)
 	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 	at java.lang.Thread.run(Unknown Source)
 Caused by: com.pmease.quickbuild.QuickbuildException: Failed to run command: git fetch origin develop -q 
 Command return code: 128
 Command error output: fatal: unable to access 'https://dev.azure.com/<url>': The requested URL returned error: 503
 	at com.pmease.quickbuild.execution.Commandline$ExecuteResult.buildException(Commandline.java:517)
 	at com.pmease.quickbuild.execution.Commandline$ExecuteResult.checkReturnCode(Commandline.java:531)
 	at com.pmease.quickbuild.plugin.scm.git.GitCli.fetch(GitCli.java:312)
 	at com.pmease.quickbuild.plugin.scm.git.GitCli.doSync(GitCli.java:45)
 	at com.pmease.quickbuild.plugin.scm.helper.ScmCli.sync(ScmCli.java:65)
 	at com.pmease.quickbuild.plugin.scm.git.GitRepository.getGit(GitRepository.java:487)
 	at com.pmease.quickbuild.plugin.scm.git.GitRepository.getHeadRevision(GitRepository.java:153)
 	at com.pmease.quickbuild.plugin.scm.git.GitRepository$$EnhancerByCGLIB$$afcea5e1.CGLIB$getHeadRevision$29(<generated>)
 	at com.pmease.quickbuild.plugin.scm.git.GitRepository$$EnhancerByCGLIB$$afcea5e1$$FastClassByCGLIB$$97bdbbe2.invoke(<generated>)
 	at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:215)
 	at com.pmease.quickbuild.DefaultScriptEngine$Interpolator.intercept(DefaultScriptEngine.java:261)
 	at com.pmease.quickbuild.plugin.scm.git.GitRepository$$EnhancerByCGLIB$$afcea5e1.getHeadRevision(<generated>)
 	at com.pmease.quickbuild.plugin.scm.git.GitRepository.getHeadRevision(GitRepository.java:52)
 	at com.pmease.quickbuild.plugin.scm.git.GitRepository$$EnhancerByCGLIB$$afcea5e1.CGLIB$getHeadRevision$28(<generated>)
 	at com.pmease.quickbuild.plugin.scm.git.GitRepository$$EnhancerByCGLIB$$afcea5e1$$FastClassByCGLIB$$97bdbbe2.invoke(<generated>)
 	at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:215)
 	at com.pmease.quickbuild.DefaultScriptEngine$Interpolator.intercept(DefaultScriptEngine.java:261)
 	at com.pmease.quickbuild.plugin.scm.git.GitRepository$$EnhancerByCGLIB$$afcea5e1.getHeadRevision(<generated>)
 	at com.pmease.quickbuild.repositorysupport.Repository.takeSnapshot(Repository.java:361)
 	at com.pmease.quickbuild.plugin.scm.git.GitRepository$$EnhancerByCGLIB$$afcea5e1.CGLIB$takeSnapshot$78(<generated>)
 	at com.pmease.quickbuild.plugin.scm.git.GitRepository$$EnhancerByCGLIB$$afcea5e1$$FastClassByCGLIB$$97bdbbe2.invoke(<generated>)
 	at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:215)
 	at com.pmease.quickbuild.DefaultScriptEngine$Interpolator.intercept(DefaultScriptEngine.java:261)
 	at com.pmease.quickbuild.plugin.scm.git.GitRepository$$EnhancerByCGLIB$$afcea5e1.takeSnapshot(<generated>)
 	at com.pmease.quickbuild.setting.configuration.snapshot.TakeReferencedSnapshots.takeSnapshot(TakeReferencedSnapshots.java:15)
 	at com.pmease.quickbuild.model.Configuration.takeSnapshot(Configuration.java:2096)
 	at com.pmease.quickbuild.CheckConditionJob.execute(CheckConditionJob.java:35)
 	at com.pmease.quickbuild.grid.GridJob.run(GridJob.java:131)
 	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
 	at java.util.concurrent.FutureTask.run(Unknown Source)
 	... 3 more

the error in the child configuration looked similar to the one before

2021-04-07_08h30_08.png
robinshen ADMIN ·

QB calls git command line to do the job, and git command line reports 503 service unavailable error. This seems to me more like a Azure issue instead of a QB issue.

SameOldSong ·

You are right. And that was not actually my question. Such sporadic connection issues are impossible to avoid completely. That's why we use "Repeat" inside the steps, so when Git connection problems occur in the steps that try to download sources from Git, we just simply repeat the step and it works wonderfully. At the second attempt the connection issue is usually gone.

But what you see here is another kind problem. We have Git error message on the level of the configuration (not inside the running step). The configuration in the screenshot is triggered from a parent build. And if this error message is displayed in the child configuration, the child configuration fails to be triggered. The parent configuration just reports an error in the triggering step. And even if build in Repeat into this triggering step, it won't help, because as long as the configuration is marked with this error - it can't be triggered. We have to remove the error message manually, and this manual intervention costs us hours of build time. Honestly, we've run out of ideas of any other kind of safety mechanism to fix this issue.

Imagine that such a configuration could be triggered by several parent builds during the night. All of them would fail, just because at some point there was a sporadic Git connection problem. It is highly likely that when another parent build tries to trigger this one, the connection problem is already long gone, but the configuration is still marked with this error message and can not be triggered any more.

Is it possible not to block the entire configuration by sporadic Git connection issues?

robinshen ADMIN ·

Thanks for the detailed explanation. Do understand your concern now. It is actually a bug that QB refuses to re-trigger the same configuration with same variables if this combination has been triggered before. The fix is easy, just add a pseudo variable say "timestamp" in the trigger build step of parent build and set the value as "${new java.util.Date()}".

Also filed a bug to solve this issue in future versions without adding this special variable:

SameOldSong ·

Thank you, I will try your tip. But I'm not sure that this is the core issue here. It seems that if the configuration is marked with this red error message, it can not be triggered any more, from any other configuration, no matter which variables are used. Until we remove the error manually.

Could you, please, check if this is the case?

Also, just wanted to mention that these connection errors displayed on the level of the configuration are much more of a hindrance than a help. Connection to Git is established when the steps inside the configuration that download from repo are actually executed. So it would be much much better if only those steps suffered from connection issues - rather than the entire configuration. Or in other words - we would like the connection issues to occur while the build is running, not prevent the build from starting in the first place.
We've always wondered why QuickBuild has this strange feature. Can it be deactivated somehow?

robinshen ADMIN ·

These errors are a configuration level error instead of step level as they are captured when configuration tries to take snapshot and evaluate build condition to check if the build is necessary. If you do not want these errors to be shown in configuration overview, you may use an empty script for snapshot taking (in advanced setting of configuration) and also specify build condition of the configuration to be "build always".

Also there is no reason that the configuration level error prevents it to be triggered further. The not triggering issue you are seeing is because that in a single build chain (identified a schedule or manual run), a build request with the same configuration and variable combo will only be handled once to maintain artifact/dependency consistency. So this is the reason why I used a timestamp to trick the system to think that subsequent triggering is a different build request.

SameOldSong ·

thank you for the explanation, we will deactivate snapshot taking on the configuration level. This should probably solve the problem

robinshen ADMIN ·

Build condition also needs to be specified as "always", or you may uncheck the "respect build condition" in the trigger build step.

SameOldSong ·

Have implemented all the suggestion. Hoping the problem is gone now