Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

Connection resets when downloading artifacts #4446

bmccart ·

We are having an intermittent problem downloading build artifacts from a composite build that can be observed by running this powershell command on certain of our build agent machines:

Invoke-RestMethod -Uri 'http://qbprod.cxxfxx.com:8810/batch_download?build_id=417046&src_path=artifacts&file_patterns=**&compress=true' -OutFile f:\temp\build.tar -Credential:$cred

The problem comes and goes and also only ever happens on certain of our build-agent machines. At times the above command works, but most times it fails with 1 to 2 seconds with an error such as:

Invoke-RestMethod : Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

It appears that the .zip file starts to download initially but then quickly the connection is dropped and the above error is emitted. I think the same problem is happening during deployments on these agents when the problem manifests like this:

10:33:41,669 INFO - Fetching input files...
10:33:42,759 INFO - Executing post-execute action...
10:33:42,759 ERROR - Step 'master>Run Deploy Steps>Custom PreDeploy>Sequential Pre-Deploy To Servers steps>Pre-Deploy ServerGroup?ServerGroupInfo=WebTierExternal:PLBUWEBHQDTB005>Pre-Deploy Parallel Grouped Servers>Pre-Deploy Server?APPSERVER=PLBUWEBHQDTB005&ServerGroupName=WebTierExternal' is failed.
java.lang.RuntimeException: java.net.SocketException: Connection reset
at com.pmease.quickbuild.util.FileUtils.untar(FileUtils.java:908)
at com.pmease.quickbuild.grid.CopyFilesJob.execute(CopyFilesJob.java:122)
at com.pmease.quickbuild.grid.GridNode.executeJob(GridNode.java:497)
at com.pmease.quickbuild.grid.GridImpl.transferFiles(GridImpl.java:161)
at com.pmease.quickbuild.BuildAwareJob.fetchInputFiles(BuildAwareJob.java:121)
at com.pmease.quickbuild.stepsupport.Step.doExecute(Step.java:673)
at com.pmease.quickbuild.stepsupport.Step.execute(Step.java:577)
at com.pmease.quickbuild.stepsupport.StepExecutionJob.executeStepAwareJob(StepExecutionJob.java:31)
at com.pmease.quickbuild.stepsupport.StepAwareJob.executeBuildAwareJob(StepAwareJob.java:56)
at com.pmease.quickbuild.BuildAwareJob.execute(BuildAwareJob.java:77)
at com.pmease.quickbuild.grid.GridJob.run(GridJob.java:131)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.SocketException: Connection reset
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
at java.base/sun.net.www.http.ChunkedInputStream.fastRead(ChunkedInputStream.java:244)
at java.base/sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:689)
at java.base/java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.base/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3495)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:290)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
at org.apache.commons.compress.archivers.tar.TarBuffer.readBlock(TarBuffer.java:224)
at org.apache.commons.compress.archivers.tar.TarBuffer.readRecord(TarBuffer.java:195)
at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:486)
at java.base/java.io.InputStream.read(InputStream.java:205)
at com.pmease.quickbuild.util.FileUtils.untar(FileUtils.java:895)

I am interested in any hints about how to debug this on our QuickBuild server. So far I've found nothing in any QB server log that I can correlate to these REST requests to download the zip. Is there any place that I can look? I suspect that QuickBuild always dynamically produces the .zip for each REST request to download it. Am I correct? I further suspect that QuickBuild is somehow failing during that process when we get these errors but I don't know how to zero in on it and prove it. Any help is appreciated.

  • replies 1
  • views 718
  • stars 0
bmccart ·

It turned out that it was some security stuff detecting a false positive and dropping the socket on both sides.