Which QB version are you using?

The latest 8 QB version.

Thanks for the info. Got fixed now in QB 8.0.42:

Thank you so much for your quickly support.

Dear Mr.@robinshen,
As I checked, I would like to share you some result and you can consider some problem:

• 1: Set Disconnect Tolerance variable > 2:15 (~140s), If this value < 140s, It seem not working.
• 2: Time to wait for re-connect = Disconnect Tolerance value. But only pass (waiting successfully) during step had not done yet.
• 3: If agent never re-connect, build will finish and be fail after waiting (Disconnect Tolerance + 2:20 (~140s)).
• 4: Only apply for Master step, not for any child step. Implement this variable for any child step is not working, alway got fail when reconnect or after ~140s.

And we are being faced a big problem, build will be failed with error log Unable to find job on node when job at agent has been done during master-agent disconnected and reconnect after.
I think we should improve this case, if job has been done at agent, it should wait for in period defined at Disconnect Tolerance value before finished job.
Thank you so much!

The disconnect tolerance does not guarantee build can tolerate all network disconnections as that will be very difficult. Instead it solves the most common issue: when a step runs for a long time, and if during that wait time, network disconnects and reconnect later, QB should ignore network heartbeat failures and should not mark the step as failed.

If network is disconnected while child step communicates with parent step (this will happen when step is finishing, child step is starting, etc), the build will still fail.

Excuse me, We'd like to you share why it fails after ~140s even it is set on child step?

Do you mean if this setting is specified as 140s on a child step, the child step will fail after 140s even if network is not disconnected?

Sorry, I mean if Disconnect Tolerance is setting any value excluding 0 for child step, the child step will fail after 140s even if network is re-connected.

I tried below and it works:

1. Create a test configuration with two steps. Master step runs on server, and a child step simply running sleep command for 300 seconds on an agent.
2. Define network disconnect tolerance as 120s on the child step.
3. Now run the build, plug out cable of agent when build runs for 60s
4. Wait for another 60s, and plugin cable in.
5. Build still succeeds when child steps reaches 300s

Can you please try this set up at your side to see if it works?

Oh, I checked again and Realize that the child step will fail after 140s not for all cases:
I set Disconnect Tolerance = 300
1 - with Thread.sleep(milliseconds)
groovy:
Thread.sleep(300000)
It's only success if disconnect-time < ~25s
2 - with sleep(milliseconds)
groovy:
sleep(300000)
It's only success if disconnect-time < ~150s
3 - with sync step (Step type: checkout)
It's only success if disconnect-time < ~14s
=> And Disconnect Tolerance variable seems not working with child step. Some child step will alway fail even if network is re-connected. Please check again for us. Thank you.

What is your server and agent OS? And how do you disconnect network for testing? I am unplugging the cable. Also please test with a brand new QB installation, and send me [robin AT pmease DOT com] the database backup.

They all are Ubuntu. I unplugged the cable too. (Or disable Ethernet port). I am using 8.0.42. We are unable to share data, please understand for us about that.
I will test again with Disconnect Tolerance in child step. Please use sync (Step type: checkout) and run a bash shell, run a scrip step to testing again instead of sleep step. Please share for us why the time to wait for accepting reconnect depended on type of step?
Thank you.

I do not want you to share your data. Just that you set up a test QB instance with minimum settings to demonstrate the issue, and send me the backup.

Step differs as "sleep" in groovy script is not interruptible while "Thread.sleep" is interruptible. To isolate the issue, please use Thread.sleep for all your testing.

We are unable to share any data, file, document... Please understand for us.

1. I implement sync step for sync ~200Gb data while ~ 20min
   The step:

<?xml version="1.0" encoding="UTF-8"?>
 
<com.pmease.quickbuild.repositorysupport.CheckoutStep revision="0.15.0.1">
  <name>Sync</name>
  <inheritable>true</inheritable>
  <enabled>true</enabled>
  <executeCondition class="com.pmease.quickbuild.setting.step.executecondition.AllPreviousSiblingStepsSuccessful"/>
  <nodeMatcher class="com.pmease.quickbuild.setting.step.nodematcher.ParentNodeMatcher"/>
  <nodePreference class="com.pmease.quickbuild.setting.step.nodepreference.PreferLeastLoadedNode"/>
  <timeout>0</timeout>
  <disconnectTolerance>300</disconnectTolerance>
  <showLinksInLog>false</showLinksInLog>
  <preExecuteAction class="com.pmease.quickbuild.setting.step.executeaction.NoAction"/>
  <postExecuteAction class="com.pmease.quickbuild.setting.step.executeaction.NoAction"/>
  <repetitions/>
  <repositoryName>MAIN</repositoryName>
</com.pmease.quickbuild.repositorysupport.CheckoutStep>

2. Define network disconnect tolerance as 300s on the this child step.
3. Run the build, Disable network (plugout cable) of agent when build runs for 60s

• 4.1. Never re-connect => Build had been fail after ~3m15s (after 2m15s from disconnecting) caused by: com.caucho.hessian.client.HessianRuntimeException: Error connecting 'http://serverURL/service/server'
• 4.2. Wait for < ~10s, and Enable network (plugin cable) => Build had been successful.
• 4.3. Wait for > ~10s, and Enable network (plugin cable) => Build had been alway fail. caused by: com.caucho.hessian.client.HessianRuntimeException: Error connecting 'http://serverURL/service/server'

If it is a checkout step, the copy operation itself will fail when cable is plugged out. Network disconnect tolerance only makes sure that QB can tolerate server/agent heartbeat failures. It can not recover the step from its own network operation failures. To verify, you may change the step to a simple sleep step (calling groovy Thread.sleep or calling external sleep command via Command/Batch execution step).

I got it, ok I will check more. Thank you.

Sorry. It does not work with sleep step:

<?xml version="1.0" encoding="UTF-8"?>
<com.pmease.quickbuild.plugin.basis.ScriptStep revision="0.15.1">

  <name>Slepp</name>
  <inheritable>true</inheritable>
  <enabled>true</enabled>
  <executeCondition class="com.pmease.quickbuild.setting.step.executecondition.AllPreviousSiblingStepsSuccessful"/>
  <nodeMatcher class="com.pmease.quickbuild.setting.step.nodematcher.ParentNodeMatcher"/>
  <nodePreference class="com.pmease.quickbuild.setting.step.nodepreference.PreferLeastLoadedNode"/>
  <timeout>0</timeout>
  <disconnectTolerance>240</disconnectTolerance>
  <showLinksInLog>false</showLinksInLog>
  <preExecuteAction class="com.pmease.quickbuild.setting.step.executeaction.NoAction"/>
  <postExecuteAction class="com.pmease.quickbuild.setting.step.executeaction.NoAction"/>
  <repetitions/>
  <script>groovy:
Thread.sleep(300000)</script>
</com.pmease.quickbuild.plugin.basis.ScriptStep>

Sleep 300s, disconnect tolerance as 240s.
If network never re-connect => Build had been fail after ~3m15s (after 2m15s from disconnecting) caused by: com.caucho.hessian.client.HessianRuntimeException: Error connecting 'http://serverURL/service/server'
If I wait for < ~10s from disconnect, and Enable network (plugin cable) => Build had been successful.
If I wait for > ~10s from disconnect, and Enable network (plugin cable) => Build had been alway fail. Caused by: com.pmease.quickbuild.QuickbuildException: Unable to find job '7f147565-99fa-4f9e-96d4-fdc08f14b20f' on node 'AgentUbuntu_18.04:8814'.

Please not that QB is able to recover the step from its own network operation failures in case we set Disconnect Tolerance for Master step instead of child step.

Please start with a new QB instance and follow steps below to see if it works:

1. Create a test configuration with two steps. Master step runs on server, and a child step simply running sleep command for 300 seconds on an agent.
2. Define network disconnect tolerance as 240s on the child step.
3. Now run the build, plug out cable of agent when build runs for 60s
4. Wait for another 60s, and plugin cable in.
5. Build still succeeds when child steps reaches 300s

Ah, I really understand why we have different result testing.

• Your scenario whom Master step run on server, and me no, my Master step run on agent which runs other steps.
• I tested with your scenario and it worked well when the network reconnect.
  But I don't know why it had not worked yet when Master Step run on agent? Please share for us know.

Currently, our system is servicing about 10k user, 2k agents connected, so we must decrease offload for server, the master step should run on agent. Please consider about that.
Thank you so much!

If master step runs on agent, and if agent can be disconnected from server, you will need to set disconnect tolerance on master step also. This setting should be specified on any step whose running node can be disconnected from other nodes.