Advice on running build step where connection may temporarily stop #3816

roneystein · 7 years ago

Using Quickbuild to run black box tests, some of them may cause the network to be temporarily (2 ou 3 seconds) unavailable. During that the agent sometimes lose connection to the server.
Does anyone use any technique to workaround this kind of issue or is there any configuration possible to make the Qb server to not abort the build once it finds the agent is no longer accessible? Of course, the agent reconnects itself but then the server is already canceling the build. From the logs it looks like a timeout has occurred, but once you look at the agent log you find the network is not ok.

replies 5
views 777
stars 0

robinshen ADMIN · 7 years ago

Please submit an improvement request at track.pmease.com to make the connection loss tolerance configurable.

roneystein · 7 years ago

Hello Robin.
Request: http://track.pmease.com/browse/QB-3023
I'm currently playing with tcp/ip stack parameters to avoid this scenario.
I'll post here if I get good results.

greha · 7 years ago

Hi,

we usually have the master step running on a node which is always available and for the steps where the agent might be offline we use resources.
QB then waits until the resource is available - in our case we start/stop/revert VMs .
This works pretty good , sometimes we get "Connection timed out: connect" errors but I think those are related to synching issues with windows because VM snapshots might be older

roneystein · 7 years ago

After some additional troubleshooting I've found that when the agent detects a connection failure it cancels the step.
TCP parameters were not enough. The solution was running the Step asynchronously (not waiting for it to finish) and another step composition to pool if the previous step finished.

drdt · 7 years ago

I am only aware of one build environment tool that can handle this situation; in fact its ability to handle this situation is the most meaningful difference between it and QuickBuild. However, the cost is nearly 100x the cost of QuickBuild for that one feature.