Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

Pausing system also pauses running builds #4109

uartie ·

In QB 9.0.9, when we "Pause System" in "Administration->System Setting", running builds finish their current step but the next step in the build gets stuck with "waiting" status and the information link states "unknown reason". This did not happen in our previously installed QB version 8.0.36. Did this policy change in new versions? On QB 9.0.9 "System Settings" page, the description still says "Check this to pause the system to prevent new builds from running (already running builds will not be affected)."

qb-system-pause-step-waits.jpg
  • replies 10
  • views 1024
  • stars 1
robinshen ADMIN ·

The description of pausing system is not accurate. It should be "already running steps will not be affected". In QB8 the step processing has a bug and it can bypass this setting when it uses the node selection setting "use same node as parent step", and it was fixed in QB9.

uartie ·

Bummer. We relied on the previous functionality quite heavily. That is, occasionally we need to take an agent offline for maintenance... so we would pause the system until that agent finishes any builds it is currently running. Now we can't do this anymore?

uartie ·

Maybe it would be nice if there was a feature that allowed us to pause an agent from starting any new "builds" (but finish current build)?

uartie ·

It should be "already running steps will not be affected".

Unless it's a "composite step".

Laba42 ·

Hey,
we realize it as follows.
We have two nodes are numbered NODE01 and NODE02. In the Masterstep we have defined the agents selection as follows
AgentSelction.png

The variable **QBNodes **is normally equal to NODE0. If we now have to shut down one agent e.g. NODE01, for updates, we set the variable QBNodes to the other agent name (e.g. NODE02) and for the following builds only the agent NODE02 is taken and the other agent runs empty.

uartie ·

That's actually a good solution, Thanks!

I think the way we'll do it is via "User Attributes" of the nodes and use the following criteria:
qb-node-maintenance.jpg

This way, all we have to do is define "MAINTENANCE" in the node "User Attributes" when we want to shut it down but let it finish current job.

robinshen ADMIN ·

You may just stop the build agent gracefully (sudo service buildagent stop) without pausing the system. QB will not dispatch new builds to the agent, but will make sure that all currently running builds completes on the stopping agent.

drdt ·

This has been an issue since version 4.0 at least. I may have filed a ticket for it once upon a time.

The workaround my team used was as follows:

  • create a root-level variable systemShutDownPending=false
  • create a root-level pre-queue script:
groovy: vars.get("systemShutDownPending").asBoolean() ? false : true;

By setting systemShutDownPending=true, no new builds will start (either by schedule or by hand), but currently-running builds will continue to completion. You could also add a maintenance configuration to toggle this value.

Michaelcc73 ·

Are there any plans to add an option on maintenance mode to allow builds that have started to run to completion? Our main use-case for maintenance mode has been for QB upgrades and plugin updates/installation. Since they system cannot come down when a build is in progress maintenance mode in its current form cannot be used. While some of the other options outlined in the thread seem to work, they can present their own challenges.

robinshen ADMIN ·

In this case, you may just stop QB service. QB will wait for all running builds to finish before shutting down, not yet running in queue will be picked up for running after restart. You may test some example builds to see how it works.