Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

Select node by left free disk space #3969

Henry ·

Hi,

i have a build, where I have to select two nodes (one windows, one linux node) out of about 20 to start two builds which each consume about 200 GB of disk space on that node.

Thats absolutley not a problem if the nodes have been freshly and utterly cleaned, but if some other builds already have run, the node may not have 200 GB space left. Still the build starts and then fails after some hours with "out of disk space error".

I tried to use a script in the node-selection to detect if the candidate node for such a build has enough disk space left. But currently the script does only detect the disk space on the quickbuild-server and never on the candidate node.

I suppose the node-selection-code is always executed on the qb-server. Is that right? Therefore my approach did not work?

Can I somehow select only nodes that have enough disk space? by script? Or somewhat?

Please let me know and many thanx in advance
Henry

  • replies 6
  • views 140
  • stars 0
robinshen ADMIN ·

Unfortunately there is no easy way to do this. Running node selection logic at agent side will be problematic as it will slow down the node selection logic considerably if there are many build steps and nodes.

drdt ·

I wanted to submit my solution to this problem. In summary:

  • we have added to the end of every build, a step which measures the disk space remaining on the workspace drive and stores it as a node attribute FREE_SPACE

  • we have added to the node-selection script the following condition:

    node.getAttribute( "FREE_SPACE" ) >= vars.getValue( "SPACE_NEEDED" );

    We are working on being able to calculate SPACE_NEEDED according to which branch is being built, but for now it is hard coded at 60 (gig).

  • we have created a configuration, that runs on nodes where FREE_SPACE < SPACE_NEEDED, to run automatic disk cleanup.

The weakness I have identified in this solution is that, when a build is running on one of two agents running on a single node (ports 8811 and 8812), it is non-trivial to update the FREE_SPACE attribute on the other agent.

robinshen ADMIN ·

Thanks for sharing. Maybe two file systems are desired in this case, one for each agent.

drdt ·

That is not the worst idea. However, it kind of leads me to think... why am I running two agents on the same node anyway?

Thanks!

robinshen ADMIN ·

This mainly because that a single build agent only has one workspace for a configuration. To run multiple builds from same configuration concurrently on a single node, multiple build agents have to be installed.

drdt ·

I apologize, that was a rhetorical question. I am questioning my design decisions out loud.