How can i get pubdirectory and artifacts with rest api?
#4546
With groovy, you can retrieve the artifacts and pubdiectory location of a specific build using the code below.
abuild = system.buildManager.load(build id)
publocation = abuild.getPublishDir()
dirlocation = abuild.getArtifactDir()
However, rest api has no explanation about its contents.
How can I retrieve the above two using rest api?
We are deleting logs and ARTIFACTS. Quick Build supports its own, but it does not match the settings we want, so we are writing it as an internal GROOVY SCRIPT and deleting it.
However, loading one that has 450,000 builds will overload the CPU and crash the system.
We have been working to resolve this. One solution is to look up the object to be deleted in the DB, load the objects to be deleted one by one, retrieve the locations of PUBDIRECTORY and ARTIFACTS, and then delete them.
However, this was not a perfect solution, so an attempt was made to artificially make the task work on only one THREAD.
However, in the inner script, "ExecutorService executorService = Executors.newSingleThreadExecutor();" When you write a program, the executorService value returns NULL, so you cannot write a program that uses one THREAD. However, if you change the step to "Build > Shell/Batch Command" and execute it by giving a command like "groovy /root/test.groovy", the executorService value returns as normal, so THREAD programming appears to be possible.
---------------- SAMPLE CODE (Misc > Execute Script) --------------
groovy:
import com.pmease.quickbuild.*;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
Runnable run = {
def db2 = groovy.sql.Sql.newInstance("jdbc:mysql://localhost:3306/quickbuild","root","YOUR PASSWD","com.mysql.jdbc.Driver");
List foodList = db2.rows('select * from QB_BUILD');
foodList.each{ row -> System.out.println(row.QB_ID) }
}
ExecutorService executorService = Executors.newSingleThreadExecutor();
logger.info("{}", System.out.println(executorService));
executorService.execute(run);
executorService.awaitTermination(10, TimeUnit.SECONDS);
executorService.shutdown();
THREAD must be used, and for this, it must be executed in the "Shell/Batch Command" method, but it was said that the system class is not supported in external script execution, so I tried to learn about PUBDIRECTORY and ARTIFACTS in the REST API.
When I set the step to "Misc > Execute Script" and use THREAD, "ExecutorService executorService = Executors.newSingleThreadExecutor();" returns null when I use this syntax. Is this intended in quickbuild? Or is there another way?
The build step itself running your groovy script runs already in a single thread, why you need to run this script in a separate single thread?
Also the built-in cleanup logic only loads build ids into memory and then delete them one by one if no longer needed, which should not consume that much of memory. Which QB version are you using?
I am using version 11.0.26.
The system has 16 core, jvm 20GB, and physical memory is 60GB.
If STEP itself already operates only with a single thread, you will not see 1300% CPU usage when loading a build with 450,000 build history. However, it shows such a usage rate. In other words, using 1300% means using more than 13 CORE and is absolutely not using a single thread. Perhaps STEP itself is SINGLE THREAD, but the function used internally seems to cause this use.
We are experiencing a huge load from loading the build itself. Therefore, we deleted some of the 450,000 builds and reduced them by half. As a result, the load on the system was greatly reduced. And as a different solution, I modified and tested the work target by searching the DB and loading it one by one to work on it.
The built-in cleanup logic simply deletes by date, which is not satisfactory for the logic we want to delete. So don't use it.
Below is the code that causes the load
If root/test has about 450,000 builds, it causes huge CPU usage when doing getBuilds.
We absolutely want to make the process work as 1 SINGLE CORE.
Can I know the TABLE and FIELD of the DB that stores the ARTIFACTS and PUBDIRECTORY locations? If we could know the information, I think we could handle it ourselves.
You definitely can not call system.buildManager.getBuilds(targetConf) for configuration with so many builds, as it will load all build records into memory.
QB's build cleanup strategy only loads build ids into memory and then load build record one by one and discard processed records. The build cleanup strategy also has a preserve condition if the default one (preserve by date or count) does not satisfy your needs.
If you'd like to clean by yourself, the build publish/artifacts directory is derived from build id like below:
File prefixDir = new File(storageDir, "builds/b" + String.format("%03d", id%1000));
File publishDir = new File(prefixDir, String.valueOf(id));
File artifactsDir = new File(publishDir, "artifacts");I was curious about how the build path is determined, so thank you for letting me know. I have no further questions on this matter. Thank you for the good answer.