Build stuck at CHECKING_BUILD_CONDITION #2903

Navezof · 1 decade ago

Hello,

One of my build is stuck while process status is CHECKING_BUILD_CONDITION

This happen every time I have a step Repository>Checkout in my build. If I disable this step, the build works just fine. (I'm using a perforce repository)

I'm using the version 5.1.31 of Quickbuild. I'm using mysql-connector-java-5.1.31.

The error that puzzle me the most is this one :

2014-07-22 09:57:21,303 [MeasurementPollingTask] ERROR com.google.common.eventbus.EventBus.QuickBuild Application - Could not dispatch event: [source=[PDC-BLD-074:8810], metricName=[jvm.memoryUsage], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.3292278102907736], source=[PDC-BLD-074:8810], metricName=[jvm.heapUsage], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.34629436327427837], source=[PDC-BLD-074:8810], metricName=[jvm.threadCount], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[178.0], source=[PDC-BLD-074:8810], metricName=[jvm.gcRuns], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[9.0], source=[PDC-BLD-074:8810], metricName=[jvm.gcTime], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[34.0], source=[PDC-BLD-074:8810], metricName=[cpu.totalUsage], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.009017405424824463], source=[PDC-BLD-074:8810], metricName=[disk.usage.C:], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.09716821440426923], source=[PDC-BLD-074:8810], metricName=[memory.usage], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.1386687543403587], source=[PDC-BLD-074:8810], metricName=[swap.usage], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.07985662857161975], source=[PDC-BLD-074:8810], metricName=[web.ui.oneMinuteRate], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.20788134509425873], source=[PDC-BLD-074:8810], metricName=[web.ui.fiveMinuteRate], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[4.7399537753577405], source=[PDC-BLD-074:8810], metricName=[web.ui.fifteenMinuteRate], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[4.554502407283335], source=[PDC-BLD-074:8810], metricName=[web.ui.requests], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[11.0], source=[PDC-BLD-074:8810], metricName=[web.ui.averageDuration], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[34.36778725539388], source=[PDC-BLD-074:8810], metricName=[web.rest.oneMinuteRate], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.0], source=[PDC-BLD-074:8810], metricName=[web.rest.fiveMinuteRate], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.0], source=[PDC-BLD-074:8810], metricName=[web.rest.fifteenMinuteRate], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.0], source=[PDC-BLD-074:8810], metricName=[web.rest.requests], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.0], source=[PDC-BLD-074:8810], metricName=[web.rest.averageDuration], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.0], source=[PDC-BLD-074:8810], metricName=[web.rpc.oneMinuteRate], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[89.24108019157747], source=[PDC-BLD-074:8810], metricName=[web.rpc.fiveMinuteRate], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[89.9029322543982], source=[PDC-BLD-074:8810], metricName=[web.rpc.fifteenMinuteRate], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[92.09278418247081], source=[PDC-BLD-074:8810], metricName=[web.rpc.requests], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[451.0], source=[PDC-BLD-074:8810], metricName=[web.rpc.averageDuration], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.44285312842178703], source=[PDC-BLD-074:8810], metricName=[web.download.oneMinuteRate], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[1.77863632503E-312], source=[PDC-BLD-074:8810], metricName=[web.download.fiveMinuteRate], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[3.3637884981424677E-90], source=[PDC-BLD-074:8810], metricName=[web.download.fifteenMinuteRate], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[4.335415892572229E-30], source=[PDC-BLD-074:8810], metricName=[web.download.requests], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.0], source=[PDC-BLD-074:8810], metricName=[web.download.averageDuration], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[24.859424920127793], source=[PDC-BLD-074:8810], metricName=[web.upload.oneMinuteRate], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[5.709052965495541E-19], source=[PDC-BLD-074:8810], metricName=[web.upload.fiveMinuteRate], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[6.724229508813826E-4], source=[PDC-BLD-074:8810], metricName=[web.upload.fifteenMinuteRate], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.12190335062684983], source=[PDC-BLD-074:8810], metricName=[web.upload.requests], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.0], source=[PDC-BLD-074:8810], metricName=[web.upload.averageDuration], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[19.06315789473684], source=[PDC-BLD-074:8810], metricName=[jetty.usedConnections], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[4.0], source=[PDC-BLD-074:8810], metricName=[jetty.totalConnections], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[64.0], source=[PDC-BLD-074:8810], metricName=[builds.total], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.0], source=[PDC-BLD-074:8810], metricName=[builds.failed], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.0], source=[PDC-BLD-074:8810], metricName=[builds.successfulRate], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.0], source=[PDC-BLD-074:8810], metricName=[builds.averageDuration], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.0], source=[PDC-BLD-074:8810], metricName=[builds.averageWaitingDuration], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.0], source=[PDC-BLD-074:8810], metricName=[buildrequests.processedTotal], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.0], source=[PDC-BLD-074:8810], metricName=[buildrequests.processingsTotal], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[3.0], source=[PDC-BLD-074:8810], metricName=[buildrequests.waitingsTotal], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.0], source=[PDC-BLD-074:8810], metricName=[steps.runningsTotal], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[8.0], source=[PDC-BLD-074:8810], metricName=[steps.waitingsTotal], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[10.0], source=[PDC-BLD-074:8810], metricName=[resources.frees.MachinePool_Clean], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.0], source=[PDC-BLD-074:8810], metricName=[resources.used.MachinePool_Clean], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[1.0], source=[PDC-BLD-074:8810], metricName=[buildrequests.processings], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[3.0], source=[pdc-prd-wd12:8811], metricName=[steps.runnings], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[2.0], source=[PDC-BLD-074:8810], metricName=[steps.runnings], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[6.0], source=[PDC-BLD-074:8810], metricName=[db.busyConnections], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[0.0], source=[PDC-BLD-074:8810], metricName=[db.numConnections], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[5.0], source=[PDC-BLD-074:8810], metricName=[network.txBytes], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[2697828.0], source=[PDC-BLD-074:8810], metricName=[network.rxBytes], timestamp=[Tue Jul 22 09:57:21 CEST 2014], value=[1947414.0]] to handler [wrapper public void com.pmease.quickbuild.plugin.measurement.core.poller.MeasurementDataHandler.onReceivedMeasurements(java.util.List)]
    java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedMethodAccessor220.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:74)
        at com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
        at com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
        at com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
        at com.google.common.eventbus.EventBus.post(EventBus.java:267)
        at com.pmease.quickbuild.plugin.measurement.core.poller.MeasurementPollingTask.execute(MeasurementPollingTask.java:56)
        at com.pmease.quickbuild.plugin.measurement.core.poller.MeasurementPollingTask.run(MeasurementPollingTask.java:108)
        at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52)
        at java.lang.Thread.run(Unknown Source)
    Caused by: java.lang.NullPointerException
        at com.pmease.quickbuild.plugin.measurement.core.poller.MeasurementDataHandler.getAlertDefinitions(MeasurementDataHandler.java:143)
        at com.pmease.quickbuild.plugin.measurement.core.poller.MeasurementDataHandler.evaluteAlerts(MeasurementDataHandler.java:63)
        at com.pmease.quickbuild.plugin.measurement.core.poller.MeasurementDataHandler.onReceivedMeasurements(MeasurementDataHandler.java:50)
        ... 12 more

This error repeat itself all along the log.

Thanks a lot for your help, and have a good day.

Ps : Here is the link to the full log : Full log

replies 41
views 35133
stars 0

robinshen ADMIN · 1 decade ago

Build stuck should not be caused by the error you saw from log (of course we will look into that issue also). Most probably the p4 process spawned by QB is blocked waiting for something (such as credentials, etc). To investigate this, please check on which node the build condition checking is taking place (it will run on the node defined to run master step of the build), and then modify "conf/log4j.properties" of that node to enable debug logging:
log4j.logger.com.pmease.quickbuild=DEBUG

Then run the build again and check console log of that node to see which p4 command is blocking the build, and then run that p4 command from command line as the same user running QB process to see if it exhibits the same behavior.

steveluo ADMIN · 1 decade ago

The root cause of the error aforementioned is Alert plugin is disabled, so please go to Administration page -> Plugin Management tab and enable the Alert plugin.

Also, this is a bug and I have filed the issue below, it will get fixed in next patch release.

http://track.pmease.com/browse/QB-2137

Navezof · 1 decade ago

Hello,

Thanks for the anwsers

The root cause of the error aforementioned is Alert plugin is disabled, so please go to Administration page -> Plugin Management tab and enable the Alert plugin.

Indeed, I re-enabled the plugin and I didn't have the error anymore. Thanks.

Then run the build again and check console log of that node to see which p4 command is blocking the build, and then run that p4 command from command line as the same user running QB process to see if it exhibits the same behavior.

I activated the debug and it seems to come from the very first command. After trying to :

jvm 1    | 2014-07-23 14:25:30,019 DEBUG - Starting new round of resource allocation request processing...
jvm 1    | 2014-07-23 14:25:30,019 DEBUG - Calculating current resource usages...
jvm 1    | 2014-07-23 14:25:30,029 DEBUG - Allocating resources for waiting steps...

In loop for an hour or two, I get the following error line.

com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure.

And then it goes again on starting round of resource allocation. Are you familiar with this error?

I tried the blocking command on the command prompt, and it blocks again. So I guess the error comes from my side <img src="{SMILIES_PATH}/icon_smile.gif" alt=":)" title="Smile" />

robinshen ADMIN · 1 decade ago

I tried the blocking command on the command prompt, and it blocks again

What blocking command do you mean? I did not see them from your response. The MySQL JDBC error does matter, can you please check if connection to MySQL is stable.

Navezof · 1 decade ago

[quote="robinshen"]What blocking command do you mean? I did not see them from your response.[/quote]
My bad, I didn't put it here. This is a command launched by quickbuild during the step Checkout. It look like this :

p4 -p p4-code -u guest -c qb-pc-074 client -i

When I try this command on a command prompt it just seems to freeze and do nothing.

[quote="robinshen"]The MySQL JDBC error does matter, can you please check if connection to MySQL is stable[/quote]
I have other tasks on quickbuild which are working just fine, even though none use a checkout step. It is also possible to access to the database with an extern software.

Thanks again,

vanderhu · 1 decade ago

This is the same issue as we are facing already for quite some time. Only way to get out of it, is by restarting the agent.

robinshen ADMIN · 1 decade ago

What OS the problematic agent is running on? And which version of Perforce are you using? Also QB will print out the client spec feeding into this command (p4 client -i) just before printing out the command. Can you please show me that?

Navezof · 1 decade ago

The problematic agent is running on Windows 7.
The perforce server version is : P4D/LINUXX26X86_64/2013.2/872110 (2014/06/04)

And here are the client spec.

jvm 1    | 2014-07-28 09:52:15,427 DEBUG - Client spec:
jvm 1    | Client: qb-PDC-BLD-074-9
jvm 1    | Owner: guest
jvm 1    | Host: PDC-BLD-074
jvm 1    | Root: D:\projects\mrc-dlc-package
jvm 1    | SubmitOptions: revertunchanged
jvm 1    | Options: noallwrite clobber nocompress unlocked nomodtime normdir
jvm 1    | LineEnd: local
jvm 1    | Description:
jvm 1    | 	Created by QuickBuild for purpose of running configuration 'root/continuous-package-dlc-solo-proto-bis'.
jvm 1    | View:
jvm 1    | 	//driver-branches/dlc01/... //qb-PDC-BLD-074-9/...
jvm 1    | 
jvm 1    | 2014-07-28 09:52:15,428 DEBUG - Executing command: p4 -p p4-dunia-code-mrc:2326 -u guest -c qb-PDC-BLD-074-9 client -i 
jvm 1    | 2014-07-28 09:52:15,428 DEBUG - Command working directory: C:\quickbuild-5.1.31\bin

I hope this is what you asked.

This is the same issue as we are facing already for quite some time. Only way to get out of it, is by restarting the agent.

Infortunatly, restarting the agent doesn't change anything for us. But thanks for the tips.

#10

robinshen ADMIN · 1 decade ago

Please login to the problematic agent machine and run below command:

p4 -p p4-dunia-code-mrc:2326 -u guest -c qb-PDC-BLD-074-9 client -i

Then from standard input, input below:

Client: qb-PDC-BLD-074-9
Owner: guest
Host: PDC-BLD-074
Root: D:\projects\mrc-dlc-package
SubmitOptions: revertunchanged
Options: noallwrite clobber nocompress unlocked nomodtime normdir
LineEnd: local
Description:
       Created by QuickBuild for purpose of running configuration 'root/continuous-package-dlc-solo-proto-bis'.
View:
       //driver-branches/dlc01/... //qb-PDC-BLD-074-9/...
Ctrl-Z

After pressing "ctrl-z" followed by a return, will the p4 command quit successfully?

If yes, please run QB agent in console mode by issuing "agent console" (shutdown QB agent service first of course), and then run the build again to see if it still hangs.

#11

Navezof · 1 decade ago

I ran the command : p4 -p p4-dunia-code-mrc:2326 -u guest -c qb-PDC-BLD-074-9 client -i on the problematic agent.

Then I interupted it with ctrl + z.
I got the following messages :

^Z
Error in client specification.
Missing required field 'Client'.

The command quit succesfully (meaning it doesn't crash my command prompt, is this what you meant with quit succesfully?)

The problematic agent is the server. So, I launch server.bat console
I then start the problematic build.
The build is still stuck at : "CHECKING_BUILD_CONDITION"
in the console prompt are disaplayed the same lines as in the log I shown you earlier.

Do you think that the problem is caused by the error I got after interupting p4 -p p4-dunia-code-mrc:2326 -u guest -c qb-PDC-BLD-074-9 client -i? Any other ideas?

#12

robinshen ADMIN · 1 decade ago

I'd to check this issue online. Are you able to schedule a WebEx or GotoMeeting session so that we can check this issue lively?

#13

Navezof · 1 decade ago

I'm really grateful for your offer but infortunatly my company have a rather stric confidentiality politic so I can't do video screensharing live. But a call or a chat is fine though <img src="{SMILIES_PATH}/icon_smile.gif" alt=":)" title="Smile" />

Thanks

#14

robinshen ADMIN · 1 decade ago

That is a pity. I'd like to run some tests directly on the machine... If this is not possible, let's continue with this thread...

1. Please login to your QB server machine, and run below command:
p4 -p p4-dunia-code-mrc:2326 -u guest login -p
Then input password of user "guest". This should remain the same when you configure password in Perforce repository defininition in QB. If correct, Perforce should be able to print a hex string, and let's name it <p4_ticket>
2. Create a test configuration in QB, to run step of type "Misc->Execute Script" on server, and script to be executed is as below:

groovy:
import com.pmease.quickbuild.execution.*;

String clientSpec = """
Client: qb-PDC-BLD-074-9
Owner: guest
Host: PDC-BLD-074
Root: D:/projects/mrc-dlc-package
SubmitOptions: revertunchanged
Options: noallwrite clobber nocompress unlocked nomodtime normdir
LineEnd: local
Description:
       Created by QuickBuild for purpose of running configuration 'root/continuous-package-dlc-solo-proto-bis'.
View:
       //driver-branches/dlc01/... //qb-PDC-BLD-074-9/...
"""

def cmd = new Commandline("p4 -p p4-dunia-code-mrc:2326 -u guest -P <p4_ticket> -c qb-PDC-BLD-074-9 client -i");
cmd.execute(new LineConsumer.InfoLogger(), new LineConsumer.WarnLogger(), clientSpec).checkReturnCode();

Here <p4_ticket> should be replaced with ticket p4 printed in step 1.

3. Run the test configuration and check if it will hang.

#15

Navezof · 1 decade ago

I know, and thanks again for your help.

1. When I run this command : p4 -p p4-dunia-code-mrc:2326 -u guest login -p
I got the message : 'login' not necessary, no password set for this user.

2. I created nevertheless the step with the script. But I removed on the commandLine -P <p4_ticket>

Below the final script with my modification.

groovy:
import com.pmease.quickbuild.execution.*;

String clientSpec = """
Client: qb-PDC-BLD-074-9
Owner: guest
Host: PDC-BLD-074
Root: D:/projects/mrc-dlc-package
SubmitOptions: revertunchanged
Options: noallwrite clobber nocompress unlocked nomodtime normdir
LineEnd: local
Description:
       Created by QuickBuild for purpose of running configuration 'root/continuous-package-dlc-solo-proto-bis'.View:
       //driver-branches/dlc01/... //qb-PDC-BLD-074-9/...
"""

def cmd = new Commandline("p4 -p p4-dunia-code-mrc:2326 -u guest -c qb-PDC-BLD-074-9 client -i");
cmd.execute(new LineConsumer.InfoLogger(), new LineConsumer.WarnLogger(), clientSpec).checkReturnCode();

3. It doesn't seem to hang anymore. The first checkout step succeeded, I guess the other checkout steps will follow its example.

Funny thing is that I launched and finished a first build with only the script above. And I'm currently launching a build without the script but with multiple checkout step, and it doesn't hang anymore. What sorcery is it? <img src="{SMILIES_PATH}/icon_smile.gif" alt=":)" title="Smile" />

#16

robinshen ADMIN · 1 decade ago

That is interesting. So the "guest" user you are specifying when define Perforce repository in QB does not have passwords? If so, please try to specify another user with non-empty password if the issue happens again.

#17

Navezof · 1 decade ago

Yes, the guest user doesn't have password. Is using a user without a password troubling QB? The other "funny" thing is that the problem just seemed to happen one day, without us changing anything (Or not that I know of)

Once again, thanks for your help, for the moment it's still working well, but if the problem happens again I will do as you said (which I hope It won't ^^')

#18

robinshen ADMIN · 1 decade ago

At my side, QB works fine even if the perforce user does not have passwords. Let's see how things going on...

#19

Navezof · 1 decade ago

And things are not going well.

It worked for a few days (from last friday (01/08) to this monday (04/08) ) and then one of my build stopped during a sync and when I tried to re-launch it, it stay at the step : CHECKING_BUILD_CONDITION

There wasn't any human intervention during this delay.

Do you have any clues? Like the authentification ticket expired?

In the meantime, I'll try to use a user with a non empty password.

Thanks,

#20

robinshen ADMIN · 1 decade ago

QB will create perforce ticket every time interacting with Perforce, so ticket expiration should not be a probelm. While it is worth trying a non-empty password user, have you tried to run the groovy script to see if it works this time?

#21

Navezof · 1 decade ago

Hello,

I tried to run the groovy script you gave me last time but, the step with the script get stuck. It's launching, but during the execution of the script it seems to stop. (I let it run for 2 hours or so)

I also tried to use another user. In QB, in a configuration Settings > Repositories I put my username and password and launched again the build. But there was no changes and the build is still stuck in CHECKING_BUILD_CONDITION.

I went on the server on which QB is installed and ran the same command : p4 -p myPerforce:2451 -u myUserName -P ****** -c myClientName -i

It react the same way as with no password. The command seems stuck and when I do ctrl+z, it prompt me that there is an "error in client specification" and "Missing required field 'Client' "

#22

robinshen ADMIN · 1 decade ago

When you try running the command from command line, please input below lines:

Client: qb-PDC-BLD-074-9
Owner: guest
Host: PDC-BLD-074
Root: D:\projects\mrc-dlc-package
SubmitOptions: revertunchanged
Options: noallwrite clobber nocompress unlocked nomodtime normdir
LineEnd: local
Description:
Created by QuickBuild for purpose of running configuration 'root/continuous-package-dlc-solo-proto-bis'.
View:
//driver-branches/dlc01/... //qb-PDC-BLD-074-9/...

Then process Ctrl-Z followed by a return to see if it works. Note that the leading spaces of description line and view line is actually a tab.

#23

Navezof · 1 decade ago

After I input those lines, I hit ctrl+z, return. And it seems that the command is stuck. Below is the command lines I used.

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\compilation.driver>p4 -p p4-dunia-code-mrc:2326 -u guest -c qb-PDC-BLD-
074-9 client -i
Client: qb-PDC-BLD-074-9
Owner: guest
Host: PDC-BLD-074
Root: D:\projects\mrc-dlc-package
SubmitOptions: revertunchanged
Options: noallwrite clobber nocompress unlocked nomodtime normdir
LineEnd: local
Description:    Created by QuickBuild for purpose of running configuration root/
continuous-package-dlc-solo-proto-bis.
View:   //driver-branches/dlc01/... //qb-PDC-BLD-074-9/...
^Z

#24

robinshen ADMIN · 1 decade ago

Please separate below lines:

Description:    Created by QuickBuild for purpose of running configuration root/
continuous-package-dlc-solo-proto-bis.
View:   //driver-branches/dlc01/... //qb-PDC-BLD-074-9/...

into:

Description:
    created by QuickBuild for purpose of running configuration root/continuous-package-dlc-solo-proto-bis. 
View:   
    //driver-branches/dlc01/... //qb-PDC-BLD-074-9/...

Note that leading space here is tab.

Then try again to see if perforce hangs, if it does, it seems like a perforce issue or perforce configuration issue.

#25

Navezof · 1 decade ago

Still hanging. I tried different writing (included the one you suggested) but no chance.

You said it could be a perforce issue, do you have an idea of which kind of error could rise this issue?

#26

robinshen ADMIN · 1 decade ago

I am not sure what kind of Perforce error can cause this. Maybe your perforce administrator or perforce support can help this. Just demonstrate this behavior to them as perforce document says that "p4 client -i" should be able to receive perforce client specifications from console without hanging.

#27

Navezof · 1 decade ago

Hello,

I have some news! The script was hanging because the client were not corresponding in the script and in the repository on QB. I changed the client in the script with the same client I put in the repository setting in QB and the script is not hanging anymore.

When I'm launching the build with the checkout steps the build is not stuck on CHECK_BUILD_CONDITION but instead I have the following error message :

Error executing check condition job.
caused by: The validated object is null

And the build is aborted.

Here is the log :

2014-08-11 11:51:09,981 [pool-1-thread-13124] DEBUG com.pmease.quickbuild.execution.Commandline - Executing command: p4 -p p4-dunia-code-mrc:2326 -u guest -c QB_PDC-BLD-074_mrc-dlc-package-code-bin client -o
2014-08-11 11:51:09,982 [pool-1-thread-13124] DEBUG com.pmease.quickbuild.execution.Commandline - Command working directory: C:\quickbuild-5.1.31\bin
2014-08-11 11:51:10,180 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # A Perforce Client Specification.
2014-08-11 11:51:10,180 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - #
2014-08-11 11:51:10,180 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # Client: The client name.
2014-08-11 11:51:10,181 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # Update: The date this specification was last modified.
2014-08-11 11:51:10,181 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # Access: The date this client was last used in any way.
2014-08-11 11:51:10,181 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # Owner: The Perforce user name of the user who owns the client
2014-08-11 11:51:10,181 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # workspace. The default is the user who created the
2014-08-11 11:51:10,181 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # client workspace.
2014-08-11 11:51:10,181 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # Host: If set, restricts access to the named host.
2014-08-11 11:51:10,181 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # Description: A short description of the client (optional).
2014-08-11 11:51:10,181 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # Root: The base directory of the client workspace.
2014-08-11 11:51:10,181 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # AltRoots: Up to two alternate client workspace roots.
2014-08-11 11:51:10,181 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # Options: Client options:
2014-08-11 11:51:10,181 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # [no]allwrite [no]clobber [no]compress
2014-08-11 11:51:10,181 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # [un]locked [no]modtime [no]rmdir
2014-08-11 11:51:10,181 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # SubmitOptions:
2014-08-11 11:51:10,181 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # submitunchanged/submitunchanged+reopen
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # revertunchanged/revertunchanged+reopen
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # leaveunchanged/leaveunchanged+reopen
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # LineEnd: Text file line endings on client: local/unix/mac/win/share.
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # ServerID: If set, restricts access to the named server.
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # View: Lines to map depot files into the client workspace.
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # Stream: The stream to which this client's view will be dedicated.
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # (Files in stream paths can be submitted only by dedicated
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # stream clients.) When this optional field is set, the
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # View field will be automatically replaced by a stream
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # view as the client spec is saved.
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # StreamAtChange: A changelist number that sets a back-in-time view of a
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # stream ( Stream field is required ).
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # Changes cannot be submitted when this field is set.
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - #
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - # Use 'p4 help client' to see more about client views and options.
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider -
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - Client:    QB_PDC-BLD-074_mrc-dlc-package-code-bin
2014-08-11 11:51:10,182 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider -
2014-08-11 11:51:10,183 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - Update:    2014/08/11 04:52:04
2014-08-11 11:51:10,183 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider -
2014-08-11 11:51:10,183 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - Access:    2014/07/07 04:18:39
2014-08-11 11:51:10,183 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider -
2014-08-11 11:51:10,183 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - Owner:    guest
2014-08-11 11:51:10,183 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider -
2014-08-11 11:51:10,183 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - Host:    PDC-BLD-074
2014-08-11 11:51:10,183 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider -
2014-08-11 11:51:10,183 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - Description:
2014-08-11 11:51:10,183 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider -      Created by QuickBuild for purpose of running configuration 'root/continuous-package-dlc-solo-proto-bis'.View:
2014-08-11 11:51:10,183 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider -      //driver-branches/dlc01/... //QB_PDC-BLD-074_mrc-dlc-package-code-bin/...
2014-08-11 11:51:10,183 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider -
2014-08-11 11:51:10,183 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - Root:    D:/projects/mrc-dlc-package
2014-08-11 11:51:10,183 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider -
2014-08-11 11:51:10,183 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - Options:    noallwrite clobber nocompress unlocked nomodtime normdir
2014-08-11 11:51:10,183 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider -
2014-08-11 11:51:10,183 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - SubmitOptions:    revertunchanged
2014-08-11 11:51:10,184 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider -
2014-08-11 11:51:10,184 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider - LineEnd:    local
2014-08-11 11:51:10,184 [pool-1-thread-13125] DEBUG com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider -
2014-08-11 11:51:10,282 [pool-1-thread-13124] DEBUG com.pmease.quickbuild.grid.GridJob - Notifying task node of job finishing (job id: 811b90ba-18ef-4898-9be9-11f5ba404f1f, task node: PDC-BLD-074:8810:dddb1331-914d-4981-9352-adbc80ff3690)...
2014-08-11 11:51:10,286 [pool-1-thread-13124] DEBUG com.pmease.quickbuild.grid.NodeServiceImpl - Handling job finish event (job id: 811b90ba-18ef-4898-9be9-11f5ba404f1f)...
2014-08-11 11:51:10,286 [pool-1-thread-13124] DEBUG com.pmease.quickbuild.grid.GridTaskFuture - Adding job to finished list (task future: com.pmease.quickbuild.grid.GridTaskFuture@4cb153a6, job id: 811b90ba-18ef-4898-9be9-11f5ba404f1f)
2014-08-11 11:51:10,286 [pool-1-thread-13124] DEBUG com.pmease.quickbuild.grid.GridJob - Finished execution of job 811b90ba-18ef-4898-9be9-11f5ba404f1f...
2014-08-11 11:51:10,312 [pool-1-thread-13045] ERROR com.pmease.quickbuild.DefaultBuildEngine - Error processing build request.
    java.lang.RuntimeException: Error executing check condition job.
        at com.pmease.quickbuild.CheckConditionTask.reduce(CheckConditionTask.java:39)
        at com.pmease.quickbuild.CheckConditionTask.reduce(CheckConditionTask.java:16)
        at com.pmease.quickbuild.grid.GridTaskFuture.get(GridTaskFuture.java:95)
        at com.pmease.quickbuild.grid.GridTaskFuture.get(GridTaskFuture.java:100)
        at com.pmease.quickbuild.DefaultBuildEngine.process(DefaultBuildEngine.java:317)
        at com.pmease.quickbuild.DefaultBuildEngine.access$000(DefaultBuildEngine.java:130)
        at com.pmease.quickbuild.DefaultBuildEngine$2.run(DefaultBuildEngine.java:1081)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
    Caused by: java.lang.IllegalArgumentException: The validated object is null
        at org.apache.commons.lang.Validate.notNull(Validate.java:222)
        at com.pmease.quickbuild.plugin.scm.perforce.Mapping.parseView(Mapping.java:63)
        at com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider.getStaticSpec(TemplateStaticSpecProvider.java:143)
        at com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider$$EnhancerByCGLIB$$a5a012e2.CGLIB$getStaticSpec$3(<generated>)
        at com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider$$EnhancerByCGLIB$$a5a012e2$$FastClassByCGLIB$$24cd8b98.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:215)
        at com.pmease.quickbuild.DefaultScriptEngine$Interpolator.intercept(DefaultScriptEngine.java:273)
        at com.pmease.quickbuild.plugin.scm.perforce.TemplateStaticSpecProvider$$EnhancerByCGLIB$$a5a012e2.getStaticSpec(<generated>)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository.getStaticSpec(PerforceRepository.java:295)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository.setupClient(PerforceRepository.java:733)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$d441c6ae.CGLIB$setupClient$42(<generated>)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$d441c6ae$$FastClassByCGLIB$$679462e7.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:215)
        at com.pmease.quickbuild.DefaultScriptEngine$Interpolator.intercept(DefaultScriptEngine.java:273)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$d441c6ae.setupClient(<generated>)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository.getHeadRevision(PerforceRepository.java:788)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$d441c6ae.CGLIB$getHeadRevision$12(<generated>)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$d441c6ae$$FastClassByCGLIB$$679462e7.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:215)
        at com.pmease.quickbuild.DefaultScriptEngine$Interpolator.intercept(DefaultScriptEngine.java:273)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$d441c6ae.getHeadRevision(<generated>)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository.getHeadRevision(PerforceRepository.java:59)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$d441c6ae.CGLIB$getHeadRevision$13(<generated>)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$d441c6ae$$FastClassByCGLIB$$679462e7.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:215)
        at com.pmease.quickbuild.DefaultScriptEngine$Interpolator.intercept(DefaultScriptEngine.java:273)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$d441c6ae.getHeadRevision(<generated>)
        at com.pmease.quickbuild.repositorysupport.Repository.takeSnapshot(Repository.java:332)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$d441c6ae.CGLIB$takeSnapshot$58(<generated>)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$d441c6ae$$FastClassByCGLIB$$679462e7.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:215)
        at com.pmease.quickbuild.DefaultScriptEngine$Interpolator.intercept(DefaultScriptEngine.java:273)
        at com.pmease.quickbuild.plugin.scm.perforce.PerforceRepository$$EnhancerByCGLIB$$d441c6ae.takeSnapshot(<generated>)
        at com.pmease.quickbuild.setting.configuration.snapshot.TakeReferencedSnapshots.takeSnapshot(TakeReferencedSnapshots.java:15)
        at com.pmease.quickbuild.model.Configuration.takeSnapshot(Configuration.java:1577)
        at com.pmease.quickbuild.CheckConditionJob.execute(CheckConditionJob.java:35)
        at com.pmease.quickbuild.grid.GridJob.run(GridJob.java:73)
        ... 5 more
2014-08-11 11:51:10,325 [Thread-73] DEBUG com.pmease.quickbuild.DefaultBuildEngine - Starting new round of resource allocation request processing...
2014-08-11 11:51:10,325 [Thread-73] DEBUG com.pmease.quickbuild.DefaultBuildEngine - Calculating current resource usages...
2014-08-11 11:51:10,325 [Thread-73] DEBUG com.pmease.quickbuild.DefaultBuildEngine - Allocating resources for waiting steps...
2014-08-11 11:51:13,325 [Thread-73] DEBUG com.pmease.quickbuild.DefaultBuildEngine - Starting new round of resource allocation request processing...
2014-08-11 11:51:13,325 [Thread-73] DEBUG com.pmease.quickbuild.DefaultBuildEngine - Calculating current resource usages...
2014-08-11 11:51:13,325 [Thread-73] DEBUG com.pmease.quickbuild.DefaultBuildEngine - Allocating resources for waiting steps...
2014-08-11 11:51:16,325 [Thread-73] DEBUG com.pmease.quickbuild.DefaultBuildEngine - Starting new round of resource allocation request processing...
2014-08-11 11:51:16,325 [Thread-73] DEBUG com.pmease.quickbuild.DefaultBuildEngine - Calculating current resource usages...
2014-08-11 11:51:16,325 [Thread-73] DEBUG com.pmease.quickbuild.DefaultBuildEngine - Allocating resources for waiting steps...

Thanks again for your help.

#28

robinshen ADMIN · 1 decade ago

Can you please run below command on command line and then send me [robin AT pmease DOT com] spec.txt?
p4 -p p4-dunia-code-mrc:2326 -u guest -c QB_PDC-BLD-074_mrc-dlc-package-code-bin client -o > spec.txt

#29

tardis4500 · 1 decade ago

This has happened to me two days in a row. The server works fine normally, but some time overnight, even though no builds are being done, the server hangs up with all the resource used up because they are all in CHECKING_BUILD_CONDITION. I am unable to cancel these or shut down the server. I have to manual kill the Java process in order to recover.

#30

robinshen ADMIN · 1 decade ago

The build stuck issues raised by Navezof turns out to be that p4 command is hanging while accepting a client spec, and the reason is that the perforce repository is using a template client and spec of the template client is incorrect (without necessary line breaks). Once the template client has been recreated to fix the spec issue, the build works without any issues. While build is stuck at your side, can you please check the agent log running the master step to see if it is also stuck at running the command "p4 client -i"? The command will only be printed if you edit "conf/log4j.properties" to turn on the debug logging.

#31

tardis4500 · 1 decade ago

We are not using templates. This does not happen every night but it happened last night. I have 4 builds stuck at "CHECKING_BUILD_CONDITION" and 4 stuck in "WAITING_PROCESS" since 2AM. Our log rolls over every 20 minutes so I don't have the logs from that time frame. I am unable to stop the server.

#32

tardis4500 · 1 decade ago

After further investigation, these four builds are not Perforce repositories but a plugin that we wrote. While it is possible that a network hiccup has caused this, I would still like to know how I can clear them or at least restart the server.

#33

robinshen ADMIN · 1 decade ago

QB server can only be restarted after build is finished. So in case of build stuck, it can only be killed. While build is stuck in this status, please check the node running CHECKING_BUILD_CONDITION (the master node), and then switch to that node and run below command to get the stack trace:
jstack <QB JVM process ID>

#34

tardis4500 · 1 decade ago

The system still hangs once every couple of weeks. I was finally able to run the jstack but I don't find it very informative.

Deadlock Detection:

No deadlocks found.

Thread 89: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- sun.net.www.http.KeepAliveCache.run() @bci=3, line=172 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 88: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Compiled frame)
- org.eclipse.jetty.util.BlockingArrayQueue.poll(long, java.util.concurrent.TimeUnit) @bci=53, line=342 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll() @bci=12, line=526 (Interpreted frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(org.eclipse.jetty.util.thread.QueuedThreadPool) @bci=1, line=44 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=275, line=572 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 87: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Compiled frame)
- org.eclipse.jetty.util.BlockingArrayQueue.poll(long, java.util.concurrent.TimeUnit) @bci=53, line=342 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll() @bci=12, line=526 (Interpreted frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(org.eclipse.jetty.util.thread.QueuedThreadPool) @bci=1, line=44 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=275, line=572 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 85: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- com.pmease.quickbuild.grid.AgentConnectivityTask.run() @bci=271, line=66 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 84: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- com.pmease.quickbuild.grid.GridTaskFuture.get(long) @bci=72, line=78 (Compiled frame)
- com.pmease.quickbuild.grid.GridTaskFuture.get() @bci=2, line=101 (Interpreted frame)
- com.pmease.quickbuild.grid.GridNode.executeJob(java.lang.Class, java.io.Serializable[]) @bci=54, line=459 (Interpreted frame)
- com.pmease.quickbuild.plugin.measurement.core.reporter.MeasurementServerReporter.run(com.pmease.quickbuild.plugin.measurement.core.extensionpoint.NodeMeasurementReport) @bci=77, line=79 (Interpreted frame)
- com.pmease.quickbuild.plugin.measurement.core.reporter.AbstractReporter.onReportReceived(com.pmease.quickbuild.plugin.measurement.core.extensionpoint.NodeMeasurementReport) @bci=9, line=12 (Interpreted frame)
- sun.reflect.GeneratedMethodAccessor186.invoke(java.lang.Object, java.lang.Object[]) @bci=40 (Interpreted frame)
- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)
- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=57, line=606 (Compiled frame)
- com.google.common.eventbus.EventHandler.handleEvent(java.lang.Object) @bci=21, line=74 (Interpreted frame)
- com.google.common.eventbus.SynchronizedEventHandler.handleEvent(java.lang.Object) @bci=2, line=45 (Interpreted frame)
- com.google.common.eventbus.EventBus.dispatch(java.lang.Object, com.google.common.eventbus.EventHandler) @bci=2, line=313 (Interpreted frame)
- com.google.common.eventbus.EventBus.dispatchQueuedEvents() @bci=62, line=296 (Interpreted frame)
- com.google.common.eventbus.EventBus.post(java.lang.Object) @bci=185, line=267 (Interpreted frame)
- com.pmease.quickbuild.plugin.measurement.core.poller.MeasurementPollingTask.execute() @bci=85, line=56 (Interpreted frame)
- com.pmease.quickbuild.plugin.measurement.core.poller.MeasurementPollingTask.run() @bci=36, line=108 (Interpreted frame)
- com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run() @bci=34, line=52 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 83: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.util.TimerThread.mainLoop() @bci=201, line=552 (Interpreted frame)
- java.util.TimerThread.run() @bci=1, line=505 (Interpreted frame)

Thread 82: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.core.QuartzSchedulerThread.run() @bci=807, line=411 (Interpreted frame)

Thread 81: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 80: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 79: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 78: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 77: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 76: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 75: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 74: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 73: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 72: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 71: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 70: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 69: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 68: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 67: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 66: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 65: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 64: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 63: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 62: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 61: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 60: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 59: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 58: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 57: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 56: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 55: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 54: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 53: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 52: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 51: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 50: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 49: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 48: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 47: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 46: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 45: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 44: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 43: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 42: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 41: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 40: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 39: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 38: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 37: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 36: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 35: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 34: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 33: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 32: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.quartz.simpl.SimpleThreadPool$WorkerThread.run() @bci=51, line=543 (Compiled frame)

Thread 31: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Compiled frame)
- org.eclipse.jetty.util.BlockingArrayQueue.poll(long, java.util.concurrent.TimeUnit) @bci=53, line=342 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll() @bci=12, line=526 (Interpreted frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(org.eclipse.jetty.util.thread.QueuedThreadPool) @bci=1, line=44 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=275, line=572 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 30: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Compiled frame)
- org.eclipse.jetty.util.BlockingArrayQueue.poll(long, java.util.concurrent.TimeUnit) @bci=53, line=342 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll() @bci=12, line=526 (Interpreted frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(org.eclipse.jetty.util.thread.QueuedThreadPool) @bci=1, line=44 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=275, line=572 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 29: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Compiled frame)
- org.eclipse.jetty.util.BlockingArrayQueue.poll(long, java.util.concurrent.TimeUnit) @bci=53, line=342 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll() @bci=12, line=526 (Interpreted frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(org.eclipse.jetty.util.thread.QueuedThreadPool) @bci=1, line=44 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=275, line=572 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 28: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Compiled frame)
- org.eclipse.jetty.util.BlockingArrayQueue.poll(long, java.util.concurrent.TimeUnit) @bci=53, line=342 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll() @bci=12, line=526 (Interpreted frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(org.eclipse.jetty.util.thread.QueuedThreadPool) @bci=1, line=44 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=275, line=572 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 27: (state = IN_NATIVE)
- java.net.DualStackPlainSocketImpl.accept0(int, java.net.InetSocketAddress[]) @bci=0 (Interpreted frame)
- java.net.DualStackPlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=37, line=131 (Interpreted frame)
- java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=398 (Interpreted frame)
- java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=35, line=198 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=530 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=498 (Interpreted frame)
- org.eclipse.jetty.server.bio.SocketConnector.accept(int) @bci=4, line=117 (Interpreted frame)
- org.eclipse.jetty.server.AbstractConnector$Acceptor.run() @bci=153, line=938 (Interpreted frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable) @bci=1, line=608 (Interpreted frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47, line=543 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 26: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Compiled frame)
- org.eclipse.jetty.util.BlockingArrayQueue.poll(long, java.util.concurrent.TimeUnit) @bci=53, line=342 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll() @bci=12, line=526 (Interpreted frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(org.eclipse.jetty.util.thread.QueuedThreadPool) @bci=1, line=44 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=275, line=572 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 25: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.TimerThread.mainLoop() @bci=201, line=552 (Compiled frame)
- java.util.TimerThread.run() @bci=1, line=505 (Interpreted frame)

Thread 24: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=151 (Interpreted frame)
- com.google.inject.internal.util.$Finalizer.run() @bci=5, line=114 (Interpreted frame)

Thread 23: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.log4j.helpers.FileWatchdog.run() @bci=11, line=104 (Compiled frame)

Thread 2: (state = BLOCKED)

Thread 21: (state = IN_NATIVE)
- java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise)
- java.net.SocketInputStream.read(byte[], int, int, int) @bci=87, line=152 (Compiled frame)
- java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=122 (Compiled frame)
- java.net.SocketInputStream.read() @bci=23, line=210 (Interpreted frame)
- java.io.DataInputStream.readByte() @bci=4, line=265 (Interpreted frame)
- org.tanukisoftware.wrapper.WrapperManager.handleBackend() @bci=47, line=5254 (Interpreted frame)
- org.tanukisoftware.wrapper.WrapperManager.run() @bci=145, line=5631 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 20: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- org.tanukisoftware.wrapper.WrapperManager$3.run() @bci=327, line=1034 (Compiled frame)

Thread 15: (state = BLOCKED)

Thread 14: (state = BLOCKED)

Thread 13: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 (Compiled frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=151 (Compiled frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=16, line=189 (Compiled frame)

Thread 12: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.Object.wait() @bci=2, line=503 (Compiled frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 (Compiled frame)

#35

robinshen ADMIN · 1 decade ago

Any possibility that we check this issue online while this is happening? I am at timezone of GMT+8 and is normally available from 8:30AM to 10:00PM.

#36

tardis4500 · 1 decade ago

No. Our network is blocked and the company doesn't allow that.

#37

robinshen ADMIN · 1 decade ago

That is a pity, but it might still be helpful if we can check the issue online when it is happening even if I can not access your machine. By then you may send me screenshots so that I can check and provide further instructions according to the situation.

#38

tardis4500 · 1 decade ago

I am upgrading from 5.1.16 to 5.1.36 now. If the problem occurs again I will let you know. I am 12 hours behind you so we could work on it form 8p-10p your time. Should I send you email directly?

#39

robinshen ADMIN · 1 decade ago

Yes, please send me email [robin AT pmease DOT com] directly for the followups.

#40

tardis4500 · 1 decade ago

I upgraded to 5.1.36 last week but came in this morning to find the server stuck. I sent you email but there was no response and I need to get the server back up and running. I will restart it to get it running again. Please let me know what we need to do next to diagnose this problem. In addition, it would be nice to have a better way to restart the server than hunting down the Java process and killing it.

#41

robinshen ADMIN · 1 decade ago

Will cancellation of these pending build requests work if they are stucked at checking build condition?

#42

tardis4500 · 1 decade ago

No. Cancelling them does not work.