Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

QuickBuild

Failure on test launching with Azure profile #4260

#1
absalom1 ·

While we are testing connection to Azure cloud, we are facing test launching failure with below message.
So we tried some command to verify connection between our server and Azure Cloud refered by GitHub page and it works well.
Moreover, testing connection to Azure cloud with Jenkins also works well.

Could you help us to resolve this issue?

2020-06-30 15:34:05,958 [qtp1144791320-68] INFO com.pmease.quickbuild.entitymanager.impl.DefaultCloudProfileManager - Running test launch for profile 'AZURE_POC'...
2020-06-30 15:36:15,307 [pool-18-thread-1] ERROR com.microsoft.aad.adal4j.AuthenticationContext - [Correlation ID: 3e0b68be-dbf3-42c5-96e7-93a29d44997b] Request to acquire token failed.
 java.net.ConnectException: Connection timed out (Connection timed out)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:607)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:681)
        at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264)
        at com.microsoft.aad.adal4j.AdalOAuthRequest.configureHeaderAndExecuteOAuthCall(AdalOAuthRequest.java:140)
        at com.microsoft.aad.adal4j.AdalOAuthRequest.send(AdalOAuthRequest.java:83)
        at com.microsoft.aad.adal4j.AdalTokenRequest.executeOAuthRequestAndProcessResponse(AdalTokenRequest.java:80)
        at com.microsoft.aad.adal4j.AuthenticationContext.acquireTokenCommon(AuthenticationContext.java:818)
        at com.microsoft.aad.adal4j.AuthenticationContext.access$100(AuthenticationContext.java:66)
        at com.microsoft.aad.adal4j.AuthenticationContext$1.call(AuthenticationContext.java:174)
        at com.microsoft.aad.adal4j.AuthenticationContext$1.call(AuthenticationContext.java:163)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-06-30 15:36:15,311 [pool-1-thread-309] ERROR com.pmease.quickbuild.plugin.cloud.azure.LinuxNodeLauncher - Error launching azure node
 java.lang.RuntimeException: java.io.IOException: java.net.ConnectException: Connection timed out (Connection timed out)
        at rx.exceptions.Exceptions.propagate(Exceptions.java:58)
        at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:464)
        at rx.observables.BlockingObservable.last(BlockingObservable.java:227)
        at com.microsoft.azure.management.resources.fluentcore.arm.collection.implementation.GroupableResourcesImpl.getById(GroupableResourcesImpl.java:69)
        at com.microsoft.azure.management.resources.fluentcore.arm.collection.implementation.GroupableResourcesImpl.getById(GroupableResourcesImpl.java:34)
        at com.pmease.quickbuild.plugin.cloud.azure.LinuxNodeLauncher$1.run(LinuxNodeLauncher.java:81)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 Caused by: java.io.IOException: java.net.ConnectException: Connection timed out (Connection timed out)
        at com.microsoft.azure.credentials.ApplicationTokenCredentials.acquireAccessToken(ApplicationTokenCredentials.java:171)
        at com.microsoft.azure.credentials.ApplicationTokenCredentials.getToken(ApplicationTokenCredentials.java:139)
        at com.microsoft.azure.credentials.AzureTokenCredentials.getToken(AzureTokenCredentials.java:59)
        at com.microsoft.azure.credentials.AzureTokenCredentialsInterceptor.intercept(AzureTokenCredentialsInterceptor.java:36)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
        at com.microsoft.azure.management.resources.fluentcore.utils.ResourceManagerThrottlingInterceptor.intercept(ResourceManagerThrottlingInterceptor.java:43)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
        at com.microsoft.azure.management.resources.fluentcore.utils.ProviderRegistrationInterceptor.intercept(ProviderRegistrationInterceptor.java:43)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
        at com.microsoft.rest.interceptors.BaseUrlHandler.intercept(BaseUrlHandler.java:43)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
        at com.microsoft.rest.interceptors.RequestIdHeaderInterceptor.intercept(RequestIdHeaderInterceptor.java:29)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170)
        at okhttp3.RealCall.execute(RealCall.java:60)
        at retrofit2.OkHttpCall.execute(OkHttpCall.java:174)
        at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$RequestArbiter.request(RxJavaCallAdapterFactory.java:171)
        at rx.Subscriber.setProducer(Subscriber.java:211)
        at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
        at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:152)
        at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:138)
        at rx.Observable.unsafeSubscribe(Observable.java:10142)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
        at rx.Observable.unsafeSubscribe(Observable.java:10142)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
        at rx.Observable.unsafeSubscribe(Observable.java:10142)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
        at rx.Observable.unsafeSubscribe(Observable.java:10142)
        at rx.internal.operators.DeferredScalarSubscriber.subscribeTo(DeferredScalarSubscriber.java:153)
        at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:32)
        at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:22)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
        at rx.Observable.subscribe(Observable.java:10238)
        at rx.Observable.subscribe(Observable.java:10205)
        at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:444)
        ... 7 more
 Caused by: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection timed out (Connection timed out)
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at com.microsoft.azure.credentials.ApplicationTokenCredentials.acquireAccessToken(ApplicationTokenCredentials.java:157)
        ... 52 more
 Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:607)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:681)
        at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264)
        at com.microsoft.aad.adal4j.AdalOAuthRequest.configureHeaderAndExecuteOAuthCall(AdalOAuthRequest.java:140)
        at com.microsoft.aad.adal4j.AdalOAuthRequest.send(AdalOAuthRequest.java:83)
        at com.microsoft.aad.adal4j.AdalTokenRequest.executeOAuthRequestAndProcessResponse(AdalTokenRequest.java:80)
        at com.microsoft.aad.adal4j.AuthenticationContext.acquireTokenCommon(AuthenticationContext.java:818)
        at com.microsoft.aad.adal4j.AuthenticationContext.access$100(AuthenticationContext.java:66)
        at com.microsoft.aad.adal4j.AuthenticationContext$1.call(AuthenticationContext.java:174)
        at com.microsoft.aad.adal4j.AuthenticationContext$1.call(AuthenticationContext.java:163)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        ... 3 more
1 Reply
  • solved #11
  • replies 20
  • views 3502
  • stars 0
#2
absalom1 ·

FYI, we have proxy to connect external website but I also wonder that Azure plugin uses system default proxy setting like "http_proxy" or "https_proxy"

#3
robinshen ADMIN ·

Just tested to launch a linux Azure node in QB and everything works fine. To narrow down the issue, please connect QB to Azure directly without proxy to see if it works at your side.

1 :thumbsup:
#4
absalom1 ·

Could you add proxy setting field on Azure plugin?

#5
robinshen ADMIN ·

Just want to confirm first: are you able to launch agent successfully without proxy?

#6
absalom1 ·

Unfortunately, I tried after making firewall exception to below urls not to use proxy but failed to launch.
I would try again if I know more URLs to be opened to launch without proxy.

"activeDirectoryEndpointUrl": "https://login.microsoftonline.com",
"resourceManagerEndpointUrl": "https://management.azure.com/",
"activeDirectoryGraphResourceId": "https://graph.windows.net/",
"sqlManagementEndpointUrl": "https://management.core.windows.net:8443/",
"galleryEndpointUrl": "https://gallery.azure.com/",
"managementEndpointUrl": "https://management.core.windows.net/"

#7
robinshen ADMIN ·

I am not sure exactly what urls should be opened either. How about checking your firewall log to see what urls QB is trying to connect?

#8
absalom1 ·

The problem was on our firewall policy. URL based firewall exception request checks IP of that URL every 4 hours and apply the exception to that IP.
However the IP for Azure management URL changes frequently and our firewall system cannot be updated as required.
So if possible, using proxy on Azure plugin would be great for us. :grin:
Please let us know your opinion.

#9
robinshen ADMIN ·

How do you specify https_proxy? Is it done by specifying JVM system property in "conf/wrapper.conf", or specifying at OS level?

#10
absalom1 ·

Specified at OS level and I could find it at "System Attributes" tab

1 #11
robinshen ADMIN ·
1 :thumbsup:
1 #12
absalom1 ·

It works GREAT!!!
Thank you so much!! :thumbsup:

#13
robinshen ADMIN ·

Glad to see it is working. Let me know if I can be of further help.

#14
absalom1 ·

One more question.
When I request a VM to Azure, Azure response within short time but QB gets the VM after 7 minutes.
Is it default setting in QB or do I need to check network connection?

#15
robinshen ADMIN ·

When VM starts at Azure side, please login to VM and run below command to see what is happening:

tail -f /logs/console.log

#16
absalom1 ·

Thanks for your support.
Below is some part of console.log while connecting to QuickBuild master.

Could you help us to reduce the time between starting of agent and starting of build?

jvm 1    | 2020-07-13 08:42:33,867 INFO  - Cleaning cached plugin libraries...
....
jvm 1    | 2020-07-13 08:42:45,521 INFO  - Checking node launch data...
jvm 1    | 2020-07-13 08:44:16,799 INFO  - Starting plugins...
jvm 1    | 2020-07-13 08:44:17,313 INFO  - Constructing report category for plugin: SCM Changes (8.0.0)
....
jvm 1    | 2020-07-13 08:44:18,726 INFO  - Constructing report category for plugin: GitLab Integration (8.0.20)
jvm 1    | 2020-07-13 08:48:24,057 INFO  - QuickBuild agent started.
#17
robinshen ADMIN ·

Looks like that you are still using QB8 which is pretty old. Can you please test with latest QB10 to see if it still requires this long star time? To use QB10, you will need to rebuild the agent image.

#18
absalom1 ·

Testing with native QB10 master and QB10 agent works great.
However, connecting QB10 agent to my QB8 master gave same result.

Comparing to Azure environment, agent on bare-metal connects to QB8 master within short time.

jvm 1    | 2020-07-15 18:46:12,328 INFO  - Constructing report category for plugin: GitLab Integration (8.0.20)
jvm 1    | 2020-07-15 18:46:12,444 INFO  - QuickBuild agent started.

I'm curious what kind of processes are running between those two log lines.

#19
robinshen ADMIN ·

When connect QB10 agent to QB8 master, QB10 agent will be downgraded to QB8 agent. Please upgrade your master to QB10, as we've fixed a lot of bugs since QB8.

1 :handshake:
#20
absalom1 ·

Found the main reason of delaying, custom plugin is the problem, and QB8 connects within 1 sec also after fixing it.

#21
robinshen ADMIN ·

Glad to see it solved. Thanks for letting me know.

Log in to reply