Introduction
Panu Sakku posted the following tweet recently asking if anyone noticed their CMG (Cloud Management Gateway) was broken after it got a recent update.
I checked my lab, and sure enough, it was also dead in the water, and could not start.
After checking the logs I replied to Panu.
The errors in the SMS_CLOUD_PROXYCONNECTOR.log file in red were many, and here’s a paste of some of them to help others find out how to resolve this problem.
ERROR: Web socket: Failed to online with Proxy server CLOUDATTACHCMG.AZURENOOB.COM:443. System.AggregateException: One or more errors occurred. —> System.Net.WebSockets.WebSocketException: Unable to connect to the remote server —> System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 20.126.223.196:443~~ at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)~~ at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)~~ at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)~~ — End of inner exception stack trace —~~ at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)~~ at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)~~— End of stack trace from previous location where exception was thrown —~~ at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()~~ at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)~~ at System.Net.WebSockets.ClientWebSocket.<ConnectAsyncCore>d__21.MoveNext()~~ — End of inner exception stack trace —~~ at System.Net.WebSockets.ClientWebSocket.<ConnectAsyncCore>d__21.MoveNext()~~ — End of inner exception stack trace —~~ at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)~~ at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)~~ at Microsoft.ConfigurationManager.CloudConnection.ProxyConnector.WebSocketConnection.Online()~~—> (Inner Exception #0) System.Net.WebSockets.WebSocketException (0x80004005): Unable to connect to the remote server —> System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 20.126.223.196:443~~ at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)~~ at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)~~ at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)~~ — End of inner exception stack trace —~~ at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)~~ at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction,
and
ERROR: Failed to build WebSocket connection 1800a2f4-5e7c-4aa7-9c5d-0b4027ab939d with server CLOUDATTACHCMG.AZURENOOB.COM:443. Exception: System.Net.WebException: Failed to online~~ at Microsoft.ConfigurationManager.CloudConnection.ProxyConnector.WebSocketConnection.Online()~~ at Microsoft.ConfigurationManager.CloudConnection.ProxyConnector.ConnectionBase.Start()~~ at Microsoft.ConfigurationManager.CloudConnection.ProxyConnector.ConnectionManager.MaintainConnections()
and
ERROR: Failed to build HttpV2 connection 1800a2f4-5e7c-4aa7-9c5d-0b4027ab939d with server CLOUDATTACHCMG.AZURENOOB.COM:443. Exception: System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 20.126.223.196:443~~ at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)~~ at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)~~ at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)~~ — End of inner exception stack trace —~~ at Microsoft.ConfigurationManager.CloudConnection.ProxyConnector.HttpConnectionV2.SendInternal(HttpMethod method, String path, String payload, Int32& statusCode, Byte[]& responsePayload)~~ at Microsoft.ConfigurationManager.CloudConnection.ProxyConnector.HttpConnectionV2.SendInternal(HttpMethod method, String path, Byte[] payload)~~ at Microsoft.ConfigurationManager.CloudConnection.ProxyConnector.HttpConnectionV2.Online()~~ at Microsoft.ConfigurationManager.CloudConnection.ProxyConnector.ConnectionBase.Start()~~ at Microsoft.ConfigurationManager.CloudConnection.ProxyConnector.ConnectionManager.MaintainConnections()
Shortly after I replied, Johnny Radeck posted an update, he solved it by uninstalling an extension and then making a change to the CMG properties.
But let’s see why he did that. If you go to the Azure portal and locate your CMG, you’ll see it’s got a Failed status (1). If you click Restart (2) after a few minutes it’ll be failed again but you’ll get a notification (3) explaining what failed.
Failed to restart virtual machine scale set
So it’s clear that Azure has problems starting the CMG due to “VM has reported a user failure when processing extension ‘InstallCMG’.”
I wonder what the ‘user failure’ means ?
Let’s try Johnny’s advice then.
Fixing the problem ?
Click on Settings, select Extensions + applications and then place a checkmark in InstallCMG, it’ll bring up it’s properties and you can now select Uninstall.
The settings in that extension are listed here, just to see if they change after the fix.
“commandToExecute”: “powershell.exe -File cmgsetup.ps1 -storageAccountName cloudattachcmg -storageEndpointSuffix core.windows.net -serviceName cloudattachcmg -serviceCName cloudattachcmg.azurenoob.com -certStoreName My -certThumbprint 2D2F89A0F44335C0D57678DA5AC80663660B0250 -crlAction enable -tls12Enforced True -nodeName localhost -bDisabledSharedKey True”,
“fileUris”: [
“https://cloudattachcmg.blob.core.windows.net/stageartifacts/cmgsetup.ps1”
]
}
After a while it’ll be uninstalled and you’ll get a notification telling you that it’s done.
After changing Client revocation settings, and changing the maintenance window to be in the future (otherwise you’ll get an error) before clicking Apply.
A quick look at the CloudMgr.log reveals it’s updating the CMG
and the status of the CMG in SCCM changes to Upgrading.
while in Azure, the CMG has a status of Updating.
and after a while everything should hopefully be fixed.
Note: If it works for you, then don’t forget to set the client revocation option back on again.
Oops
In my case however, no matter how many times I tried my CMG remained well and truly broken.
It’s still broken.
I’ll update this post if/when I come up with a solution that works for me, but for now, this is just where I’m at with this problem and I’m blogging this as I’ve spent so many hours on it already.
