[Azure] News for Developers, August 2019

This entry is part 27 of 28 in the series Azure news for Developers

Are you having trouble keeping track of everything that’s going around in Azure? You’re not alone! In an effort to do so myself, I’m starting a monthly series called “News for developers” which is exactly that: a summary of all of the Azure flavored news specifically for software developers.

This is based on my personal feeds and my personal opinion, so you might miss things or see things which in your opinion do not matter. Feel free to comment below and I’ll see what I can do for the next edition. And honestly, this is more a personal reference than anything else so having actual readers would already be awesome 🙂 Enjoy!

Read More

[Azure] Application Gateway certificate gotchas

At my current assignment, my team is using the Azure Application Gateway to securely make available some services within Azure such as API Management and WebApps. Up to a couple of weeks ago, we were using the “old” (what’s old, right?) version of the gateway to do this. Until a production outage woke us up, let me describe what was happening.

 

End-to-end SSL

The Application Gateway allows you to configure a different listening URL compared to the URL that your back-end is using. In our case, some of our backends are simply using the *.azurewebsites.net certificate, but our front-ends are using customized URLs on the customers domain. This effectively means that the gateway will terminate the “outside” SSL and switch to using the internal back-end certificate for internal communication. This way, the entire connection is still secure and thus we have end-to-end SSL.

The V1 gateways have a restriction in the fact that you either can provide your own certificate to do so, or you can provide a custom one. We were using the latter because our API management endpoints also run a custom cert for internal traffic (which in turn is a ‘restriction’ / requirement of API management instances).

 

Certificate updating

The issue we identified boiled down to the fact that Microsoft had updated their *.azurewebsites.net certificate, but this update didn’t make it to the application gateway instances. So when the back-end hosts started to deploy the new certificate, the gateways started marking the hosts as unhealthy due to an invalid certificate: “BackendServerCertificateNotWhitelisted“. Whoops. It took us a while to find out what has happening as the configuration didn’t change and the certificate itself seemed fine to us. Eventually, forcing an update of the gateway config somehow triggered the certificates to be refreshed which resolved the issue. Microsoft Support confirmed we were not the only ones to have this issue.

 

Gateway V2: the importance of the certificate chain

After fixing the above issue, support indicated that we might want to consider moving to the V2 SKU of the application gateway. This does not have the limitation of having to pick between either a platform managed certificate or custom certificates, instead it can mix both. It should also be more resilient to updates of the platform certificates, which I guess we just have to believe then.

And so we updated to V2, only to run into the next certificate based issue.

Great, so now what? We noticed a part of our Java-based landscape falling over with the new gateways in place. Certificate issues, even though we were using the exact same certificates as before. After again a bit of investigation we found (using ssllabs.com) that the V2 gateway was returning only the primary certificate, where the V1 gateway was returning a full chain. I again got in touch with Microsoft support, who pointed me to https://docs.microsoft.com/en-us/azure/application-gateway/ssl-overview#end-to-end-ssl-with-the-v2-sku.

The problem was with the fact that our PFX we were using did not contain the full chain of the certificate. For the V1 gateway this didn’t matter, but for V2 this does. So again the fix for this issue was a lot simpler than finding the actual issue: we exported the certificate again using the “Include all certificates in the certification path if possible” option. This will create a PFX including the certificate chain.

 

After uploading these new certs to KeyVault and updating the gateway instance, everything started working again!

[Azure] News for Developers, July 2019

This entry is part 26 of 28 in the series Azure news for Developers

Are you having trouble keeping track of everything that’s going around in Azure? You’re not alone! In an effort to do so myself, I’m starting a monthly series called “News for developers” which is exactly that: a summary of all of the Azure flavored news specifically for software developers. Now this is based on my personal feeds and my personal opinion, so you might miss things or see things which in your opinion do not matter. Feel free to comment below and I’ll see what I can do for the next edition. And honestly, this is more a personal reference than anything else so having actual readers would already be awesome 🙂 Enjoy!

Read More

[Azure] News for Developers, June 2019

This entry is part 24 of 28 in the series Azure news for Developers

Are you having trouble keeping track of everything that’s going around in Azure? You’re not alone! In an effort to do so myself, I’m starting a monthly series called “News for developers” which is exactly that: a summary of all of the Azure flavored news specifically for software developers. Now this is based on my personal feeds and my personal opinion, so you might miss things or see things which in your opinion do not matter. Feel free to comment below and I’ll see what I can do for the next edition. And honestly, this is more a personal reference than anything else so having actual readers would already be awesome 🙂 Enjoy!

Read More

[Azure] News for Developers, May 2019

This entry is part 23 of 28 in the series Azure news for Developers

Are you having trouble keeping track of everything that’s going around in Azure? You’re not alone! In an effort to do so myself, I’m starting a monthly series called “News for developers” which is exactly that: a summary of all of the Azure flavored news specifically for software developers. Now this is based on my personal feeds and my personal opinion, so you might miss things or see things which in your opinion do not matter. Feel free to comment below and I’ll see what I can do for the next edition. And honestly, this is more a personal reference than anything else so having actual readers would already be awesome 🙂 Enjoy!

Read More

[Azure] News for Developers, April 2019

This entry is part 22 of 28 in the series Azure news for Developers

Are you having trouble keeping track of everything that’s going around in Azure? You’re not alone! In an effort to do so myself, I’m starting a monthly series called “News for developers” which is exactly that: a summary of all of the Azure flavored news specifically for software developers. Now this is based on my personal feeds and my personal opinion, so you might miss things or see things which in your opinion do not matter. Feel free to comment below and I’ll see what I can do for the next edition. And honestly, this is more a personal reference than anything else so having actual readers would already be awesome 🙂 Enjoy!

Read More

[Azure] News for Developers, March 2019

This entry is part 21 of 28 in the series Azure news for Developers

Are you having trouble keeping track of everything that’s going around in Azure? You’re not alone! In an effort to do so myself, I’m starting a monthly series called “News for developers” which is exactly that: a summary of all of the Azure flavored news specifically for software developers. Now this is based on my personal feeds and my personal opinion, so you might miss things or see things which in your opinion do not matter. Feel free to comment below and I’ll see what I can do for the next edition. And honestly, this is more a personal reference than anything else so having actual readers would already be awesome 🙂 Enjoy!

Read More

[Azure] News for Developers, February 2019

This entry is part 20 of 28 in the series Azure news for Developers

Are you having trouble keeping track of everything that’s going around in Azure? You’re not alone! In an effort to do so myself, I’m starting a monthly series called “News for developers” which is exactly that: a summary of all of the Azure flavored news specifically for software developers. Now this is based on my personal feeds and my personal opinion, so you might miss things or see things which in your opinion do not matter. Feel free to comment below and I’ll see what I can do for the next edition. And honestly, this is more a personal reference than anything else so having actual readers would already be awesome 🙂 Enjoy!

Read More

[Azure] Local Service Fabric cluster won’t hit debug breakpoints

Today one in the “what the heck!?”-category. I had setup a freshly installed VM with Service Fabric SDK and Visual Studio, to debug a Service Fabric application. When I ran the app from VS, it got deployed into Service Fabric, all lights green. But weirdly, none of my breakpoints in Visual Studio were hit. So I tried numerous things like checking the debug profile in VS, rebuilding, removing the application, etc. etc. All to no avail.

When I tried resetting the cluster itself, I got the following error:

Access Control Lists are usually tied to folders, but of course this very helpful error does not include the actual folder. In the event log, a number of warnings were also feature, like:

Hmmm ok… again not that helpful. I’ll spare the Google query I’ve undertaken and skip to the end result:

Add the “ServiceFabricAdministrators” and “ServiceFabricUsers” groups to C:\SFDevCluster with Full Control permissions. Those groups were not there in my case and once added, everything started behaving normally again. So not sure why they were not there, but adding them fixed everything. You’ll need to reset your cluster to get everything working afterwards.

[Azure] Setting custom domain with indirect verification via ARM

I ran across this problem trying to link a custom domain to an Azure storage account. Custom domains can be used to use your own domain instead of having <storageaccountname>.blob.core.windows.net as the default domain. For using a custom domain, the domain needs to be validated to ensure that you really own the domain you’re planning to use. There’s two ways of validating:  Read More