In continuation from my blog post Visual Studio 2017 Load Testing of an Azure Web App with Application Insights Part 1 of 2 , I have shown the web app and azure environment setup that is prepared for a load test of 10,000 simulated users. In this post, I will show the execution, some of the observed metrics and share my opinions, tips and challenges.
The Visual Studio Load Test Result
If you click on the Detail button in the tool bar, you can see the step load pattern of virtual users. Over the course of 15 minutes, I had virtual users run through any of the 3 recorded webtests. As I scroll to the end of the test, I can identify a virtual user of id 9998. Therefore, this shows that I have very closely achieved my 10,000 user test load target.
Azure App Service
The overview blade shows monitoring metrics.
Azure Application Insights
Live Metrics Stream
The following shows real time monitoring within the last 60 second window.
Request rate hitting upwards of 100/s and some request duration taking around 2.5 seconds.
Process CPU averaging ~50%
Performance Overview
Server
While the load test was executing, I turned to my browser to manually test responsiveness as I click into the UI and make edits. The responsiveness was slow to 5-7 seconds.
Azure SQL – Performance Overview
Max CPU consumption close to 100%. Green box highlight is the duration of the load test.
A really neat feature of Azure SQL is the ability to review and accept action of performance recommendations. The following shows two indexes being created and another two being validated for creation. Creating indexes will help with query performance.
App Service Plan
Auto Scale
The following shows the auto scale rules being enforced to increase the instance count due to a load test and then decrease back to 1 instance when CPU average < 5% over a 15 min period.
My thoughts and opinion on my load test scenario and results show that an app service plan and azure SQL DB can take decent load for a relatively small traffic site of 10,000 users, but will be noticeably slow for end users. Therefore, in this situation consider scaling out to more 2 more instances and/or scale up to a larger compute size. Also increase the DTU of the Azure SQL DB to perhaps 20 DTU.
Tips and suggestions collected from this load testing experience:
- Azure SQL DB Performance Issue
When the Azure SQL DB was set at Basic pricing tier at 5 DTU, I had the following message in Live Metrics Stream telemetryTime 9:32:38 PM Duration 10.46 sec Outgoing Command tcp:ABC-cc.database.windows.net,1433 | ABC | Open Result code 40613 ErrorMessage Database 'ABC' on server 'ABC-cc' is not currently available. Please retry the connection later. If the problem persists, contact customer support, and provide them the session tracing ID of '7AB82EFB-53D7-4EB8-A875-81D06BFBEE06'.
To resolve, I set the pricing tier to Standard at 10 DTU and executed 3 more load tests and was not able to reproduce this issue.
- VS Load test system out of memory exception.
Upon load test completion, one of the errors you may see is an Out of memory exception.
The VS Load test application is 32-bit. And so even though I used a VM with 32 GB of RAM, the memory usage is limited since it is using a 32 bit address space.For details and background read https://blogs.msdn.microsoft.com/mahipalkante/2010/06/10/visual-studio-2010-load-test-out-of-memory-exceptions-on-32-bit-machines-or-32-bit-run/
- No metrics from load test into App Insights Browser monitoringIn App Insights Browser blade, if you don’t notice any metrics from your VS load test, that is because the simulation of the tests are recorded http requests and don’t involve the app insights JavaScript code running which would be in the client-side browser.
- Pushing towards max load users of >15,000
In my load testing environment of 8GB RAM, I seemed to manage 10,000 users but ran into various issues when trying to push around 15,000 and more. Careful planning of the load test, web test, Visual Studio settings and the hardware specs are needed to push for much more load of virtual users.
- Regarding multiple Azure VMs to distribute load tests
According this article Distributing Load Test Runs Across Multiple Test Machines Using Test Controllers and Test Agents,
“Test Controllers and Test Agents and not supported on Azure Virtual Machines. To perform distributed load testing for Azure VMs, use the cloud-based load testing service.” - Cloud-based testing is limited to 20,000 virtual user minutes for free
Through your visual studio team services (or visual studio online), you can run these cloud-based load tests. There is no option through Azure. I find that the free option is very little for high load you will use the free capacity very quickly. And so I opted for doing my load test on virtual machines.
For details read:
https://docs.microsoft.com/en-us/vsts/billing/buy-load-testing-vs#_buy-load-testing
https://www.visualstudio.com/team-services/cloud-load-testing/
https://docs.microsoft.com/en-us/vsts/load-test/get-started-simple-cloud-load-test
Roy Kim on Azure and Microsoft 365 