Performance Impact and Troubleshooting IIS Worker (W3PWP.EXE) Process.

Performance of web applications hosted in IIS can be affected by the worker process reaching or utilizing the CPU resources to maximum and shows 100% which caused the subsequent requests to pile up due to lack of resources. I wrote this article as I had similar problems with one of the projects on the production machine which hindered users, and we got to hear a lot of complaints of the application being down and slow.

The IIS worker process utilising CPU resources can be due to various reasons. In this blog, we will investigate some of the top possibilities and how to troubleshoot them.

Some of the common reasons which might cause the IIS worker process (w3pwp.exe) to utilize the maximum CPU resources w3pwp.exe are:

1. Requests getting blocked or stuck in the asp.net pipeline
2. High Error rates in the web application
3. High Web Traffic
4. Garbage collection
5. Inefficient .net code
6. Outdated or Issues with the application dependencies

I will only cover certain important points in the scenarios above and dig deeper into point 1, to identify and analyse it.

Requests getting blocked or stuck in the asp.net pipeline

The first thing to do here is to look at the requests which are currently running and investigate whether they are stuck or hanged. This might give a brief indication of which requests are taking a long time or getting stuck. There might be requests which are queued and not directly part of the problem or cause for alarm.

Identification

There are many steps in the lifecycle of an ASP.NET request. The basic steps include authentication, authorisation, evaluation of the http request, and finishing the request. If facing any performance problems, the first place to start is by identifying the specific http module.

Open IIS management console, you can view the running worker process and select the IIS application pool in question which is causing high CPU and view current running requests.

Select the worker processes from the IIS, you can see the currently running IIS worker processes.

Double click on the application pool name, and it will let you view all the currently executing requests. You can see that each request is in different parts of the ASP.NET pipeline and currently executing different HTTP modules.

The above snapshot gives you the information below for the currently executing requests.

• URL: The complete URL that is being executed
• Time: The total amount of time in milliseconds, the web request has been executing
• Client: The IP address of the user that initiated the request
• State: the stage of the IIS pipeline that the request is currently running in
• Module: the ASP.Net module that is currently executing.

Analysis

To dig deeper into the reason for the actual issue, you can obtain a memory dump from the w3wp.exe process similar to below:

Then use tools like debugdiag by Microsoft or visual studio, to analyse the memory dump and narrow down the threads which are causing the issue for the request to get stuck. Let’s see how we can understand the same.

Hit Start Analysis which will generate a report like a sample below, containing the top threads by CPU time and the details if any errors are occurring in the threads which are causing them to get stuck.

Open the same file in visual studio as below and running analysis on the threads will give you much more information for you to identify and act upon.

Click on Run Diagnostic Analysis on the top right, of the image above to get the details further identify the root cause.

Possible Solutions

The exception on threads will give a clear indication of which part of the code is causing the thread to break and the request to get stuck. Some pointers which I have come across.

1. In the case above we had a scenario, concurrency dictionary modification pitfalls
2. Check and set the application app pool to reset to a timely manner to clear out.
3. Another possible scenario is to increase RequestQueueLimitPerSession.

High Error rates in the web application

Another probable reason for the high CPU usage by the worker process is the high number of application errors which you can be unaware of, some of which can be masked by the generic message that users get, or a common message. We can ensure we identify the errors using error monitoring or application performance management tool. Other places or more common places to look for these are the IIS logs generated, these contain the details of the request, the response from it, any issues with the request, network-related failures etc. This should be the ideal place to start your investigation.

Also, you can look at the Performance Monitor in windows to analyse it with the counters as below:

• NET CLR Exceptions -> # of Exceptions Thrown / sec: Check this to see if a lot of exceptions are being thrown by your application. It is possible for your application to have a lot of hidden errors that can cause big performance problems. Any exceptions at all are bad, but some are unavoidable.
• W3SVC_W3WP -> % 500 HTTP Response Sent: Any requests with a 500-status code are internal server errors. Make sure this percentage is very low. It should be 0–1%.

High Web Traffic

Sometimes one of the simple explanations for the high CPU usage by worker processes could be the increase in web traffic.

You can use the IIS log files to see the number of requests pooled for the particular day, You can use tools like IIS log browser to analyse the requests this gives out major details from timestamp, URL, status, failure reason, client IP etc. and also gives you reports to work out of to identify most hit URL, most failed URL etc.

Garbage Collection

Microsoft .NET utilizes garbage collection to manage the allocation and release of memory.

Depending on what your application does, the allocation and clean-up of application memory can cause a lot of garbage collection activity. For example, the usage of lots of large string variables on the large object heap causes garbage collection problems.

To measure if garbage collection could be causing problems for your application, check this Windows Performance Counter:

.NET CLR Memory -> % Time in GC: This counter reflects what % of the time your application spends doing garbage collection. If this number spikes a lot above 5–10%, you want to investigate memory usage further.

Garbage collection also has two modes. You may need to enable server mode, which is not the default.

Inefficient .NET Code

Inefficient .NET code needs to be looked at if nothing else pans out. You can use profilers such as .NET Profiler or ANTS performance profiler to capture functions being called and analyse them further. Profilers provide an overhead look on CPU usage so if you are already on 85%+ it would be difficult to run a profiler which might cause it to go to 100% and make the application unusable.

Outdated or Issues with the application dependencies

Having third-party dependencies leads to a version being used which is old and vulnerable, Identify the dependencies on the application and update them regularly to be up to date on security, loopholes, memory leaks, or any other issues which might occur due to these dependencies which don’t log clearly.

Conclusion

Hopefully, after reading this article, you will have a starting point to analyse your performance problem, improve it and not at a critical stage where your application is going for a toss because of the 100% CPU usage. I hope this brief article has given you a perspective and guided you to take the necessary steps to tackle the problem.

About the Author:
Darshan Raviprakash is a Senior .Net Developer here at Version 1.