For the past few months we have been upgrading our CMS from Umbraco 4 to Umbraco 7. Development was going smoothly, until it was time to go live. Before deploying the site to the public, we wanted to do some thorough testing from the whole company so we set up HAProxy to route internal traffic to the new servers. Feedback on the new design was very positive, but we received a lot of complaints of really slow page loads… Occasionally some pages would take up to 45 seconds to load, which is absolutely crazy. The rest of the time, pages were still taking up to a couple of seconds to load. Whilst this is more reasonable, it was still much too slow for a release to the public.
After a focused investigation, we identified four separate issues:
- After each page publish, Umbraco wrote to its disk cache (App_Date/umbraco.config) about 4000 times, causing IIS to restart the app with the reason ConfigurationChanged
- Every 15 minutes we had 30 second delays due to app pool resets as default setting of numRecompilesBeforeAppRestart=15 being exceeded
- The default value of Umbraco.ModelsBuilder.ModelsMode=PureLive caused 5–10 second page loads every 15 minutes
- Each page was loading macros that took about 800ms
The chart below shows how the response time changed as we fixed each issue:
4000 disk writes after each page publish
Umbraco keeps a cache of the whole published site in an xml file. The default behaviour is to update this file every time a change is made in the back office. We have one root node in our Umbraco instance, with about 4000 indirect children. So it appears that Umbraco writes to the umbraco.config cache file once for every page on our site, even if only one page is changed.
We initially tried applying this hotfix as we were running Windows Server 2012 R2, but it didn’t help. In the end, the solution to this was to change the ContinouslyUpdateXmlDiskCache key from true to false. This fixed the issue with the app pool restarts after each publish, but does mean that when redeploying the site the cache must be built from the database rather than from file, which is a little slower.
After fixing the issue with app pool restarts after publishing pages, we were still getting occasional slowdowns. We set up a simple console app to ping the server every 3 seconds and left it on overnight to see if we could find any patterns. In the morning we saw slowdowns every 15 minutes, and slightly longer slowdowns every 2 hours or so.
Max Recompilations Reached
This one was quite a crude fix. We simply increased the numRecompilesBeforeAppRestart value, as we weren’t sure of the cause. We spent a little time investigating, but we were still getting slow downs every 15 minutes of up to 10 seconds without any app pool restarts.
Umbraco ModelsBuilder ModelsMode
Umbraco ships with a NuGet package called ModelsBuilder. This generates strongly typed models that can be used in Umbraco views. The default behaviour is to generate the models on the fly, in memory, at runtime. This has the advantage of the models always being up to date whenever a content type is modified. But after a considerable amount of investigation we found that it was also the reason for the slowdowns every 15 minutes. We changed the setting to Dll mode, which compiles the models to a Dll that is then copied into the bin folder. This fixed the 10s slowdowns, and also fixed the Max Recompilations Reached errors we were seeing above. The downside with this is that it requires a manual step after a deploy to build the models from within the back office. We are looking to integrate the model building into our deployment process to remove the manual step, as the rest of the process is automated.
After we solved the periodic issues, we were still left with slow page loads (800ms) on every page request. After some investigation the following three sections were found to be the culprits:
- Each page request we were building the main menu from the Umbraco cache each time, which involved iterating over nearly 100 pages. The solution to this was to use the Umbraco HtmlHelper CachedPartial extension method. We set the cache timeout to 24 hours, and the cache is also emptied each time a page is published, so we don’t have worry about updated content not appearing in the menu. This shaved around 100–200ms off of the page load.
- We show the latest tweet from our twitter account on the financials home page, and although we were caching the latest tweet, we were still authenticating with Twitter each time in case the cache was empty. We refactored this code to remove the Twitter API call, which saved 150–250ms.
- We have a macro that shows the latest market update blog post. We have quite a lot of blog posts (around 1000) so we need to iterate through all of these to find the one with the latest date. This was another 150–250ms, so we used the built in Umbraco macro caching to reduce the load time once the article has been found.
Umbraco is a great CMS, but for a large site such as ours, some of the default settings can be show stoppers. There is not a great deal of documentation around performance tuning, so looking for solutions was challenging. Hopefully this article will help point people that are having similar issues in the right direction.