Common Causes and Troubleshooting Sitecore App Restarts on Azure Web App Services

This post is divided into two parts; the first part will go through some common causes that lead to random Sitecore app pool restarts on Azure App Services. The second part will provide some useful troubleshooting approaches to identify the causes.

Common Causes

• Website Dynamic Cache is enabled
  Please ref to this Sitecore Knowledge Base article for more detail on how to turn off Website Dynamic Cache for Sitecore apps hosted on Azure Web App Services: Sitecore XP stability issues when using Azure Web Apps 
• File Change Notification (FCN) is enabled
  This was provided by Sitecore support as a solution when our app kept on crashing even after setting Website Dynamic Cache to false in Azure. The change is basically web.config change on <httpRuntime fcnMode="Disabled" .../>. The original Reference is: https://shazwazza.com/post/all-about-aspnet-file-change-notification-fcn/
  Also, Sitecore support suggested increasing number for numRecomplesBeforeAppRestart in the web.config like this: 

<compilation defaultLanguage="c#" debug="false" targetFramework="4.5.2" numRecompilesBeforeAppRestart="999">

• Auto Heal is on
  This needs to be set specifically, so it's not on by default. References can be found here:  https://azure.microsoft.com/en-us/blog/auto-healing-windows-azure-web-sites/ and looks like it can also be done from KUDO:  https://benperk.github.io/waws/2015/2015-11-auto-heal-your-azure-web-app.html
• Proactive Auto Heal is on
  This is normally on by default for all Azure web apps. However, it's not that easy to be triggered; if this is the real reasons that your app is restarting, you have bigger problems to solve than just the random restarts.
  Reference can be found here:  https://koukia.ca/proactive-auto-heal-on-azure-app-services-8d136d7e894e
• Always on is not on
  This should be enabled by default if your instance is above a certain level of the pricing tier. More details can be found here:  https://docs.microsoft.com/en-us/azure/app-service/configure-common. It disables both periodic recycle and idle timeout recycle.
• Azure Maintainance
  Yep, Azure machines need to be restarted sometime too 😉 If your app is a single instance, this most likely happens during Azure maintenance, Azure moves your app to a new instance however the starting process still occurs.
  Those platform upgrades and restarts usually can be found when you use the Web App Restarted tab mentioned below. One way to avoid this is to make sure that you have multiple instances, as Microsoft stated, they won’t apply at the same time to all the apps in the same plan: https://feedback.azure.com/forums/169385-web-apps/suggestions/37746457-high-availability-app-service. Another way to reduce the impact is to use App Initialisation Module. Please see more details here: https://ruslany.net/2015/09/how-to-warm-up-azure-web-app-during-deployment-slots-swap/

Some troubleshoot approaches

• Web App Restarted tab
  In your Azure Web App "Diagnose and solve problems" tab, search for "Web App Restarted". It can provide you with details of why the app restarted, including Azure maintenance information, which is excellent information when sometimes you looked everywhere and couldn't see why the app was restarted.
• Event logs
  In the same tab above, search for "Application Events". You can also find the raw XML files by using FTP into the application, but this is a much easier way to troubleshoot.
• Sitecore logs
  This official Knowledge Base article provides a full picture for Sitecore log viewing in Azure: https://kb.sitecore.net/articles/911837
  Don't want to repeat the same things, just some additional notes here:
• If "Application Logging (Filesystem)" is enabled in "App Service Logs" tab, Live logs can be viewed from "Log Stream" tab. Server logs can be seen following the same way, which is helpful when the issue is on IIS or .Net level.
• Other than Analytics in Application Insight, you can use "Search" to see the latest traces as well. However, you will have more controls in Analytics by using queries.
• AzureTools.aspx is the latest helper page Sitecore provides to view Sitecore logs by using Azure Application Insights REST APIs. It also supports Azure Search. There is an older version which is more lightweight and works without requiring an IIS/.Net environment can be downloaded here: AzureAILogs.html. You can also modify the HTML and save app ID and key directly with the file, then just place it in your CM Sitecore Admin folder. So you can use it just like how you use logs.aspx before. (Note, make sure security is applied before it's exposed like this)