Friday, March 07, 2014

Tips to diagnose and Debug IIS web application crash issue with DebugDiag and Windbg

The WebPortal application pool crash once a while in a customer production web server. The error couldn’t be recreated on the development environment as it only happened in a certain condition. I spent some times to investigate this issue and finally used DebugDiag and Windbg with some help from Microsoft to find out the root cause of this issue. I shared this tips to trouble shoot this issue and hopefully it may help if you are encountering the similar issue.
Evidence: In the window event log, there is warning message “A process serving application pool 'WebPortal' suffered a fatal communication error with the Windows Process Activation Service. The process id was '3000'. The data field contains the error number.” 
The client also report that the website was down some times and had to reset the IIS to get it back online.
Configuration Changes in IIS application pool
a. Open IIS Manager and expand Application Pools
b. Right-click the Application Pool and choose Recycling.
c. Under the Edit Application Pool Recycling settings, uncheck all the recycling settings and click Next and then click on Finish.
d. Right click the application pool and click on Advanced Settings.
e. Under the Process Model section change the Ping Enabled to false and configure Idle Time-out (minutes) to 0
f. Under the Rapid-Fail Protection section, change Enabled to False
g. Click OK
h. Recycle the application pool for these settings to take effect.

Create a Crash Rule in Debug Diagnostic 1.2 with the following steps: 
a. Open DebugDiag (Start -> Programs -> Debug Diagnostic Tool 1.2)
b. Select "Crash" and click Next
c. If the crashing process is w3wp.exe then choose the option  A specific IIS application pool and click Next and choose the application pool that is crashing from the list and click Next. (Note: The Application pool list may be empty if the IIS 6 Metabase Compatibility is not installed. In such a case you can always type the name of the application pool and click Next)
d. If the crashing process is not w3wp.exe then choose the option A specific process and click Next and choose the process name from the list and click Next.
e. Click Next in Advanced Configuration (Optional), click on Breakpoints and then click on Add Breakpoint
f. Choose NTDLL!ZwTerminateProcess from the list and change the Action Type to Full User Dump and Action Limit to 5 and click on Ok. Once you hit OK, the window should look like this.
g. Click on Save and Close 
h. Click Next for "Rule Name". The "Userdump Location" can be changed here. 
i. Select "Activate the rule now" and click Finish
Notice the Status is Active. The Userdump Count will increase
Wait for the crash dump file generated and Run windbg.exe and open crash dump file with the following steps:
a. Go to file->Open crash dump, then selected the dump file. A sample dump file name is w3wp__WebPortal__PID__3000__Date__02_21_2014__Time_04_47_54PM__250__Ntdll!ZwTerminateProcess.dmp
b. Go to File->Symbol File Path to make sure the symbol path is SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
c. Run .cordll -ve -u –l to verify the required .net assembly
d. Run the following command
!clrstack in the windbg, the dump message will display as followings. From this message, we can see there is infinite call in iVision.BusinessObjects.DataAccessHelper.GetNextChild from  iVision.UserControls.SponsorChild.uc_SponsorChildDetail.uc_SponsorChildbtnNext_Click

1d22e0b4 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e118 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e17c 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e1e0 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e244 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e2a8 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e30c 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e370 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e3d4 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e438 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e49c 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e500 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e564 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e5c8 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e62c 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e690 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e6f4 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e758 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e7bc 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e820 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e884 1d3167b1 iVision.BusinessObjects.DataAccessHelper.GetNextChild(System.Collections.Generic.List`1 ByRef, System.Guid)
1d22e8e8 1b4e928e iVision.BusinessObjects.DataAccessHelper.FindNextChild(ReleaseCycleTypes, Int32, System.Guid, Int32 ByRef, Int32 ByRef, Int32 ByRef, Int32 ByRef, Int32 ByRef, Boolean ByRef, Boolean ByRef, Boolean ByRef, System.Collections.Generic.List`1 ByRef)
1d22ed40 1d31641b iVision.UserControls.SponsorChild.uc_SponsorChildDetail.uc_SponsorChildbtnNext_Click(System.Object, System.EventArgs)
1d22ed5c 66adcfb0 System.Web.UI.WebControls.LinkButton.OnClick(System.EventArgs)
1d22ed74 66add030 System.Web.UI.WebControls.LinkButton.RaisePostBackEvent(System.String)
1d22ed8c 66adcfcb System.Web.UI.WebControls.LinkButton.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(System.String)
1d22ed94 66adb26e System.Web.UI.Page.RaisePostBackEvent(System.Web.UI.IPostBackEventHandler, System.String)
1d22eda0 66adb250 System.Web.UI.Page.RaisePostBackEvent(System.Collections.Specialized.NameValueCollection)
1d22edb4 66aedb7e System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
1d22ef0c 66aed404 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
1d22ef44 66aed331 System.Web.UI.Page.ProcessRequest()
1d22ef7c 66aed2c6 System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
1d22ef88 66aed2a2 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
1d22ef9c 00cb2fa5 ASP.wvipages_sponsorchild_aspx.ProcessRequest(System.Web.HttpContext)
1d22efa0 66af3776 System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
1d22efd4 66ac5c2c System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
1d22f014 66ad1723 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception)
1d22f064 66ac51ac System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)
1d22f080 66ac871c System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)
1d22f0b4 66ac83c3 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest)
1d22f0c4 66ac755c System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32)

Wednesday, March 06, 2013

Query to get the store procedure text

SELECT ROUTINE_NAME, ROUTINE_DEFINITION
    FROM INFORMATION_SCHEMA.ROUTINES
    WHERE ROUTINE_DEFINITION LIKE '%foobar%'
    AND ROUTINE_TYPE='PROCEDURE'



Friday, September 14, 2012

strip out the time portion in sql server

SELECT CAST(FLOOR(CAST(getdate() AS FLOAT)) AS DATETIME)

Thursday, September 13, 2012

How to delete the window services

Recently, I was trying to delete a windows service. Normally it should not be necessary to manually delete a service. Uninstalling an application should remove its associated service (if any).
However, I installed some beta products and a service created by one of the applications was not removed automatically. Its very easy to remove a service from registry if you know the right path. Here is how I did that:
1. Run Regedit or Regedt32
2. Find the registry entry "HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services"
3. Look for the service there and delete it. You can look at the keys to know what files the service was using and delete them as well (if necessary).
alternatively, you can also use command prompt and delete a service using following command:
sc delete < SERVICE name>

or to create, simply type
sc create
Update:
If you have space in the file path you need to use quotation marks ("). For example:
sc create "MySQL" binpath= "C:\Archivos de programa\MySQL\MySQL Server 5.1\bin\mysqld.exe"
Thanks to Maya.
NOTE: You may have to reboot the system to get the list updated in service manager.

Monday, September 10, 2012

Specified argument was out of the range of valid values



Error:
System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.  Parameter name: utcDate     at System.Web.HttpCachePolicy.UtcSetLastModified(DateTime utcDate)     at System.Web.HttpCachePolicy.SetLastModified(DateTime date)     at Telerik.Web.UI.CombinedScriptWriter.WriteCombinedScriptFile()     at Telerik.Web.UI.WebResource.ProcessRequest(HttpContext context)     at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()     at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Resolution:
for %i in (*.dll) do copy /b %i+,,

in the Bin directory of the application.
it means that the assembly was built in future according to the webserver, the command above will 'touch' them so they appear to be of current date

Tuesday, July 31, 2012

PORTQRY COMMAND TO TEST THE PORT LISTENING

portqry -n MyMailServer -p tcp -e 25


SMTP Connectivity TEST

telnet your.server.com 25
Connected to ....
Escape character is '^]'
220 your.server.com ESMTP
mail from: user@yourdomain.com
250 ok
mail subject: test
250 ok
rcpt to: someone@aol.com
250 ok
data
354 go ahead
Here is a test message body text.
.
250 ok ...
quit