KB: The import (crawl) or index operation stops suddenly when running under ASP.NET.

Knowledgebase Home Page > SearchUnit

Search the Knowledge Base

The import (crawl) or index operation stops suddenly when running under ASP.NET.
https://keyoti.com/kb/Default.aspx?ToDo=view&questId=187&catId=54

Options

Print this page
Email this to a friend

Problem Description

When running an import or index operation under an ASP.NET process (eg. the Admin web application included, or custom code run from an ASPX page), the operation stops and;

-It may show the login screen if you require a login (under the Admin web app.)

-It may not do anything for 30 minutes (note that sometimes during an import/index, the progress indicator may not move for a few minutes, this may be normal and due only to a large document being processed)

-It may show a "Server Application Unavailable" error message (indicating cause #1 below is the likely problem)

Possible Causes

1. Some customers with this issue have found that the cause is "ASP.NET process recycling". To diagnose this, look in the system Event Log for events related to ASP.NET

Eg.

>> > > +++++++++++++++++++++++++++++++++++++++++++++++
>> > >
>> > > Event Type: Error
>> > > Event Source: ASP.NET 2.0.50727.0
>> > > Event Category: None
>> > > Event ID: 1000
>> > > Date: 4/14/2008
>> > > Time: 2:20:12 PM
>> > > User: N/A
>> > > Computer: compname
>> > > Description:
>> > > aspnet_wp.exe (PID: 1396) stopped unexpectedly.
>> > >
>> > > +++++++++++++++++++++++++++++++++++++++++++++++

Note: the machine.config can actually be set to NOT log ASP.NET recycling events, if you don't find any events it is worthwhile to check the machine.config file to see if this is the case (see below and here http://msdn.microsoft.com/en-us/library/7w2sway1.aspx)

2. Something about the pages on your site is causing the search engine to fail. To diagnose this, enable Logging under Configuration (see Help for info on this) and check the .txt files output in the Index Directory for any information. You are welcome to email log files to support for us to analyse.

How To Solve "ASP.NET process recycling" Issues

First it's important to understand that recycling can occur due to a few possible reasons, and is considered 'normal'.

-set period of time (regardless of process health)
-set amount of idle time "
-set number of requests "

-maximum number of queued requests exceeded

-maximum amount of ASP.NET memory usage exceeded

The settings for these are in the machine.config

Therefore it's not possible to say what the exact cause will be without looking at the machine.config settings and understanding how the app. is running.

Secondly it's important to know that the cause of the recycling could be any application running on the server, including;

-a completely unrelated application

-the web application that is actually BEING indexed

-the web application that is DOING the indexing (eg. the Admin web app.)

Tips for finding the problem application;

1. Watch the memory use. If possible watch as the search engine operation runs to see if the memory use rises. It should be very near flat, and any high use (eg 100MB) is considered abnormal. Remember that it could be the search engine using the memory, OR the application it is indexing.

2. Look at the Event Log for other messages with ASP.NET as the source. If the application being indexed is recording exceptions, then these could be the cause, especially if they are close (in time) to the recycle event.

3. Use logging to find (eg. in the WebSiteSpider log file) the last URL being processed, this may point to a problem on that page that can be easily identified (such as it taking too long to run).

General Information About ASP.NET Recycling

The following was published on the GotDotNet web-site before it was shut-down. The following is a reproduction of it from the waybackmachine archive, for your convenience.

========================================================================

http://web.archive.org/web/20070410090501/http://samples.gotdotnet.com/quickstart/aspplus/doc/procmodel.aspx

Using the Process Model

- Process Model Configuration

- Reactive Process Recycling

- Proactive Process Recycling

- Logging Process Model Events

- Enabling Web Gardens

- Section Summary

One of the most important requirements for ASP.NET Framework applications is reliability. The architecture of applications running inside the server process (in IIS, Inetinfo.exe) does not produce a solid foundation for building reliable applications that can continue to run over a long period of time. Too many resources are shared on the process level, and it is too easy for an error to bring down the entire server process.

To solve this problem, ASP.NET provides an out-of-process execution model, which protects the server process from user code. It also enables you to apply heuristics to the lifetime of the process to improve the availability of your Web applications. Using asynchronous interprocess communication enables you to provide the best balance of performance, scalability, and reliability.

Process Model Configuration

Process model settings are exposed in the root configuration file for the computer, Machine.config. The configuration section is named <processModel> and is shown in the following example. The process model is enabled by default (enable="true").

<processModel
    enable="true"
    timeout="infinite"
    idleTimeout="infinite"
    shutdownTimeout="0:00:05"
    requestLimit="infinite"
    requestQueueLimit="5000"
    memoryLimit="80"
    webGarden="false"
    cpuMask="0xffffffff"
    userName=""
    password=""
    logLevel="errors"
    clientConnectedCheck="0:00:05"
/>

Most of these settings control when a new worker process is started to serve requests (gracefully taking the place of an old worker process). The process model supports two types of recycling: reactive and proactive.

Reactive Process Recycling

Reactive process recycling occurs when a process is misbehaving or unable to serve requests. The process typically displays detectable symptoms, such as deadlocks, access violations, memory leaks, and so on, in order to trigger a process recycle. You can control the conditions that trigger a process restart by using the configuration settings described in the following table.

Setting	Description
requestQueueLimit	Handles deadlock conditions. The DWORD value is set to the maximum allowed number of requests in the queue, after which the worker process is considered to be misbehaving. When the number is exceeded, a new process is launched and the requests are reassigned. The default is 5000 requests.
memoryLimit	Handles memory leak conditions. The DWORD value is set to the percentage of physical memory that the worker process can consume before it is considered to be misbehaving. When that percentage is exceeded, a new process is launched and the requests are reassigned. The default is 80%.
shutdownTimeout	Specifies the amount of time the worker process has to shut itself down gracefully (string value in hr:min:sec format). When the time out expires, the ASP.NET ISAPI shuts down the worker process. The default is "00:00:05".

Proactive Process Recycling

Proactive process recycling restarts the worker process periodically even if the process is healthy. This can be a useful way to prevent denials of service due to conditions the process model is unable to detect. A process can be restarted after a specific number of requests or after a time-out period has elapsed.

Setting	Description
timeout	String value in hr:min:sec format that configures the time limit after which a new worker process will be launched to take the place of the current one. The default is infinite, a keyword indicating that the process should not be restarted.
idleTimeout	String value in hr:min:sec format that configures the amount of inactivity, after which the worker process is automatically shut down. The default is infinite, a keyword indicating that the process should not be restarted.
requestLimit	DWORD value set to the number of requests after which a new worker process will be launched to take the place of the current one. The default is infinite, a keyword indicating that the process should not be restarted.

Logging Process Model Events

The process model can write events to the Windows event log when process cycling occurs. This is controlled by the logLevel attribute in the <processModel> configuration section.

Setting	Description
logLevel	Controls that process cycling events are logged to the event log. The value can be: All: All process cycling events are logged. None: No events are logged. Errors: Only unexpected events are logged.

When a cycling event occurs, if logging is enabled for that event, the following events are written to the application event log.

Shutdown Reason	Event Log Type	Description
Unexpected	Error	The ASP.NET worker process has shut down unexpectedly.
RequestQueueLimit	Error	The ASP.NET worker process has been restarted because the request queue limit was exceeded.
RequestLimit	Information	The ASP.NET worker process has been restarted because the request limit was exceeded.
Timeout	Information	The ASP.NET worker process has been restarted because the time-out interval was met.
IdleTimeout	Information	The ASP.NET worker process has been shut down because the idle time-out interval was met.
MemoryLimitExceeded	Error	The ASP.NET worker process was restarted because the process memory limit was exceeded.

Enabling Web Gardens

The process model helps enable scalability on multiprocessor computers by distributing the work to several processes, one per CPU, each with processor affinity set to its CPU. This eliminates cross-processor lock contention and is ideal for large SMP systems. This technique is called Web gardening. The configuration settings for enabling Web gardens are listed in the following table. Note that these settings take effect only after a server is restarted. IIS must be cycled in order for this change to take place.

Setting

Description

webGarden

Controls CPU affinity. True indicates that processes should be affinitized to the corresponding CPU. The default is False.

cpuMask

Controls the number of processes and how the Web garden works. One process is launched for each CPU where the corresponding bit in the mask set to 1. When UseCPUAffinity is set to 0, the cpuMask setting only controls the number of worker processes (number of bits set to 1). The maximum-allowed number of worker processes is the number of CPUs. By default, all CPUs are enabled; the same number of worker processes is launched as there are CPUs. The default value is 0xffffffff.

Web gardening has some side effects that you should be aware of:

If your application uses session state, it must choose an out-of-process provider (NT Service or SQL).
Application state and application statics are per process, not per computer.
Caching is per process, not per computer.

Section Summary

ASP.NET provides an out-of-process execution model, which protects the server process from user code. It also enables you to apply heuristics to the lifetime of the process to improve the overall availability of Web applications.
The <processModel> settings are exposed in the root configuration file for the computer's Machine.config file. The process model is enabled by default.
The process model supports two types of recycling: reactive and proactive. Reactive process recycling occurs when a process is misbehaving or unable to serve requests. Proactive process recycling restarts the worker process periodically, even when the process may be healthy.
The process model can write events to the Windows event log when process cycling occurs. This is controlled by the log-level attribute in the <processModel> configuration section.
The process model helps enable scalability on multiprocessor computers by distributing the work to several processes, one per CPU, each with processor affinity set to its CPU. This technique is called Web gardening.

=================================================================