Crawls web-sites to find new links.
Declaration Syntax| C# | Visual Basic |
public class WebSiteSpider
Public Class WebSiteSpider
Members| All Members | Constructors | Methods | Properties | Events | |
| Icon | Member | Description |
|---|---|---|
 | WebSiteSpider(Configuration) |
Creates a new instance.
|
 | AddedDocument |
Fired when a document has been added to the index.
|
| AddingDocument |
Fired when a document is about to be added to the index.
|
| CancelCrawl()()()() |
Cancels the crawl.
|
| Close()()()() |
Closes the index for use
|
 | Configuration |
Gets the instance of the Configuration class that holds the settings to be used.
|
| Crawl()()()() | Obsolete.
Starts the spidering process, which will recrawl all the existing documents for new links.
|
| Crawl(String, Boolean) | Obsolete.
Starts the spidering process, which will recrawl all the existing documents for new links.
|
| Crawl(String, Boolean, WebsiteBasedIndexableSourceRecord) | Obsolete.
Starts the spidering process, which will recrawl all the existing documents for new links.
|
| Crawl(ArrayList) | Obsolete.
Crawls all URL strings in urlList, adds new document URLs (including those from urlList)
to the 'database' and returns list of newly found URLs.
|
| Crawl(ArrayList, ArrayList) | Obsolete.
Crawls all URL strings in urlList, adds new document URLs (including those from urlList)
to the 'database' and returns list of newly found URLs.
|
| Crawl(ArrayList, ArrayList, WebsiteBasedIndexableSourceRecord) | Obsolete.
Crawls all URL strings in urlList, adds new document URLs (including those from urlList)
to the 'database' and returns list of newly found URLs.
|
 | CreateNewDocument(DocumentRecord) |
Called whenever a Document object is created, override this method to use Document subclasses.
|
| DataAccess |
The data-access layer.
|
| DiscoverLinkedURLs(ArrayList, ArrayList, ArrayList) |
Crawls all URLs in urlList, and returns list of all newly found
URLs (whether they are known to the document index or not, and not including urls in urlList)
|
| DiscoverLinkedURLs(ArrayList, ArrayList, ArrayList, ArrayList) |
Crawls all URLs in urlList, and adds found URLs to discoveredURLs list (whether they are known to the document index or not, and not including urls in urlList)
|
| DiscoverLinkedURLs(ArrayList, ArrayList, ArrayList, ArrayList, ArrayList, Boolean, ArrayList, WebSiteSpider..::..DiscoveredDocumentHandler) |
Crawls all URLs in urlList, and adds found URLs to discoveredURLs list (whether they are known to the document index or not, and not including urls in urlList)
|
| DocumentShouldBeCrawled(Document) | Obsolete.
Method called from Crawl - checks if the current document type is
specified as to be crawled in config file, returning true - if so
(files types from config file are stored in class member when the object is instantiated)
|
| Equals(Object) | Determines whether the specified Object is equal to the current Object. (Inherited from Object.) |
| Finalize()()()() | Allows an object to try to free resources and perform other cleanup operations before it is reclaimed by garbage collection. (Inherited from Object.) |
 | ForceHttps(Uri) |
Converts an HTTP Uri to HTTPS.
|
| GetHashCode()()()() | Serves as a hash function for a particular type. (Inherited from Object.) |
| GetIndexedDocuments()()()() | Obsolete.
Gets all the indexed documents from the document table, as an ArrayList of Document objects
|
| GetType()()()() | Gets the type of the current instance. (Inherited from Object.) |
| IsDocumentToBeCrawled(Document, Boolean, Boolean, String, ArrayList, ArrayList) |
Method called during crawl - checks if the current document type is
specified as to be crawled in config file, returning true - if so
(files types from config file are stored in class member when the object is instantiated)
|
| MemberwiseClone()()()() | Creates a shallow copy of the current Object. (Inherited from Object.) |
| NewLinkNo |
Number of new links found so far.
|
| OnReaderExceptionOccurred(ReaderException, Uri) |
Called whenever a readerException is caught
|
| Open()()()() |
Opens the index for use.
|
| PrefilterDocumentToCrawl(Document) |
Called during crawl to identify if the document might be crawled. Will return false if the document won't be crawled based on it's URL.
|
| ProcessedLinkNo |
The number of processed links.
|
| ReaderExceptionOccurred | Fired whenever the (web) reader encounters a network exception, eg. 404 (this can be useful for identifying dead links) |
| ToString()()()() | Returns a string that represents the current object. (Inherited from Object.) |
Remarks
Before any operations (Crawl, GetIndexedDocuments) can be performed the index must
be opened with Open (otherwise a KeyotiFatalException is thrown) - and should be closed afterwards with Close.
Inheritance Hierarchy| Object | |
 | WebSiteSpider |
Assembly: Keyoti4.SearchEngine.Core (Module: Keyoti4.SearchEngine.Core.dll) Version: 2022.8.22.610