DiscoverLinkedURLs Method (crawlUriList, discoveredURLs, pathMatchesToBeIgnored, pathMatchesToBeIncluded)

Keyoti SearchUnit v6

Crawls all URLs in urlList, and adds found URLs to discoveredURLs list (whether they are known to the document index or not, and not including urls in urlList)

Declaration Syntax

C#	Visual Basic

public void DiscoverLinkedURLs(
	ArrayList crawlUriList,
	ArrayList discoveredURLs,
	ArrayList pathMatchesToBeIgnored,
	ArrayList pathMatchesToBeIncluded
)

Public Sub DiscoverLinkedURLs ( 
	crawlUriList As ArrayList,
	discoveredURLs As ArrayList,
	pathMatchesToBeIgnored As ArrayList,
	pathMatchesToBeIncluded As ArrayList
)

Parameters

crawlUriList (ArrayList): List of string objects specifying Uris to be crawled

discoveredURLs (ArrayList): ArrayList of string objects with new URLs found through crawling

pathMatchesToBeIgnored (ArrayList): Any strings which will be used to determine if a file should be ignored based on its Url path.

pathMatchesToBeIncluded (ArrayList): Any strings which will be used to determine if a file should exclusively be included based on its Url path.

Remarks

Unlike Crawl, this method doesn't add new URLs to the 'database'. Crawl uses this method, overriding this method will affect Crawl. This override is most suitable for polling discoveredURLs from another thread, to track progress.

Assembly: Keyoti4.SearchEngine.Core (Module: Keyoti4.SearchEngine.Core.dll) Version: 2022.8.22.610