exclude files from indexing. - SearchUnit - Forum

Hi Jim,
How can I avoid to index files with extantions like (.js, .mdb, .dll) . Is there configiration I can use to exclude this type of files from indexing. The plugin that I use to create lastModifiedDate Crashed for some reason when trying to add date to custom data from httpwebResponse.

Thanks,
Alex

ag

Ok, Here is my plugin. I try to put the logic not to add unwanted files in MetaCustomData.

Public Sub CentralEventDispatcher_Action(ByVal sender As Object, ByVal e As ActionEventArgs)

Try

'Log everything - comment this line after debugging to optimize speed.
'DataAccess.Log.WriteLogEntry("PluginProject", "Ready", conf)

If e.ActionData.Name = ActionName.ResponseFromServerReceived Then

lastResponseURI = CType(e.ActionData.Data, Net.HttpWebResponse).ResponseUri
lastResponseLastModified = (CType(e.ActionData.Data, Net.HttpWebResponse)).LastModified

End If

If e.ActionData.Name = ActionName.ReadText Then

If lastResponseURI.ToString.IndexOf(".dll", System.StringComparison.OrdinalIgnoreCase) = -1 And _
lastResponseURI.ToString.IndexOf(".css", System.StringComparison.OrdinalIgnoreCase) = -1 And _
lastResponseURI.ToString.IndexOf(".js", System.StringComparison.OrdinalIgnoreCase) = -1 And _
lastResponseURI.ToString.IndexOf(".jpg", System.StringComparison.OrdinalIgnoreCase) = -1 And _
lastResponseURI.ToString.IndexOf(".gif", System.StringComparison.OrdinalIgnoreCase) = -1 And _
lastResponseURI.ToString.IndexOf(".png", System.StringComparison.OrdinalIgnoreCase) = -1 And _
lastResponseURI.ToString.IndexOf(".xml", System.StringComparison.OrdinalIgnoreCase) = -1 And _
lastResponseURI.ToString.IndexOf(".rpt", System.StringComparison.OrdinalIgnoreCase) = -1 And _
lastResponseURI.ToString.IndexOf(".ini", System.StringComparison.OrdinalIgnoreCase) = -1 And _
lastResponseURI.ToString.IndexOf(".ssi", System.StringComparison.OrdinalIgnoreCase) = -1 And _
lastResponseURI.ToString.IndexOf(".swf", System.StringComparison.OrdinalIgnoreCase) = -1 Then

If lastResponseURI = CType(sender, Document).Uri Then
CType(e.ActionData.Data, DocumentText).MetaCustomData = lastResponseLastModified.ToShortDateString()
End If

End If




End If



Catch ex As Exception

Throw New Exception("Source Description: " & ex.Message & ControlChars.CrLf & _
"Source Trace: " & ex.StackTrace)

End Try

End Sub

thanks,
Alex

Hi Jim,
I still have this error: Object reference not set to an instance of an object. Even that I checked for lastResponseLastModified being nothing.

Thanks,
Alex

ag

No That was e.actionData.Data is nothing I actually got it work. But now I have this:
Exception: A fatal exception has prevented the search engine from continuing. counter: 2000

its getting frustrating. I will disconnect plugin to see if this is plugin issue.

ag

Thanks for the files - so there are server errors when it makes requests for files like

http://slwebdev.xxxx/_vti_cnf/Global.asax
and other resource type files, dlls etc.

Each of these errors is counted and when it gets to 2000 it stops - you can change that limit in the configuration, just scan it for the number 2000 and increase to whatever you like, its only there for sanity.

Better though is to narrow down what you want to import, with the target match list, in the import params (click 'More') - add items like .doc, .pdf .aspx

One other thing to note, with ASPX pages, they don't always import very well from the filesystem import, because GET parameters obviously don't exist for the file sys import, so if you have links like default.aspx?id=1234 then you'll need a website import to pick them up.

Jim






-your feedback is helpful to other users, thank you!

It's just for information, if there are only 474 files being imported, then it's a problem, but if you imported 1000 files, it's probably fine. You can always check in the Reader.txt log file, it will list the server errors, and anything suspicious, you can try out the URL in a browser.

-your feedback is helpful to other users, thank you!