MSSQL Indexing Limits - SearchUnit - Forum

Hello, I've been using Keyoti Search for some time now with very good results. However we need to add a DB source to our search results for 1.8 Million records.

I began the indexing with no paging in my query and is successfully imported 573,500 items.

When I attempted to use paging, the Management Tool was throwing an exception. I believe this was caused by our Unique Field index starting at 8700 instead of 0.

My second attempt was to utilize paging but adding 8700 to the page indexes. This was successful without throwing any exceptions. My first attempt of indexing 500 more pages didn't add any items to my results.

My third attempt was to add 573,500 to my paging index. I let the import run for 5,000 records. Again this did not add any more items to my result set. It is still at 573,500 items.

Is there a limit to the amount of items we can index? Our Index Text for each item is around 100 characters, not very large.

Any help with this is appreciated.
Joe

Joe, let me address some points please.

1. Yes you are right if you use the paging query from our help

SELECT * FROM Table1 where id>={0} AND id<{1}

then {0} is going to be zero based, and your idea to add to it is correct. So that should have worked.

2. The purpose of paging is only to address the issue of moving a large data set from the DB to the indexer machine, and holding it in memory. But, since your initial attempt managed to index a third of your records my guess would be that paging isn't required.
Imagine trying to index a 20GB index in one go, that would all have to be sent down the wire and held before indexing could start.
Your index (assuming the chars are stored as 8 bytes each) is only about 120MB.

3. What I say above and that it got to 573,500 items with or without paging makes me think something else is wrong, unrelated to paging. And to answer your question there are no limits, beyond what is practically possible with the equipment - it should be able to do your index OK.

Questions for you:
When it stopped at 573,500, was that because you stopped it? Did it just seem like it had stalled?

If it stopped itself, then it would be useful to enable logging in the configuration and do it again, then send me all .txt files (via support at keyoti.com)

Best
Jim



-your feedback is helpful to other users, thank you!

Hi Jim, Thanks for your reply.

Questions for you:
When it stopped at 573,500, was that because you stopped it? Did it just seem like it had stalled?

The import had been running for more than 12 hours, my estimation was that it would take less than 7 hours to complete. I watched it for a while to see if it would increase. It seemed like it was stalled so I stopped it.

I will turn on logging when I start it again this evening.
Thanks again for your help on this. Keyoti seems to be very quick with 500,000 results.

Joe

Jim,
Thanks for your help.
On Thursday and Friday I was able to index 600,000 DB records. On Friday evening I re-started the index and it ran for 60 hours. It only indexed 250,000 new records.

I Assume as the index gets larger the indexing slows down. Is this correct?
Joe