my index state is green still there are left over indexed vectors why ?

[vihariazure]

hi team
below are the get collection details

i inserted 10k , after data insert , i changed the indexing_threshold=10000

i got below get collection details

status=<CollectionStatus.YELLOW: 'yellow'> vectors_count=None, indexed_vectors_count=0, points_count=10000

after 1hr i check it again. below are the details
Status: green, Indexed Vectors Count: 4000

i know when the status is green i can go for search. but the indexed vectors should equal to points_count right for ideal case.

the status is green. it means i can go for search which will drop some accuracy.
when will be the left over vectors also will be indexed.

what are the parameters or steps i need to make for getting indexed_vectors_count=points_count

[timvisee]

10k points is a relatively low amount. In that case it's entirely possible that not all points are indexed, because the current state it is in is already fast enough.

You could reduce the indexing threshold which will likely increase the number of indexed points. Though with just 10k points it'd also be fine to leave as is, and see what happens once you insert a lot more points.

Note that indexing_threshold=10000 means that the threshold is 10MB on each segment. It's possible that you have some segments that are smaller, so optimizations are not triggered.

[vihariazure]

@timvisee

thanks for the reply
i tired inserting 100k records ,
it took 4m with batch size of 2000

1. what is data size limit for upsert ? in terms of mb

i check the collection details after inserting. after 2hrs below are the details
Status: green, Indexed Vectors Count: 96000, points_count=100000

status is green , i can go for search ? if i go for search there is any drop in accuracy
2) what is the reason for indexed vectors count is less then points count even after 2hrs also.

the parameters are default i did not changes any thing for building index

3. i have 1m vectors which are indexed. if i add 100k after 1 day . does it will reindex everything (1m +100k). or how it happens in that case

[timvisee]

The remaining 4000 vectors are probably in a small segment, which is why there currently is no need to optimize that portion. This is expected behavior, and there's nothing to worry.

Yes, 100% of your vectors is properly searchable while being indexed or not. Accuracy of the non-indexed vectors is 100%, so there's nothing to worry there either.

what is the reason for indexed vectors count is less then points count even after 2hrs also.

Internally, Qdrant uses segments. Over time the goal isn't to index 100% of the vectors, but rather to index segments that reached a certain size threshold.

i have 1m vectors which are indexed. if i add 100k after 1 day . does it will reindex everything (1m +100k). or how it happens in that case

It depends on how your data is currently distributed. Generally speaking it does not reindex everything.

One tip might be to temporarily disable the indexers while doing a bulk upload of data. Disabling it during upload and enabling it after will likely result in a higher indexed vector percentage. Read more about it here.