What happens
With client-side batching, when a whole chunk fails to be sent, BatchObjectReturn.errors
reports fewer failures than actually happened, and the surviving keys describe the wrong
objects.
BatchObjectReturn documents the opposite (weaviate/collections/classes/batch.py, class
docstring):
The keys of the errors and uuids dictionaries will always be equivalent to the
original_index of the objects as you added them to the batching loop
Root cause
The wholesale-failure handler in _BatchBase.__send_batch keys its errors by the position
inside the chunk that was sent:
# weaviate/collections/batch/base.py:648
errors_obj = {
idx: ErrorObject(message=repr(e), object_=obj) for idx, obj in enumerate(objs)
}
Every other place that fills this dictionary uses the global index — obj.index in
grpc_batch.py:157, sync.py:367/:410, async_.py:409/:452 — because
BatchObjectReturn.__add__ merges chunk results with a plain self.errors.update(other.errors)
(classes/batch.py:231) and does no re-keying. So with a chunk size smaller than the batch,
chunk N re-uses keys 0..len(chunk)-1 and overwrites chunk N-1's entries.
Two observable symptoms:
len(result.errors) under-reports the number of failed objects.
- If one chunk succeeds and a later chunk fails wholesale, the successful chunk's
uuids
keys and the failed chunk's errors keys collide and describe different objects —
result.uuids[2] and result.errors[2] refer to two different inputs, so a caller that
retries "the failed indices" re-inserts rows that already went in and skips rows that did not.
Reproduction
Offline, mock gRPC only, no server and no API keys. Four objects with
fixed_size(batch_size=2), and the mock rejects every object of both chunks (which makes
_BatchGRPC.objects raise WeaviateInsertManyAllFailedError, landing in that except Exception):
result.errors keys : [0, 1] # 4 objects failed
len(result.errors) : 2
len(failed_objects): 4
errors[0] -> object_.index=2 (input position 2)
errors[1] -> object_.index=3 (input position 3)
And for "first chunk succeeds, second fails wholesale": uuids keys [0, 1] while errors
keys are also [0, 1], describing objects 2 and 3.
Expected: errors keys [0, 1, 2, 3] in the first case and [2, 3] in the second, each
describing the object it names.
Tested on main @ 142d798, Python 3.11.15. I have a patch and mock_tests/ regression tests
for both symptoms, in #2166
What happens
With client-side batching, when a whole chunk fails to be sent,
BatchObjectReturn.errorsreports fewer failures than actually happened, and the surviving keys describe the wrong
objects.
BatchObjectReturndocuments the opposite (weaviate/collections/classes/batch.py, classdocstring):
Root cause
The wholesale-failure handler in
_BatchBase.__send_batchkeys its errors by the positioninside the chunk that was sent:
Every other place that fills this dictionary uses the global index —
obj.indexingrpc_batch.py:157,sync.py:367/:410,async_.py:409/:452— becauseBatchObjectReturn.__add__merges chunk results with a plainself.errors.update(other.errors)(
classes/batch.py:231) and does no re-keying. So with a chunk size smaller than the batch,chunk N re-uses keys
0..len(chunk)-1and overwrites chunk N-1's entries.Two observable symptoms:
len(result.errors)under-reports the number of failed objects.uuidskeys and the failed chunk's
errorskeys collide and describe different objects —result.uuids[2]andresult.errors[2]refer to two different inputs, so a caller thatretries "the failed indices" re-inserts rows that already went in and skips rows that did not.
Reproduction
Offline, mock gRPC only, no server and no API keys. Four objects with
fixed_size(batch_size=2), and the mock rejects every object of both chunks (which makes_BatchGRPC.objectsraiseWeaviateInsertManyAllFailedError, landing in thatexcept Exception):And for "first chunk succeeds, second fails wholesale":
uuidskeys[0, 1]whileerrorskeys are also
[0, 1], describing objects 2 and 3.Expected:
errorskeys[0, 1, 2, 3]in the first case and[2, 3]in the second, eachdescribing the object it names.
Tested on
main@142d798, Python 3.11.15. I have a patch andmock_tests/regression testsfor both symptoms, in #2166