Summary
On the AWS (S3 + MySQL) backend, garbageCollect is permanently wedged: every invocation fails with MalformedXML from s3:DeleteObjects, GCCoord.fromSize is never advanced, and no partial tile or entry bundle is ever collected. Because the failure is deterministic and fromSize never moves, the log retries the identical range forever and obsolete partials accumulate without bound.
We hit this on a production log running rekor-tiles v2.3.0 (tessera v1.0.2). It has failed every 60s since the log was created, and has collected exactly zero objects. The same code is present unchanged in v1.0.3, v1.0.4 and main.
Symptom
level=INFO msg="GarbageCollect failed: failed to delete one or more objects: failed to delete objects:
operation error S3: DeleteObjects, https response error StatusCode: 400,
api error MalformedXML: The XML you provided was not well-formed or did not validate against our published schema"
Root cause
deleteObjectsWithPrefix calls DeleteObjects unconditionally, even when ListObjectsV2 matched nothing:
l, err := s.s3Client.ListObjectsV2(ctx, &s3.ListObjectsV2Input{
Bucket: aws.String(s.bucket),
Prefix: aws.String(objPrefix),
})
...
di := &s3.DeleteObjectsInput{
Bucket: aws.String(s.bucket),
Delete: &types.Delete{
Objects: make([]types.ObjectIdentifier, 0, len(l.Contents)),
},
}
for _, k := range l.Contents { ... }
if _, err := s.s3Client.DeleteObjects(ctx, di); err != nil { // <-- empty Objects => MalformedXML
return fmt.Errorf("failed to delete objects: %v", err)
}The S3 DeleteObjects API requires Delete.Objects to contain at least one element. An empty list is rejected with 400 MalformedXML.
garbageCollect calls this with .p/ prefixes for every in-scope bundle:
eg.Go(func() error { return deleteWithPrefix(ctx, entriesPath(ri.Index, 0)+".p/") })
eg.Go(func() error { return deleteWithPrefix(ctx, layout.TilePath(0, ri.Index, 0)+".p/") })A bundle or tile that was only ever written in its full form has no .p/ objects, so that prefix matches nothing and the call fails.
Note this is not the more familiar ">1000 keys per DeleteObjects" variant of MalformedXML. ListObjectsV2 here is unpaginated (no ContinuationToken loop), so l.Contents can never exceed 1000 and the batch cannot overflow. The empty list is the only way this call produces MalformedXML.
Why it never recovers
In garbageCollect, the progress write happens after the errgroup:
if err := eg.Wait(); err != nil {
return fmt.Errorf("failed to delete one or more objects: %v", err)
}
if _, err := tx.ExecContext(ctx, "UPDATE GCCoord SET fromSize=? WHERE id=?", fromSize, 0); err != nil {A single failing prefix returns early, the transaction rolls back, and GCCoord.fromSize is never updated. The next run recomputes the same range and fails on the same prefix. This makes a transient-looking error permanent.
Worse, the very first bundle is usually affected, so GC never makes any progress. On our log we probed the .p/ sibling prefix of ten known-full objects; nine were empty, including the index-0 entries:
| prefix |
objects |
tile/entries/000.p/ |
0 |
tile/0/000.p/ |
0 |
tile/entries/x001/000.p/ |
0 |
tile/0/x001/000.p/ |
0 |
tile/1/000.p/ |
0 |
tile/2/000.p/ |
3 |
With fromSize = 0, the first two prefixes GC touches are both empty, so it fails before it can advance past index 0.
Impact
Unbounded growth of obsolete partial tiles and entry bundles. Our log is at ~8.4M objects / ~690 GB, growing ~200-270k objects and ~20 GB/day, roughly half of which are uncollected .p/ partials.
There is no way to work around this from configuration — rekor-server serve exposes no GC-related flag, and an S3 lifecycle rule cannot distinguish a superseded partial tile from a live one.
Suggested fix
Return early when there is nothing to delete:
if len(l.Contents) == 0 {
return nil
}Two adjacent issues worth considering in the same change:
- Paginate
ListObjectsV2. It currently returns at most 1000 keys with no ContinuationToken loop, so a prefix holding more than 1000 partials can never be fully collected.
- Let GC make partial progress. Since the deletes are idempotent, it may be preferable to advance
GCCoord.fromSize past ranges that succeeded rather than discarding the whole run's progress when one prefix errors.
I'm happy to send a PR for the guard plus pagination if that's useful.
Environment
- tessera
v1.0.2 (also present unchanged in v1.0.3, v1.0.4, main)
- via rekor-tiles
v2.3.0, AWS driver (S3 + Aurora MySQL)
Summary
On the AWS (S3 + MySQL) backend,
garbageCollectis permanently wedged: every invocation fails withMalformedXMLfroms3:DeleteObjects,GCCoord.fromSizeis never advanced, and no partial tile or entry bundle is ever collected. Because the failure is deterministic andfromSizenever moves, the log retries the identical range forever and obsolete partials accumulate without bound.We hit this on a production log running rekor-tiles
v2.3.0(tesserav1.0.2). It has failed every 60s since the log was created, and has collected exactly zero objects. The same code is present unchanged inv1.0.3,v1.0.4andmain.Symptom
Root cause
deleteObjectsWithPrefixcallsDeleteObjectsunconditionally, even whenListObjectsV2matched nothing:The S3
DeleteObjectsAPI requiresDelete.Objectsto contain at least one element. An empty list is rejected with400 MalformedXML.garbageCollectcalls this with.p/prefixes for every in-scope bundle:A bundle or tile that was only ever written in its full form has no
.p/objects, so that prefix matches nothing and the call fails.Note this is not the more familiar ">1000 keys per DeleteObjects" variant of
MalformedXML.ListObjectsV2here is unpaginated (noContinuationTokenloop), sol.Contentscan never exceed 1000 and the batch cannot overflow. The empty list is the only way this call producesMalformedXML.Why it never recovers
In
garbageCollect, the progress write happens after the errgroup:A single failing prefix returns early, the transaction rolls back, and
GCCoord.fromSizeis never updated. The next run recomputes the same range and fails on the same prefix. This makes a transient-looking error permanent.Worse, the very first bundle is usually affected, so GC never makes any progress. On our log we probed the
.p/sibling prefix of ten known-full objects; nine were empty, including the index-0 entries:tile/entries/000.p/tile/0/000.p/tile/entries/x001/000.p/tile/0/x001/000.p/tile/1/000.p/tile/2/000.p/With
fromSize = 0, the first two prefixes GC touches are both empty, so it fails before it can advance past index 0.Impact
Unbounded growth of obsolete partial tiles and entry bundles. Our log is at ~8.4M objects / ~690 GB, growing ~200-270k objects and ~20 GB/day, roughly half of which are uncollected
.p/partials.There is no way to work around this from configuration —
rekor-server serveexposes no GC-related flag, and an S3 lifecycle rule cannot distinguish a superseded partial tile from a live one.Suggested fix
Return early when there is nothing to delete:
Two adjacent issues worth considering in the same change:
ListObjectsV2. It currently returns at most 1000 keys with noContinuationTokenloop, so a prefix holding more than 1000 partials can never be fully collected.GCCoord.fromSizepast ranges that succeeded rather than discarding the whole run's progress when one prefix errors.I'm happy to send a PR for the guard plus pagination if that's useful.
Environment
v1.0.2(also present unchanged inv1.0.3,v1.0.4,main)v2.3.0, AWS driver (S3 + Aurora MySQL)