Set up minio-based testing on CI

[kylebarron]

Some functionality, like the upcoming substring-based list prefixes (#545) have to be tested through an S3/GCS/Azure store, not a memory store / local store.

I think it would be ideal to have a CI-based minio test framework.

@maxrjones Do I recall that you recently set up minio CI testing for a project? Was that virtualizarr? If so, would you be able to point me to that setup/configuration?

[maxrjones]

Yeah, I setup minio for VirtualiZarr in zarr-developers/VirtualiZarr#507. The relevant changes are to pyproject.toml, conftest.py, and virtualizarr/tests/conftest.py. IIUC the history of this setup is fsspec -> zarr -> virtualizarr; there may be a more modern approach. In particular, that setup does support running the tests that use minio in parallel, which is why they're defined using a separate test env and run command.

Note that minio is for testing S3 compatible APIs; you'll need something like azurite for Azure (xref zarr-developers/zarr-python#3146). Someone in that issue mentioned https://github.com/fsouza/fake-gcs-server as an option for GCS.

[kylebarron]

Yeah, I setup minio for VirtualiZarr in zarr-developers/VirtualiZarr#507.

🙏

Note that minio is for testing S3 compatible APIs; you'll need something like azurite for Azure (xref zarr-developers/zarr-python#3146). Someone in that issue mentioned fsouza/fake-gcs-server as an option for GCS.

The object_store crate already tests against all of these. So I don't need to test against all of them myself, I just need to test against each variant of approaches in my bindings. In #545 I just have two variants: (S3/GCS/Azure) and (Local files/memory store), so it's ok if I just test against S3.

[ion-elgreco]

Try rustfs :p