fix(storage): improve robustness of cloud storage operations#890
Conversation
*** * Make key fetching deterministic for S3-compatible stores by improving sorting logic and handling objects that lack standard metadata * Add robust error handling to AWS S3 lookups to gracefully handle missing credentials or failed API calls * Update development configuration defaults for consistent use of local MinIO/RustFS endpoint settings
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b5eda51222
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - MINIO_SECRET_KEY | ||
| - MINIO_SECURE | ||
| # Export storage defaults to host-published local RustFS/MinIO-compatible storage. | ||
| - EXPORT_SERVICE=${EXPORT_SERVICE-core.services.storages.cloud.minio.MinIO} |
There was a problem hiding this comment.
When docker-compose.yml is used without an explicit EXPORT_SERVICE, this line now injects core.services.storages.cloud.minio.MinIO into the API container, overriding the application default in core/settings.py of core.services.storages.cloud.aws.S3. In production-style compose deployments that already provide AWS credentials/bucket but have never set EXPORT_SERVICE, exports and downloads will now try to initialize MinIO against host.docker.internal:${MINIO_PORT:-9090} instead of S3, and the base compose file does not start a RustFS/MinIO service. This should stay opt-in for the local MinIO setup rather than changing the default backend for every compose run.
Useful? React with 👍 / 👎.
* Remove redundant MinIO service configurations from local docker-compose setup * Refine cloud storage tests by removing specific integrity and permission checks
1 participant
I did it so rustfs and oclweb2 can run seamless with export tests locally @snyaggarwal
Make key fetching deterministic for S3-compatible stores by improving sorting logic and handling objects that lack standard metadata
Add robust error handling to AWS S3 lookups to gracefully handle missing credentials or failed API calls
Update development configuration defaults for consistent use of local MinIO/RustFS endpoint settings