`virtual_hosted_style_request=True` needs the bucket in the endpoint — undocumented, and no error when it's missing

[mscolnick]

obstore 0.11.0, Python 3.13

obstore and boto3 interpret the endpoint differently in virtual-hosted mode:

	boto3 (addressing_style="virtual")	obstore (virtual_hosted_style_request=True)
endpoint you pass	service root (https://cwobject.com)	full bucket host (https://my-bucket.cwobject.com)
bucket host derived by	client (prepends my-bucket.)	caller

Coming from boto3, passing the service root is the natural thing to do:

store = S3Store(
    "my-bucket",
    endpoint="https://cwobject.com",  # service root, boto3-style
    virtual_hosted_style_request=True,
)
obstore.list(store).collect()

The bucket then appears nowhere in the request. This makes env-driven args for endpoint not-reusable (and currently not overridable #753)

[kylebarron]

Thanks for the issue!

It isn't the goal for obstore to be similar to boto3. Obstore follow the exact behavior of the upstream object_store library, with as minimal modifications as possible to make maintenance as easy as possible.

That said, it would be good to document that the bucket is required in the URL (I've never used boto3 for virtual hosted requests). Would you like to make a PR?

Our docs are, in general, copied from upstream docs, so this could also be a good thing to raise upstream.

[kylebarron]

Also, this is already documented:

https://developmentseed.org/obstore/latest/api/store/aws/#obstore.store.S3Config.endpoint

[mscolnick]

You are right, it is documented. I may have just misunderstood that I needed to include that in the endpoint kwarg - I read it as my actual endpoint needed the bucket name it in.

This makes an env-driven endpoint (AWS_ENDPOINT) with multiple buckets not usable though unfortunately. And some cloud services inject AWS_ENDPOINT which then cannot be overwritten because of #753

[kylebarron]

This makes an env-driven endpoint (AWS_ENDPOINT) with multiple buckets not usable though unfortunately. And some cloud services inject AWS_ENDPOINT which then cannot be overwritten because of #753

I think this would have to be an upstream change?

I thought that object_store behaved this way because it was the best way to handle non-AWS implementations of S3, but I'm not very familiar with this part of the upstream codebase

[mscolnick]

Yea maybe an upstream change. I'm honest not sure what virtual_hosted_style_request does then if its not moving the bucket from the path into the hostname - that is what that config does.

[kylebarron]

I don't remember exactly on the top of my head, but virtual hosted style requests change a few things about how the request is actually sent to the backend. I think how the bucket/path are passed in is orthogonal to turning on this type of requests.

I don't think upstream would change the style of paths it needs, since that would be a backwards incompatible change.