Hello, I'm seeing unexpected behavior with LocalStore.copy() when using a Databricks Unity Catalog Volume.
Environment:
- obstore: 0.11
- Databricks Runtime: 17.3
- Storage: Unity Catalog Volume (
/Volumes/...)
It works on a local temporary directory:
import tempfile
from obstore.store import LocalStore
tmpdir = tempfile.mkdtemp()
store = LocalStore(tmpdir)
store.put("a.txt", b"test")
store.copy("a.txt", "b.txt")
But it fails on a Unity Catalog Volume:
from obstore.store import LocalStore
store = LocalStore(
"/Volumes/conf/hydrologie/datasets/"
)
store.copy(
"file.csv",
"copy.csv"
)
Error:
GenericError: Generic LocalFileSystem error:
Unable to copy file from
/Volumes/conf/hydrologie/datasets/file.csv
to
/Volumes/conf/hydrologie/datasets/copy.csv:
Function not implemented (os error 38)
Read operations work correctly:
from obstore.store import LocalStore
store = LocalStore(
"/Volumes/conf/hydrologie/datasets/"
)
obj = store.get("file.csv")
print(len(obj.bytes()))
I also verified that copying the file using Databricks utilities works correctly:
dbutils.fs.cp(
"dbfs:/Volumes/conf/hydrologie/datasets/file.csv",
"dbfs:/Volumes/conf/hydrologie/datasets/copy.csv"
)
This suggests that:
- Access to the Unity Catalog Volume is working.
- Read and write permissions are available.
- The issue appears to be specific to the implementation used by
LocalStore.copy() on the /Volumes filesystem.
Thanks!
Hello, I'm seeing unexpected behavior with
LocalStore.copy()when using a Databricks Unity Catalog Volume.Environment:
/Volumes/...)It works on a local temporary directory:
But it fails on a Unity Catalog Volume:
Read operations work correctly:
I also verified that copying the file using Databricks utilities works correctly:
This suggests that:
LocalStore.copy()on the/Volumesfilesystem.Thanks!