Can't pass DataType between Python and Rust - 'DataType' object cannot be cast as 'DataType'

[rabernat]

Hi! Thanks Kyle for this great project! 🙌

I have a little side project about bridging the gap between Zarr and Arrow data types. (Yes, I know about existing efforts in this space. This is something different. 😁 ) I decided to explore arro3 and pyo3_arrow as a way of sending arrow data type definitions between Python and Rust.

Unfortunately I'm stuck on something that feels a bit trivial. I'm also a Rust n00b, so apologies if there is some obvious mistake here. Here is the most minimal example I can come up with

Rust module in src/lib.rs

use pyo3::prelude::*;
use pyo3_arrow::PyDataType;

#[pyfunction]
fn parse_data_type(data_type: &PyDataType) {
    let arrow_data_type = data_type.as_ref();
}


#[pymodule]
fn zarr_arrow_dtypes(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(parse_data_type, m)?)?;
    Ok(())
}

Python code

from arro3.core import DataType
from zarr_arrow_dtypes import parse_data_type  # data_type_to_json

data_type = DataType.int32()
parse_data_type(data_type)
# -> TypeError: argument 'data_type': 'DataType' object cannot be cast as 'DataType'

Cargo.toml

[package]
name = "zarr_arrow_dtypes"
version = "0.1.0"
edition = "2024"

[dependencies]
pyo3-arrow = "0.15.0"

[dependencies.pyo3]
version = "0.27.0"
features = ["chrono-tz"]

This is a puzzling error which I don't really know how to debug. Is pyo3_arrow::PyDataType not the same thing as arro3.core.DataType?

[kylebarron]

tl;dr: I think it might be just because you have fn parse_data_type(data_type: &PyDataType) instead of fn parse_data_type(data_type: PyDataType).

---

It's not a noob question. A background issue is this: PyO3/pyo3#1444. In general, you can't take a pyo3-based Python class that was created in one library and share it with another pyo3-based library, because differences in Rust package version, pyo3 version, python version, and rustc versions mean that it might not be ABI stable.

Except in the case of Arrow we have a format that is explicitly ABI stable. So the point of pyo3-arrow is to ensure that this sort of data sharing works because we access the object's underlying C-stable data.

That error is saying that this Arrow data sharing isn't getting activated, and it's trying to extract the input as the type exported from your module. So it's saying that the two types don't match because they come from different modules.

The question is why the Arrow data sharing isn't activated and I suspect that it's because you're trying to access the data as a reference, and we only implement FromPyObject for the owned type

arro3/pyo3-arrow/src/ffi/from_python/datatypes.rs

Lines 6 to 13 in 62b52e9

	impl<'py> FromPyObject<'_, 'py> for PyDataType {
	type Error = PyErr;
	fn extract(obj: Borrowed<'_, 'py, PyAny>) -> Result<Self, Self::Error> {
	let capsule = call_arrow_c_schema(&obj)?;
	Self::from_arrow_pycapsule(&capsule)
	}
	}

I wonder if we could also implement it for the reference to avoid this kind of issue.

This is also why in the README we try to suggest people to return data by passing the data from Rust into the runtime-available arro3-core module. Because if you return a PyArray from a function, that will be the version of pyo3-arrow compiled with your library instead of the version the user chose when installing arro3-core.

(It was worth going through all this detail because I suspect we'll have a similar discussion about icechunk python soon 😉)

[rabernat]

Thanks Kyle, very helpful!

tl;dr: I think it might be just because you have fn parse_data_type(data_type: &PyDataType) instead of fn parse_data_type(data_type: PyDataType).

Unfortunately this change had no effect; same error.

But I think I understand the general problem better--small differences between the pip-installed, runtime-available arro3-core module and my compiled module are likely the source of the type incompatibility.

[kylebarron]

Hmm, let me debug. That was background but it should be irrelevant for arro3 (that's the point of pyo3-arrow)

[rabernat]

Scratch that... it does work! I didn't rebuild properly after making the change.

Thank you!

[kylebarron]

In #448 I can't reproduce your error. Does that example help you?

I can't tell what you're doing differently?

[kylebarron]

Ok yeah, I tested on that branch and it was the reference that's the issue. You need to put it in your signature without a reference.