Commit 11ff8ee
committed
[Enhancement](test) Add tests for Python UDF (apache#60499)
Some versions of pyarrow do not support directly converting Python's
`dict` type to the `arrow Map` type, so we need to manually convert the
`dict` type to a `list of tuple` before converting it to `arrow Map`
(consistent with the type mapping in `convert_arrow_field_to_python`).
```python
def convert_arrow_field_to_python(field, column_metadata=None):
if pa.types.is_map(field.type):
# pyarrow.lib.MapScalar's as_py() returns a list of tuples, convert to dict
list_of_tuples = field.as_py()
return dict(list_of_tuples) if list_of_tuples is not None else None
```1 parent 34a827d commit 11ff8ee
25 files changed
Lines changed: 7130 additions & 4 deletions
File tree
- be
- src
- runtime
- udf/python
- test
- runtime
- udf/python
- regression-test
- data/pythonudf_complex_p0
- pipeline
- cloud_p0/conf
- cloud_p1/conf
- suites/pythonudf_complex_p0
- py_udf_complex_scripts
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
| 65 | + | |
65 | 66 | | |
| 67 | + | |
66 | 68 | | |
67 | 69 | | |
68 | 70 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
373 | 373 | | |
374 | 374 | | |
375 | 375 | | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
376 | 394 | | |
377 | 395 | | |
378 | 396 | | |
| |||
0 commit comments