-
Notifications
You must be signed in to change notification settings - Fork 37
Closed
Description
Got the following error after preparing a task, uploading the test_operator.tar given in the project repository.
[INFO]
2026-01-27 20:04:51.609 | INFO | datamate.wrappers.executor:__init__:32 - Initing Ray ...
[WARNING]
2026-01-27 20:04:54,846 WARNING services.py:2137 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 67076096 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=4.72gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM.
[INFO]
2026-01-27 20:04:56,983 INFO worker.py:1998 -- Started a local Ray instance. View the dashboard at �[1m�[32mhttp://127.0.0.1:8265 �[39m�[22m
[WARN]
/usr/local/lib/python3.11/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
[WARN]
warnings.warn(
[INFO]
2026-01-27 20:04:58.045 | INFO | __main__:run:30 - Loading dataset with Ray...
[INFO]
2026-01-27 20:04:59.344 | INFO | __main__:run:41 - Processing data...
[INFO]
2026-01-27 20:04:59.354 | INFO | datamate.core.dataset:load_ops_module:146 - Import Ops module TestMapper Success.
[INFO]
2026-01-27 20:04:59.356 | INFO | __main__:run:45 - All Ops are done in 0.012s.
[INFO]
2026-01-27 20:04:59,371 INFO logging.py:397 -- Registered dataset logger for dataset dataset_2_0
[ERROR]
2026-01-27 20:04:59,388 ERROR exceptions.py:73 -- Exception occurred in Ray Data or Ray Core internal code. If you continue to see this error, please open an issue on the Ray project GitHub page with the full stack trace below: https://github.com/ray-project/ray/issues/new/choose
[ERROR]
2026-01-27 20:04:59,389 ERROR exceptions.py:81 -- Full stack trace:
[ERROR]
Traceback (most recent call last):
[ERROR]
File "/usr/local/lib/python3.11/site-packages/ray/data/exceptions.py", line 49, in handle_trace
[ERROR]
return fn(*args, **kwargs)
[ERROR]
^^^^^^^^^^^^^^^^^^^
[ERROR]
File "/usr/local/lib/python3.11/site-packages/ray/data/_internal/plan.py", line 474, in execute_to_iterator
[ERROR]
bundle_iter = execute_to_legacy_bundle_iterator(executor, self)
[ERROR]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR]
File "/usr/local/lib/python3.11/site-packages/ray/data/_internal/execution/legacy_compat.py", line 44, in execute_to_legacy_bundle_iterator
[ERROR]
day, stats = _get_execution_dag(
[ERROR]
^^^^^^^^^^^^^^^^^^^
[ERROR]
File "/usr/local/lib/python3.11/site-packages/ray/data/_internal/execution/legacy_compat.py", line 146, in _get_execution_dag
[ERROR]
dag = get_execution_plan(plan._logical_plan).dag
[ERROR]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR]
File "/usr/local/lib/python3.11/site-packages/ray/data/_internal/logical/optimizers.py", line 113, in get_execution_plan
[ERROR]
physical_plan = plan(optimized_logical_plan)
[ERROR]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR]
File "/usr/local/lib/python3.11/site-packages/ray/data/_internal/planner/planner.py", line 165, in plan
[ERROR]
physical_dag, op_map = self._plan_recursively(
[ERROR]
^^^^^^^^^^^^^^^^^^^^^^^
[ERROR]
File "/usr/local/lib/python3.11/site-packages/ray/data/_internal/planner/planner.py", line 205, in _plan_recursively
[ERROR]
physical_op = plan_fn(logical_op, physical_children, data_context)
[ERROR]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR]
File "/usr/local/lib/python3.11/site-packages/ray/data/_internal/planner/plan_udf_map_op.py", line 296, in plan_udf_map_op
[ERROR]
return MapOperator.create(
[ERROR]
^^^^^^^^^^^^^^^^^^^
[ERROR]
File "/usr/local/lib/python3.11/site-packages/ray/data/_internal/execution/operators/map_operator.py", line 422, in create
[ERROR]
return ActorPoolMapOperator(
[ERROR]
^^^^^^^^^^^^^^^^^^^^^
[ERROR]
File "/usr/local/lib/python3.11/site-packages/ray/data/_internal/execution/operators/actor_pool_map_operator.py", line 156, in __init__
[ERROR]
per_actor_resource_usage = ExecutionResources(
[ERROR]
^^^^^^^^^^^^^^^^^^^
[ERROR]
File "/usr/local/lib/python3.11/site-packages/ray/data/_internal/execution/interfaces/execution_options.py", line 34, in __init__
[ERROR]
self._cpu: Optional[float] = safe_round(cpu, 5)
[ERROR]
^^^^^^^^^^^^^^^^^^
[ERROR]
File "/usr/local/lib/python3.11/site-packages/ray/data/_internal/execution/interfaces/execution_options.py", line 371, in safe_round
[ERROR]
elif math.isinf(value):
[ERROR]
^^^^^^^^^^^^^^^^^
[ERROR]
TypeError: must be real number, not str
[INFO]
2026-01-27 20:04:59.459 | INFO | datamate.sql_manager.sql_manager:_get_engine:50 - Database Engine initialized successfully.
[ERROR]
ray.data.exceptions.SystemException
[ERROR]
[ERROR]
The above exception was the direct cause of the following exception:
[ERROR]
[ERROR]
Traceback (most recent call last):
[ERROR]
File "/opt/runtime/datamate/wrappers/datamate_executor.py", line 74, in <module>
[ERROR]
raise e
[ERROR]
File "/opt/runtime/datamate/wrappers/datamate_executor.py", line 71, in <module>
[ERROR]
executor.run()
[ERROR]
File "/opt/runtime/datamate/wrappers/datamate_executor.py", line 47, in run
[ERROR]
for _ in dataset.data.iter_batches():
[ERROR]
File "/usr/local/lib/python3.11/site-packages/ray/data/iterator.py", line 193, in _create_iterator
[ERROR]
) = self._to_ref_bundle_iterator()
[ERROR]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR]
File "/usr/local/lib/python3.11/site-packages/ray/data/_internal/iterator/iterator_impl.py", line 27, in _to_ref_bundle_iterator
[ERROR]
ref_bundles_iterator, stats = self._base_dataset._execute_to_iterator()
[ERROR]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR]
File "/usr/local/lib/python3.11/site-packages/ray/data/dataset.py", line 6737, in _execute_to_iterator
[ERROR]
bundle_iter, stats, executor = self._plan.execute_to_iterator()
[ERROR]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR]
File "/usr/local/lib/python3.11/site-packages/ray/data/exceptions.py", line 89, in handle_trace
[ERROR]
raise e.with_traceback(None) from SystemException()
[ERROR]
TypeError: must be real number, not str
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels