hop-run freeze

[jeromechafr]

Hello,
I imported a pipeline from Kettle.
This workflow has 172 steps
After some adaptations (principaly on javascript steps) , the workflow run in 130 seconds when i used the Debug button in hop-gui

2026/01/29 15:36:53 - X3_V6_ventes_commandes_bl_factures_avec_montants_en_devise_prog - Pipeline duration : 129.471 seconds [  2'  9.471" ]

2026/01/29 15:36:53 - X3_V6_ventes_commandes_bl_factures_avec_montants_en_devise_prog - Execution finished on a local pipeline engine with run configuration 'local'

When i use the Run button on hop-gui the duration is around 30 minutes (!!!) with the same dataset (30 000 lines) and the same 'run configuration' (local).

And when i used the command hop-run the pipeline freeze after 'Transform initialisation'

.........
2026/01/28 16:32:44 - X3_V6_ventes_commandes_bl_factures_avec_montants_en_devise_prog - Transform [Table input.0] initialized flawlessly.

2026/01/28 16:32:44 - X3_V6_ventes_commandes_bl_factures_avec_montants_en_devise_prog - Transform [Qté, HT et TTC pour Factures des commandes 2.0] initialized flawlessly.

2026/01/28 16:32:44 - X3_V6_ventes_commandes_bl_factures_avec_montants_en_devise_prog - Transform [Données factures.0] initialized flawlessly.

2026/01/28 16:32:44 - X3_V6_ventes_commandes_bl_factures_avec_montants_en_devise_prog - Transform [Qté, HT et TTC pour Factures des BL.0] initialized flawlessly.

2026/01/28 16:32:44 - X3_V6_ventes_commandes_bl_factures_avec_montants_en_devise_prog - Transform [Données factures pour BL.0] initialized flawlessly.

Do you have a idea about this issue ?

Nota : "small" pipeline (30 steps) don't have this issue.

J.C

[dmainou]

You are running out of memory or CPU.
You can increase the amount of memory allocated to the JVM (look for the amount allocated in the other system)

However, the rule of thumb is 4-5 steps per core. i.e. 34-43 cores just for the JVM (you'll need more for the OS and other apps.
This is true for Kettle as well.

You are likely processing just a few lines in kettle and therefore why it doesn't fail.

I would recommend that you divide this pipeline into seperate items and orchestrate it with a job.
Your performance will likely increase as well.

[dmainou]

I recently imported this pipeline and migrated it into one pipeline or workflow per shape with an overarching workflow orchestrating the lot.

[sramazzina]

It seems a good idea my friend ;-)

[jeromechafr]

Hi,
It seems to be a good idea to split my huge pippeline into multiple readdable pipeplines !!
I think i have to use "Serialize To File" step to transfer data from a pipeline to another.
Thanks

[dmainou]

You can write to in memory SQLite or DuckDb

[enricomariac]

You can olso serialize in memory with VFS: ram:///any/path/to/file

[jeromechafr]

It works !!!
I splitted my pipeline into 9 pipelines, created a worflow to orchestrating the lot and call this worflow in the 'master' workflow.
For the moment i used local file with (De)Serialize steps.
Thanks

[dmainou]

Cool!