Feature "async" - "slave" controller getting synchronized with a hardware

[Nibanovic]

This PR implements the "async slave controller" functionality present in realtime_tools PRs:

• Feature "async" - "slave" execution of HW Components enabling synchronizaiton to a robot controller and therefore better communicaiton stability. realtime_tools#473
• Feature "async" - "slave" controller getting synchronized with a hardware realtime_tools#478

In short:

• ability to spawn async hardware and have it synchronized to an external robot clock via blocking read() (first PR)
• ability to pin async controllers to that hardware so their update() is executed between hardware read/write. This is mediated via signaling thread wakeups using the Monitor pattern

For details and validation, check the comments in their respective PRs.

[mergify]

@Nibanovic, all pull requests must be targeted towards the master development branch.
Once merged into master, it is possible to backport to jazzy, but it must be in master
to have these changes reflected into new distributions.

[github-actions]

This PR is stale because it has been open for 45 days with no activity. Please tag a maintainer for help on completing this PR, or close it if you think it has become obsolete.

[mergify]

This pull request is in conflict. Could you fix it @Nibanovic?

[saikishor]

Personally, I'm not a big fan of this approach. At least from what I understood from the code, it basically might run in SLAVE mode here too and wait for hardware sync to run the update cycle? or run in synchronized mode and then wait for the hardware sync to happen before running the update cycle.

This means the user have to explicity set the hardware they are relying on to every controller and this is not intuitive at all. IMO, we should try to explore the functionality as How Orocos handles ports and you can query if the new data is available and then trigger the rest of the chain, that way it is already working as you required.

[Nibanovic]

Before I saw your message @saikishor , based on feedback from @urfeex , I've identified some deadlock possibilities with this implementation and made changes to amend them.

---

This means the user have to explicity set the hardware they are relying on to every controller and this is not intuitive at all.

I see your point, and I kind of agree. Even though this works, implementing it was really tough, sorting out these signals. Controllers should be more loosely coupled, and users shouldn't hae to specify which specific hardware their controller waits on.

Although this approach I proposed works, it got really complicated since previous PR ros-controls/realtime_tools#473

---

IMO, we should try to explore the functionality as How Orocos handles ports and you can query if the new data is available and then trigger the rest of the chain, that way it is already working as you required.

This is a really interesting approach which I was not aware of. Something like building these OldData/NewData/NoData concepts into the state/command interfaces?

I think this warrants exploring if we want to consider a robust synchronization features in ros2_control, and this would mean a significant change to how execution of component read/update/write would be handled. I see three approaches we could take:

A simplest case to consider would be a case where we have:

• main cm thread + hw interface read/write + controller 1 update
• hw interface thread with blocking read() + controller 2 update() + write()

The question is, **how do we elegantly "slot" the controller 2 update to execute between async hw interface read/write?

1. split ros2_control threads per control cycle, and not per component

In this case, we'd add a function which would be something like control_cycle_done() for each control loop in our system.
Then, each thread we spawn would have a full control cycle (read/update/write), and different "triggers" which signal when can the next control loop start.
For the simplest case, in ros2_control_node, this control_cycle_done() would be just sleep(update_period - time_spent_on_read_update_write_this_cycle) For a second thread that is slave to a robot controller, control_cycle_done()would be the hardware_interface blockingread()`.

In each of those thread we'd then do a full read/update/write of all components that are a part of that control cycle, and finish it with a control_cycle_done() function.

2. Orocos-style state/command interface timestamps for new/old/none data

This would be some mechanism where we timestamp the data in state/command interfaces (either exact timestamp, or some enum for new/old/none data) so we know when it is posted by the hw interface on states, or when new commands are posted by the controllers.

Then, the async hw interface can see the current time and the timestsamp of the command and either disregard it, or interpolate it closer to current time or something else, giving us the loose coupling we need between controller and hardware components.

This feels connected to an older idea of timestamping the data inside the handles: #331

This way we could get event-driven scheduling of read/update/writes based on when new states/commands are posted.

Conclusion

After talking with @destogl and based on feedback from @saikishor , we're stopping work on this PR. We've given this solution a shot, but it is not a good enough solution and we should do better. We still learned a lot about this problem during implementation.

ros-controls/realtime_tools#473 is still open and ready for review, as it is completely encapsulated withing realtime_tools package, and it atleast eliminates drift between robot controller and our ros2_control node

[urfeex]

Thank you for the efforts, though! Another idea that I think @destogl also had in the past is doing some kind of interpolation in the hardware to resolve synchronizing between the controller and the hardware communication loop. Maybe, we can discuss this in the next WG meeting.

[saikishor]

Thank you for the efforts, though! Another idea that I think @destogl also had in the past is doing some kind of interpolation in the hardware to resolve synchronizing between the controller and the hardware communication loop. Maybe, we can discuss this in the next WG meeting.

Yes, that would be an ideal solution that way you can rate limit your commands or commands that mininize the jerk is what it is interesting

[Nibanovic]

Thank you for the efforts, though! Another idea that I think @destogl also had in the past is doing some kind of interpolation in the hardware to resolve synchronizing between the controller and the hardware communication loop. Maybe, we can discuss this in the next WG meeting.

Yes, something like what already exists in old kuka driver.