Cluster Singleton Grain Service?

[AlgorithmsAreCool]

So, I have a weird use case that i don't know if the in-box Orleans tools are a good fit for. Basically, i need a cluster-singleton persistent service. A Grain Service works just fine for this, except that it is replicated to all Silos (maybe they should be called Silo Services 😄).

I have taken to implementing a grain service, but with some fancy logic that basically elects a "leader" that directs the other follower instances. The "election" is just whatever Silo Address is sorted first in the list of active silos, which updates made on every ring size change. It kinda seems to work, but i wonder if there is a better way to tackle this?

The obvious alternative is to spawn an external "gateway" service outside of orleans that relays commands into the cluster, which works, but I was just curious if there was a way to handle this in an orleans native way.

The concrete usecase is a pub/sub subscription that needs to maintain a single persistent connection to the broker to send/receive commands.

[haondt]

Just spitballing, would a grain with a known, fixed ID (e.g. Guid.Empty) and a Reminder to keep it alive work for your purposes? Or you could cut out the middleman and keep it alive with pings from a GrainService or IHostedService.

[AlgorithmsAreCool]

I suppose it could? As long as the reminders are fully reliable (which they should be...).

It does feel a little like shoehorning the concept of a "forever" service into an "ephemeral" grain lifetime that we just keep extending, but I think your idea could work.

[haondt]

True, tbh I think GrainServices with leader election is probably the most elegant you're going to get (would be happy to be proven wrong though). As far as I can tell Orleans is built to be reactive so I am not sure you'll find any sort of definitive solution.

That said, I wouldn't think of it as extending an ephemeral lifetime, more like renewing a subscription, since the upside is if the silo dies the reminder reactivates it on a different silo.

[AlgorithmsAreCool]

Yeah i like the concept of using the grain service because not only does it allow for a leader, but also code to execute on follower nodes if you wanted since the service executes on all peers.

I have hooked the ring update notification to reevaluate who is the leader. The downside is i think there might be some racing possible depending on what node executes the change first or something like that

Basically, whenever a node receives a notification that the ring has changed, it checks the new definition to see if it now owns the magic point on the ring (defined as position 0), if so then that node acknowledges itself as the leader and sends a "stand-down" command to all it's peers, once acknowledged, it takes over the leader responsibilities.

There is probably a little wiggle room if the cluster is thrashing that multiple nodes could step on each other. This can probably be defeated by including the version of the ring issuing the commands and saying a node can only accept a command from a equal or higher version of the ring or something like that, maybe idk...

[haondt]

What about just using a grain for the election? Whenever a grain service receives the notification it must acquire ownership of the leader-lease-grain.

I'm not sure what the ring update notifications look like, but if you know what the current set of silos is, you can include the owners silo address in the lease, and invalidate the lease if that silo is no longer part of the ring. Whoever can acquire the lease first becomes the new leader.

Alternatively, you could just expire the lease after some time and the leader would have to continually renew the lease to maintain ownership. On shutdown the leader explicitly releases ownership of the grain.

At that point though the reminder grain solution would be essentially the same thing without using the ring update notifications.

[AlgorithmsAreCool]

I kind of like this idea of the leader needing to renew it's position. Ideally incase the leader experiences a fault someone else would take over...