Add remote actors to the name registry

[calebfletcher]

This PR registers remote actors into the node's local registry, and adds an integration test to demonstrate that this works.

[slawlor]

So I actually thought about this when writing all the remote actor logic (I should probably have written it down somewhere when I thought about it lol).

What are your thoughts on if there's already an actor with the same name registered?

I.e.
Node A - has actor "test" running
Node B - has actor "test" running

Node A then joins Node B, which trying to re-register the same name will fail. What are your thoughts on that?

[slawlor]

Right now, I think it'd fail to spawn the relevant RemoteActor instance, and might just swallow the error?

[calebfletcher]

So I actually thought about this when writing all the remote actor logic (I should probably have written it down somewhere when I thought about it lol).

What are your thoughts on if there's already an actor with the same name registered?

I.e. Node A - has actor "test" running Node B - has actor "test" running

Node A then joins Node B, which trying to re-register the same name will fail. What are your thoughts on that?

I've spent some time thinking about this, and I think there are a few different options going forward:

1. Register names with a first come first served strategy, ignoring subsequent requests to register the names. This is the closest aligned with the current behaviour, but has some sharp edges when actually trying to use it. It would make the whole name registry have a race condition, since if you connect to 2 different nodes which each have actors with the same name, then it would depend on which one completes authentication first which is not deterministic.
2. Allow more than one actor to be registered with a particular name (and change the where_is function to return a Vec). This would solve the issue and prevent conflicts, but would effectively turn the registry into a way to automatically register actors into process groups.
3. Split the registry based on which node the actors come from, changing the API to be something like where_is_by_node(name; ActorName, node: NodeId). Since ractor itself doesn't seem to know about the nodes in ractor_cluster, this would probably have to go into ractor_cluster. This would fundamentally change the philosophy of ractor/ractor_cluster (that it is completely transparent for most purposes whether an actor is on your own node vs a remotely connected node).

My personal opinion is that option 2 is the most flexible and reliable, but the significant overlap with process groups may mean that it may not be worth implementing.

[slawlor]

My personal opinion is that option 2 is the most flexible and reliable, but the significant overlap with process groups may mean that it may not be worth implementing.

This was essentially where I settled when I first wrote the functionality. However I believe Erlang does indeed support globally addressable actors, i.e. global in a full cluster like this. So it should theoretically be possible. Let me look into how they did it, and see if we can replicate that kind of functionality.

[NilsDeckert]

Hey, has there been any progress on this topic? I am currently trying to achieve the following:

Actor A casts a message and a reference of Actor B to the remote Actor C.

Actor C handles the message and sends a result to Actor B.

My specific use-case is the following:
A and B are reading from / writing to a TCP stream respectively.
I want A to continue reading from the stream without waiting for the result.

Do you have any suggestions on what would be the most idiomatic approach here?

[slawlor]

Do you have any suggestions on what would be the most idiomatic approach here?

So the problem you're going to have here is ActorId translation. An ActorRef isn't serializable over the channel, mostly because of the internal data stored within are like memory-local async Tokio channels. Additionally the PID locally is different from the PID remotely, as the remote PID will have a non-zero "node id" which defines the active node session.

You could send an ActorId over the wire, with just the local pid, and when you receive it extract the node which it was sent from. That would let you construct the address of the actor B in your example. For example consider Servers A & B.

Server A is connected to B, and A views B as it's "node 1".
Server B is connected to A, and B views A as it's "node 1".

If A.actor_1 sends a message to B.actor_2 with the address of A.actor_3, actor_3 is local to 0 and therefore will have a node id of 0. You want to send 1.<actor_3_pid> to B, so that when B looks it up in its PID registry, it can find the actor_3 which is on its "node 1" (A).

I'm not sure that's super clear, and cross-node addressing is actually quite painful, so it's not a trivial problem unfortunately.

[cerceris]

Basically, all you need to communicate between actors is a task/actor UUID/name. A node has a router that listens for incoming messages. A message is in a common "se/deserializable" format. An incoming message contains the destination task/actor UUID. The router passes a raw message to the (local) distributor/dispatcher. The distributor/dispatcher sends the message to the (local) target actor. The message also contains the sender UUID. The question is how would you like to implement the transport. zmq works pretty well for this kind of tasks. How the distributed actors know the hosts of each other actors? I guess the Erlang paradigm offers many solutions :) Preconfigured (if you have "small enough" amount of nodes), the central "actor discovery" facility, other types of brokers. I personally prefer direct connections where it's possible. But brokers could serve as node info distribution instruments.