Consider key-based channels

[gavin-norman-sociomantic]

Many applications are structured as follows:

1. Consume a DMQ channel.
2. For each record, make a DHT request (using some field of the DMQ record as the DHT key).

It's currently very difficult for such applications to implement support for partial DHT handshaking, as the app doesn't know how to handle any record it receives from the DMQ that would trigger a request to a node in the DHT whose hash range is not yet known. The best thing such an app can do, right now, is to store "orphaned" records in a local queue (in memory / on disk) for processing once the hash range of the missing DHT node is known. This is just a workaround, though, effectively duplicating the behaviour of the DMQ locally in an application instance.

What if we added a feature to DMQ Consume to allow subscription to a subsection of a channel, based on a hash range? That way, an app could start consuming only those DMQ records destined for DHT nodes that have already been handshaked.

[joseph-wakeling-sociomantic]

So the idea here would be that the DMQ takes care of the backlog of messages that need still need pushing?

[gavin-norman-sociomantic]

Right. Which is exactly the intended function of a DMQ.

[joseph-wakeling-sociomantic]

I'm not entirely sure that one can obviate the need for app-side handling, because there will be a race condition between the app losing access to a DHT hash range and the app communicating that to the DMQ.

Given that constraint, this feature might reduce the volume of backlog that an app would have to take care of, but it would not allow the app to entirely outsource any backlog to the DMQ, no ... ?

[gavin-norman-sociomantic]

Yeah. This was actually proposed to fix a different problem, though. You're talking about the situation where a DHT node is accessible but then the connection is interrupted. I was talking about the situation where a DHT node is not accessible when an app starts up, but later becomes accessible.

In the legacy client, at least, we already have a solution (the request queue + overflow) for the problem you describe, whereas we don't have a solution for the other problem.

[joseph-wakeling-sociomantic]

I was talking about the situation where a DHT node is not accessible when an app starts up, but later becomes accessible.

I guess what is not entirely clear is the reason why (unavoidable) app-side overflow support is suboptimal as a solution for this use-case.

[gavin-norman-sociomantic]

See sociomantic-tsunami/dhtproto#77 (comment).

[joseph-wakeling-sociomantic]

Coming at this from a different angle: it's not clear to me what is unique about those requirements to the partial-handshake startup, as opposed to when one loses access to a DHT hash range at a later point.

I guess the argument could be made that with this feature in place, app-side overflow support for the latter case could be limited to the relatively small number of DMQ messages that make it through before the DMQ is notified about the hash-range outage ... ?

[gavin-norman-sociomantic]

I guess the argument could be made that with this feature in place, app-side overflow support for the latter case could be limited to the relatively small number of DMQ messages that make it through before the DMQ is notified about the hash-range outage ... ?

No, the idea being discussed in this issue is that the application would only start consuming a certain range of hashes from the DMQ when the handshake for the corresponding DHT node has succeeded. So if there was a hash-range outage, no records in that range would be consumed from the DMQ.

[nemanja-boric-sociomantic]

I think discussions here and at dhtproto should be merged into a single one.

[gavin-norman-sociomantic]

Yeah, this issue was intended more as an "action step" based on the discussion in the other issue ;)

[joseph-wakeling-sociomantic]

Agree to move discussion. Just want to respond on one point:

No, the idea being discussed in this issue is that the application would only start consuming a certain range of hashes from the DMQ when the handshake for the corresponding DHT node has succeeded. So if there was a hash-range outage, no records in that range would be consumed from the DMQ.

I understand the intention is to support the partial-handshake case and not the lose-hash-range-access-later case. What I don't understand is why the problems described for the former case do not also apply to the latter.

[ben-palmer-sociomantic]

One option to provide "automatic load balancing" ™️ would be for the DMQ node to split the hash range itself based on the number of subscribers to a channel (id % number of subscribers). This would mean that applications would reliably get the same range or records apart from brief recovery periods where a subscriber is added/removed.

[gavin-norman-sociomantic]

Yeah, that would certainly be useful for some apps, at least. Probably the option of either (flexible or static hash ranges) might be nice.