[tunnelbroker] implement TokenDistributor
ClosedPublic
Actions

Authored by kamil on Aug 22 2025, 10:49 AM.

Details

Reviewers

bartek
tomek

Commits

rCOMM499efc54981d: [tunnelbroker] implement TokenDistributor

Summary

ENG-11118.

TOKEN_DISTRIBUTOR_SCAN_INTERVAL - interval at which we scan for orphaned tokens
TOKEN_DISTRIBUTOR_HEARTBEAT_INTERVAL - interval at which the instance updates its heartbeat to mark that the token is claimed and handled
TOKEN_DISTRIBUTOR_HEARTBEAT_TIMEOUT - after this time, we consider the instance to be dead, and the token can be reclaimed
TOKEN_DISTRIBUTOR_MAX_CONNECTIONS - limit of connections per instance
TOKEN_DISTRIBUTOR_PING_TIMEOUT - Farcaser is sending pings. After this time, we considerthe connection to be dead, and it should be reconnected

Depends on D15229

Test Plan

Tested later in the stack

Diff Detail

Repository

rCOMM Comm

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

kamil created this revision.Aug 22 2025, 10:49 AM

kamil held this revision as a draft.

Herald added a subscriber: ashoat. · View Herald TranscriptAug 22 2025, 10:49 AM

Harbormaster failed remote builds in B35516: Diff 49981!Aug 22 2025, 10:51 AM

kamil edited the summary of this revision. (Show Details)Aug 22 2025, 10:55 AM

kamil added inline comments.Aug 22 2025, 10:57 AM

services/tunnelbroker/src/config.rs
61–80 ↗	(On Diff #49981)	Curious how to set values here to make it optimal (cc. @bartek, @tomek)
services/tunnelbroker/src/token_distributor/mod.rs
62–65 ↗	(On Diff #49981)	docs
151 ↗	(On Diff #49981)	implemented in the next diff in the stack
198 ↗	(On Diff #49981)	implemented later in the stack

kamil added a child revision: D15231: [tunnelbroker] implement TokenConnection.Aug 22 2025, 10:59 AM

kamil published this revision for review.Aug 22 2025, 11:13 AM

Nice work on this! Looks neat, good job on handling async tasks and graceful shutdown.

services/tunnelbroker/Cargo.toml
41 ↗	(On Diff #49981)	Nit: we could add this to root `Cargo.toml` and use `workspace = true` here
services/tunnelbroker/src/config.rs
61–64 ↗	(On Diff #49981)	This one is good IMO
66–68 ↗	(On Diff #49981)	I would lower this value to 30/60 seconds - Tunnelbroker doesn't die too often
69–72 ↗	(On Diff #49981)	As a rule of thumb, I'd say timeout = 3*heartbeat_interval + small margin
73–76 ↗	(On Diff #49981)	Not sure if 200 ws connections per instance isn't too much. Hard to say, the only way to check it is stress testing and observing resource usage. It also depends on AWS instance size
77–80 ↗	(On Diff #49981)	How often does Farcaster send pings? Perhaps we could also use 3x interval + margin?