Differential D13607

[tunnelbroker] Trigger AMQP reconnect when closing session
Needs ReviewPublic
Actions

Authored by bartek on Thu, Oct 3, 1:37 PM.

Tags

None

Referenced Files

	F2889899: D13607.id.diff
	Fri, Oct 4, 7:34 AM

	F2888816: D13607.id44895.diff
	Fri, Oct 4, 5:47 AM

	F2888815: D13607.id44887.diff
	Fri, Oct 4, 5:47 AM

	F2888784: D13607.id.diff
	Fri, Oct 4, 5:46 AM

	F2888773: D13607.diff
	Fri, Oct 4, 5:46 AM

	F2887560: D13607.id.diff
	Fri, Oct 4, 3:09 AM

	F2887192: D13607.id44887.diff
	Fri, Oct 4, 1:51 AM

	F2885880: D13607.diff
	Thu, Oct 3, 8:50 PM

Subscribers

Details

Reviewers

kamil
varun
will

Summary

Most of errors were happening here. WS session was closed so consumer cancelation and queue removal calls weren't able to succeed.

Depends on D13606

Test Plan

Observing logs on staging. Possible but hard to reproduce with commtest
Can also be reproduced by:

Connecting a device to healthy Tunnelbroker
Don't make any traffic
Disconnect RabbitMQ
Then disconnect device

Diff Detail

Repository

Lint

No Lint Coverage

Unit

No Test Coverage

Event Timeline

bartek created this revision.Thu, Oct 3, 1:37 PM

bartek held this revision as a draft.

Herald added subscribers: tomek, ashoat. · View Herald TranscriptThu, Oct 3, 1:37 PM

bartek added a child revision: D13608: [tunnelbroker] Ignore connection error messages handled elsewhere.Thu, Oct 3, 1:38 PM

Harbormaster completed remote builds in B32058: Diff 44887.Thu, Oct 3, 2:17 PM

Rebase

bartek mentioned this in D13611: [tunnelbroker] Recover AMQP before publishing messages.Fri, Oct 4, 3:38 AM

bartek edited the summary of this revision. (Show Details)Fri, Oct 4, 4:04 AM

bartek edited the test plan for this revision. (Show Details)

Harbormaster failed remote builds in B32064: Diff 44895!Fri, Oct 4, 4:07 AM

Remove connection pool

bartek edited the summary of this revision. (Show Details)Fri, Oct 4, 8:40 AM

bartek published this revision for review.Fri, Oct 4, 8:44 AM

bartek added inline comments.

services/tunnelbroker/src/websockets/session.rs
743–747	According to RabbitMQ docs, if the channel is closed, all its consumers are also closed, so instead of recovering here, we can simply ensure connection is okay and exit early
757–762	This is to avoid spamming error alerts while connection errors are handled by our code elsewhere
766–771	We can probably consider having a separate task for cleaning up queues in case they're not able to be properly deleted here

Harbormaster completed remote builds in B32072: Diff 44904.Fri, Oct 4, 8:55 AM

Revision Contents
Changeset List

Path

Size

services/

tunnelbroker/

src/

5 lines

websockets/

34 lines

Diff 44904

services/tunnelbroker/src/amqp.rs

Loading...

services/tunnelbroker/src/websockets/session.rs

Loading...