[Tunnelbroker] publish messages from DDB to RabbitMQ when client connects
Summary:
This code enqueues messages in RabbitMQ, instead of directly flushing to the client.
Why we need this:
- In the next diffs I'll introduce code responsible for confirming message, the message is sent here, but confirmation is received here - handling third place, where we also need to confirm messages (directly from DDB) could complicate the code.
- This allows us to maintain the proper order of messages - messages from DDB that have higher priority are delivered first.
- This gracefully handles the case where different brokers send messages to the device while sending persisted messages, in the previous version of the code that case could change messages order.
Depends on D9570
Test Plan:
- Run tests Tunnelbroker tests (most important are tunnelbroker_persist_tests and test_messages_order )
- Make sure priority works, do the following:
- declare queue
- send messages A, and B with priority 10
- send message C with priority 1
- send message D with priority 10
- create consumer
- consume messages - the order should be A, B, D, C
Reviewers: michal, bartek, varun, jon
Reviewed By: bartek
Subscribers: ashoat, tomek, wyilio
Differential Revision: https://phab.comm.dev/D9571