Address ENG-10523.
Previously, there was raw JoinHandle which, when stopped, was just aborting the task and forgetting.
Replaced this with more structured approach, that uses CancellationToken to gracefully quit the task, and makes sure that previous task exited before starting a new one.
The token gives us control over the places where the task can gracefully exit. Here, these are the two tokio::select! calls.
Introduced two structs:
- BackupHandlerTask which represents a single running task. Also introduced a task_id variable for better debugging/monitoring experience.
- BackupHandler which is a static singleton, and is responsible for management of BackupHandlerTask instances.
Slightly improved logging as a part of the monitoring task, used task_id.