We should be able to run multiple instances on the same background service on multiple servers. At the same time, only one instance should be run in "Active" mode, which means do work. Others should run in "Passive" mode, waiting until the "Active" one goes down, and then automatically switch from "Passive" to "Active".
As you can see in the image above, we will implement a push-oriented service discovery functionality to fulfil requirements. Each instance will register itself at the start in the external data source (can be whatever Azure Database/SQL Server/No-SQL). All instances will contain a logic that will tell them which mode they should work. In case of an unexpected server failure, we will implement the logic of auto un-registration of instance.
Each Background Service has to be able to manipulate data.
Each Background Service has to be able to work in "Passive" mode.
Each Background Service has to be able to work in "Active" mode.
Each Background Service will register itself at the start.
Each Background Service will re-register itself in X seconds.