Leader Election & Daemons

When you run several instances of your application, some background work must run on one instance only. Emit handles this with leader election, which picks one node in charge, and daemons, which run your work on exactly one node at a time.

Both need a persistence provider: coordination goes through its database, so there is no extra infrastructure to run.

Services sharing a database

Coordination is scoped to a service: all replicas of the same application. An API and a worker sharing one database each get their own leader and run their own daemons.

The service name defaults to the application name. Set ServiceName when two applications sharing a database have the same application name, and keep it stable across deployments.

Leader election

Every node sends a heartbeat to the database (every 15 seconds by default), and one node per service holds the leadership. If the leader stops, another node takes over once its lease runs out; a clean shutdown hands leadership over at once.

Each node has a stable identity, exposed through INodeIdentity. Register your own implementation before AddEmit() to tie it to something meaningful, such as a pod name.

Querying leadership status

Inject ILeaderElectionService to check the current node’s role:

public class MyService(ILeaderElectionService leaderElection)
{
    public void DoWork()
    {
        if (leaderElection.IsLeader)
        {
            // Only the leader reaches this branch
        }

        // Cancelled when the node loses leadership.
        // Pass it to long-running leader-only operations.
        var token = leaderElection.LeadershipToken;
    }
}

Observing leadership changes

Register an ILeaderElectionObserver with emit.AddLeaderElectionObserver<T>() to react when this node is elected or loses leadership, or when nodes join or leave the service.

Daemons

A daemon is background work that runs continuously on exactly one node of the service, and moves to another node when that node goes away: a scheduler, a poller, a reconciliation job. The outbox runs as a daemon too (emit:outbox).

Writing a daemon

Implement IDaemonAgent and register it with AddDaemon:

public sealed class InvoiceReminderDaemon(IServiceScopeFactory scopes) : IDaemonAgent
{
    public string DaemonId => "invoice-reminders";

    public async Task RunAsync(DaemonRunContext context)
    {
        using var timer = new PeriodicTimer(TimeSpan.FromMinutes(1));
        while (await timer.WaitForNextTickAsync(context.AssignmentToken))
        {
            await using var scope = scopes.CreateAsyncScope();
            var reminders = scope.ServiceProvider.GetRequiredService<IReminderService>();

            foreach (var invoice in await reminders.GetDueAsync(context.AssignmentToken))
            {
                // Stop as soon as this node no longer holds the daemon
                if (!context.HoldsAssignment)
                {
                    return;
                }

                await reminders.SendAsync(invoice, context.AssignmentToken);
            }
        }
    }
}

builder.Services.AddEmit(emit =>
{
    // ... a persistence provider
    emit.AddDaemon<InvoiceReminderDaemon>();
});

A few rules:

• Run until AssignmentToken is cancelled, then stop promptly (within DrainTimeout). A run that ends or throws on its own counts as a failure and is restarted; one that keeps failing moves to another node.
• Check HoldsAssignment right before each side effect. It turns false the moment the node must stop.
• Create a scope per unit of work for scoped services such as a DbContext: the daemon itself lives as long as the application.
• Keep DaemonId stable and register the same daemons on every replica. Identifiers starting with emit: are reserved. One class can back several daemons with AddDaemon(sp => new ReportDaemon("daily-report")).

If your daemon writes somewhere that must never see two runs overlap, even after a frozen process wakes up, pass context.FencingKey and context.Generation along with the write and have that system refuse a generation lower than the last one it accepted. Each new assignment of a daemon gets a higher generation.

To unit test a daemon, call RunAsync with a context you build yourself: new DaemonRunContext(token, generation: 1, fencingKey: Guid.NewGuid(), holdsAssignment: () => true).

When nodes come and go

• A node joins: daemons spread out, one at a time, so no node runs two more than another.
• A node shuts down: it stops its daemons and hands them over right away.
• A node crashes or loses the database: it stops its own daemons, and they move once its registration expires (90 seconds by default).

Use an IDaemonObserver or the daemon metrics to see where daemons run.

Daemon, distributed lock or IsLeader?

You need	Use
Work that runs continuously on exactly one node, and moves when that node goes away	A daemon
A short critical section that any node may run, one at a time	A distributed lock
A leader-only branch in code that already runs on every node	ILeaderElectionService.IsLeader and LeadershipToken

Configuration

The defaults suit most deployments. Give every node of a service the same values.

Option	Default	Notes
ServiceName	Application name	Shared by all replicas of a service
HeartbeatInterval	15 seconds	How often each node sends a heartbeat
LeaseDuration	60 seconds	How long leadership lasts without a heartbeat before another node takes over
NodeRegistrationTtl	90 seconds	How long a silent node keeps its daemons before they move
DrainTimeout	8 seconds	How long a daemon may take to stop. Keep the host’s shutdown timeout comfortably longer (its default of 30 seconds is fine)
MaxConsecutiveFailures	3	Failures in a row before a daemon moves to another node

See the Configuration Reference for every option and the rules between them.