top of page

Custom State Management in Hangfire




Most enterprise applications require asynchronous, scheduled, or recurring tasks such as batch imports from a file, mass mails or notifications, deleting temporary files, and so on. Hangfire is an open-source library that easily creates and manages background jobs for .NET and .NET Core frameworks.

Hangfire is mainly to perform tasks we do not want to handle during the request pipeline and scheduled or recurring jobs. Many features of Hangfire, including scaling, can be used free of charge, even for commercial uses.

Hangfire supports several background job types such as;

  • fire-and-forget (executed only once, almost immediately),

  • delayed (executed only once after a specific time),

  • recurring (executed many times according to CRON schedule) jobs, and so on.

It uses several persistent storages to ensure that a job is done at least once, such as MSSqlServer, Redis, PostgreSQL, MongoDB. In addition, it has an automatic retry mechanism to deal with failed jobs and built-in monitoring UI.

Hangfire also features a state machine for background jobs. Each background job has a state, and there are transitions between states, where some states are final. The background job is completed when the state is final. Let us look at the state diagram below for a recurring job:

Recurring job state diagram (succeeded and deleted are final states)


In addition to predefined states, custom states may be needed in some cases. For example, a scheduler triggers three different services. These services are used to complete the actual process. After jobs are completed, services will send an acknowledgment to Hangfire to maintain their state. Therefore, Hangfire needs an intermediate state between processing and a succeeded (or failed) state. Let be called WaitingAck. The diagram below shows the basic flow. In our team, we implement such a solution to use Hangfire in our multi-tenant application.


Let us look at the new state diagram with the WaitingAck state. Firstly, a background job gets Succeeded state when it triggered to application service. Right after, it moved the WaitingAck state. The application service sends an acknowledgment to the Hangfire to notify the job is completed successfully or failed. In this article, it is explained how the WaitingAck custom state can be implemented in the Hangfire.

Recurring job state diagram with WaitingAck intermediate state.


WaitingAckState, Handler and Filter

In order to define a custom state to extend the processing pipeline, IState interface must be implemented. WaitingAck state is not a final state because the job is not yet completed at this state. If we want to keep a counter for the state, we should implement a state handler.

public class WaitingAckState : IState
{
    public string Name => StateName;
    public string Reason => "Waiting for acknowledgement from an 
                                            external service.";
    public bool IsFinal => false;
    public bool IgnoreJobLoadException=>true;
    
    public static readonly stringStateName="WaitingAck";
    
    public Dictionary<string, string> SerializeData() => new 
                                    Dictionary<string, string>();
    
    public class Handler : IStateHandler    
    {
        public const string STATE_STAT_KEY="stats:waitingack";
        public void Apply(ApplyStateContextcontext, 
                        IWriteOnlyTransactiontransaction)        
        {
            transaction.IncrementCounter(STATE_STAT_KEY);        
        }
        public void Unapply(ApplyStateContextcontext, 
                    IWriteOnlyTransactiontransaction)        
        {
            transaction.DecrementCounter(STATE_STAT_KEY);        
        }
        
        public string StateName => WaitingAckState.StateName;    
    }
}

IElectFilter interface is an interceptor that enables us to change the candidate set to our custom state. The code below changes the next state to the WaitingAck state if certain conditions are met. The first part of conditions is the job is moving from processing state to succeeded state. The second condition checks whether this job is a WaitingAck job or not. WaitingAckJobBase sets the parameter.

public class WaitingAckStateFilter : IElectStateFilter
{
    public const string JOB_PARAMETER="waitingAck";
    
    public void OnStateElection(ElectStateContextcontext)    
    {
        if (context.CurrentState == ProcessingState.StateName
            && context.CandidateState is SucceededState
            && context.GetJobParameter<bool>(PARAMETER_JOB))        
        {context.CandidateState = new WaitingAckState();        
        }    
    }
}

WaitingAck state handler and filter must be added to the global configuration of Hangfire, as shown below. They can be called in ConfigureServices method at Startup.cs

public void ConfigureServices(IServiceCollectionservices)
{
    //...some code
    GlobalConfiguration.Configuration.UseFilter(new     
                                    WaitingAckStateFilter());
    GlobalStateHandlers.Handlers.Add(new WaitingAckState.Handler());
    //...some code
 }

Base Class for WaitingAck Jobs

In order to prevent setting the same parameter every time, we have developed an abstract class for jobs that use the WaitingAck state.

public abstract class WaitingAckJobBase
{
    public TaskExecute(PerformContextcontext, object[] args)    
    {
        context.SetJobParameter(WaitingAckStateFilter.PARAMETER_JOB, 
        true);
        PerformJob(context, args);
        return Task.CompletedTask;    
    }
    
    protected abstract TaskPerformJob(PerformContextcontext, object[] args);
}

Job Client for Changing State from WaitingAck

As described above, the WaitingAck state is not a final state. Therefore, we developed a job client to change the job state in the WaitingAck state to a final state such as Succeeded or Failed.

public class WaitingAckJobClient
{
    public static void MarkAsSucceeded(string jobId, 
                                        object result, 
                                        long latency, 
                                        long performanceDuration)    
    {
        new BackgroundJobClient().ChangeState(jobId, 
        new SucceededState(result, latency, performanceDuration));    
    }
    
    public static void MarkAsFailed(string jobId, Exceptionexception)    
    {
        new BackgroundJobClient().ChangeState(jobId, 
        new FailedState(exception));    
    }
    
    public static void MarkAsDeleted(string jobId)    
    {
        new BackgroundJobClient().Delete(jobId);    
    }
}

Custom State in Dashboard

The image below shows how the WaitingAck state looks on the job details page.

Example of hangfire dashboard job detail page in WaitingAck state.


Then, when the state changes to Succeeded, the image will be as follows.

Example of hangfire dashboard job detail page in Succeeded state.


A custom page can be added to the hangfire dashboard to display the jobs in the WaitingAck state as a list via Hangfire.Dashboard.RazorPage abstract class. The source code of the page is accessible at the link below. https://github.com/bordatech/hangfire-customstate/blob/main/src/Hangfire.WaitingAckState/WaitingAckJobsPage.cs

Let’s see how it looks:


Id and Job columns link to the job detail page, and jobs can be deleted with the Delete action button. The following extension method can be used to register this page to the Hangfire dashboard.

public static void UseHangfireWaitingAckPage(
    this IApplicationBuilderapp,
    string connectionString,
    string schema = "hangfire")
{
    DashboardRoutes.Routes.AddRazorPage("/waitingack", page => new 
                    WaitingAckJobsPage(connectionString, schema));
    NavigationMenu.Items.Add(menu => new MenuItem("WaitingAck Jobs", 
                                        menu.Url.To("/waitingack")));
    DashboardRoutes.Routes.AddCommand("/waitingack/(?<JobId>.+)/delete",
        context =>        
        {
            WaitingAckJobClient.MarkAsDeleted(context.UriMatch.Groups
                                                    ["JobId"].Value);
            return true;        
        });
}

The repository methods written for the page are accessible from the link below. https://github.com/bordatech/hangfire-customstate/blob/main/src/Hangfire.WaitingAckState/PostgreSqlRepository.cs

Conclusion

This article shows how to implement a custom state to the state machine in Hangfire with a use case. In addition, a custom page is developed for the new state in the dashboard.



Source: Medium - Emre Teoman


The Tech Platform

0 comments
bottom of page