Failed Jobs

Enqueued 0 / 1 Scheduled 0 Processing 41 Succeeded 4,136,746 Failed 720 Deleted 711 Awaiting 0

Failed jobs do not become expired to allow you to re-queue them without any time pressure. You should re-queue or delete them manually, or apply AutomaticRetry(OnAttemptsExceeded = AttemptsExceededAction.Delete) attribute to delete them automatically.

Id	Job	Failed
#212975	RetrieveOrdersTask.ProcessRecurringAsync An exception occurred during performance of the job. Fewer details...	07/20/2025 11:40:15
#212975	System.InvalidOperationException A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913. System.InvalidOperationException: A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913. at Microsoft.EntityFrameworkCore.Infrastructure.Internal.ConcurrencyDetector.EnterCriticalSection() at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at EkolPY.Integration.Service.IntegrationTaskService.AddTaskLog(TaskLogLevel logLevel, String[] messages) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 154 at EkolPY.Integration.Service.IntegrationTaskService.AddTaskError(Exception ex, String[] msgs) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 144 at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(String message, Exception ex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 244 at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContext context, String queueName, TaskType taskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
#212966	RetrieveCancellationRequestsTask.ProcessRecurringAsync An exception occurred during performance of the job. More details...	07/20/2025 11:40:15
#212966	System.InvalidOperationException A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913. System.InvalidOperationException: A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913. at Microsoft.EntityFrameworkCore.Infrastructure.Internal.ConcurrencyDetector.EnterCriticalSection() at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at EkolPY.Integration.Service.IntegrationTaskService.AddTaskLog(TaskLogLevel logLevel, String[] messages) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 154 at EkolPY.Integration.Service.IntegrationTaskService.AddTaskError(Exception ex, String[] msgs) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 144 at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(String message, Exception ex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 244 at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContext context, String queueName, TaskType taskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
#175883	RetrieveCancellationRequestsTask.ProcessRecurringAsync An exception occurred during processing of a background job. More details...	07/18/2025 13:30:04
#175883	Npgsql.NpgsqlOperationInProgressException The connection is already in state 'Executing' Npgsql.NpgsqlOperationInProgressException (0x80004005): The connection is already in state 'Executing' at Npgsql.Internal.NpgsqlConnector.<StartUserAction>g__DoStartUserAction\|259_0(<>c__DisplayClass259_0& ) at Npgsql.Internal.NpgsqlConnector.StartUserAction(ConnectorState newState, NpgsqlCommand command, CancellationToken cancellationToken, Boolean attemptPgCancellation) at Npgsql.Internal.NpgsqlConnector.Reset(Boolean async) at Npgsql.NpgsqlConnection.CloseAsync(Boolean async) at Npgsql.NpgsqlConnection.Close() at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Close() at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.ClearTransaction() at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.Dispose() at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.ResetState(Boolean disposeDbConnection) at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.Dispose() at Microsoft.EntityFrameworkCore.DbContext.Dispose() at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.Dispose() at Hangfire.JobActivatorScope.Dispose() at Hangfire.Server.CoreBackgroundJobPerformer.Perform(PerformContext context) at Hangfire.Server.BackgroundJobPerformer.<>c__DisplayClass9_0.<PerformJobWithFilters>b__0() at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation) at Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(PerformContext context, IEnumerable`1 filters) at Hangfire.Server.BackgroundJobPerformer.Perform(PerformContext context) at Hangfire.Server.Worker.PerformJob(BackgroundProcessContext context, IStorageConnection connection, String jobId)
#175879	RetrieveOrdersTask.ProcessRecurringAsync An exception occurred during processing of a background job. More details...	07/18/2025 13:29:11
#175879	Npgsql.NpgsqlOperationInProgressException The connection is already in state 'Executing' Npgsql.NpgsqlOperationInProgressException (0x80004005): The connection is already in state 'Executing' at Npgsql.Internal.NpgsqlConnector.<StartUserAction>g__DoStartUserAction\|259_0(<>c__DisplayClass259_0& ) at Npgsql.Internal.NpgsqlConnector.StartUserAction(ConnectorState newState, NpgsqlCommand command, CancellationToken cancellationToken, Boolean attemptPgCancellation) at Npgsql.Internal.NpgsqlConnector.Reset(Boolean async) at Npgsql.NpgsqlConnection.CloseAsync(Boolean async) at Npgsql.NpgsqlConnection.Close() at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Close() at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.ClearTransaction() at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.Dispose() at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.ResetState(Boolean disposeDbConnection) at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.Dispose() at Microsoft.EntityFrameworkCore.DbContext.Dispose() at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.Dispose() at Hangfire.JobActivatorScope.Dispose() at Hangfire.Server.CoreBackgroundJobPerformer.Perform(PerformContext context) at Hangfire.Server.BackgroundJobPerformer.<>c__DisplayClass9_0.<PerformJobWithFilters>b__0() at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation) at Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(PerformContext context, IEnumerable`1 filters) at Hangfire.Server.BackgroundJobPerformer.Perform(PerformContext context) at Hangfire.Server.Worker.PerformJob(BackgroundProcessContext context, IStorageConnection connection, String jobId)
#175842	RetrieveOrdersTask.ProcessRecurringAsync An exception occurred during processing of a background job. More details...	07/18/2025 13:26:03
#175842	Npgsql.NpgsqlOperationInProgressException The connection is already in state 'Executing' Npgsql.NpgsqlOperationInProgressException (0x80004005): The connection is already in state 'Executing' at Npgsql.Internal.NpgsqlConnector.<StartUserAction>g__DoStartUserAction\|259_0(<>c__DisplayClass259_0& ) at Npgsql.Internal.NpgsqlConnector.StartUserAction(ConnectorState newState, NpgsqlCommand command, CancellationToken cancellationToken, Boolean attemptPgCancellation) at Npgsql.Internal.NpgsqlConnector.Reset(Boolean async) at Npgsql.NpgsqlConnection.CloseAsync(Boolean async) at Npgsql.NpgsqlConnection.Close() at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Close() at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.ClearTransaction() at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.Dispose() at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.ResetState(Boolean disposeDbConnection) at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.Dispose() at Microsoft.EntityFrameworkCore.DbContext.Dispose() at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.Dispose() at Hangfire.JobActivatorScope.Dispose() at Hangfire.Server.CoreBackgroundJobPerformer.Perform(PerformContext context) at Hangfire.Server.BackgroundJobPerformer.<>c__DisplayClass9_0.<PerformJobWithFilters>b__0() at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation) at Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(PerformContext context, IEnumerable`1 filters) at Hangfire.Server.BackgroundJobPerformer.Perform(PerformContext context) at Hangfire.Server.Worker.PerformJob(BackgroundProcessContext context, IStorageConnection connection, String jobId)
#175830	RetrieveCancellationRequestsTask.ProcessRecurringAsync An exception occurred during performance of the job. More details...	07/18/2025 13:26:02
#175830	System.InvalidOperationException A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913. System.InvalidOperationException: A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913. at Microsoft.EntityFrameworkCore.Infrastructure.Internal.ConcurrencyDetector.EnterCriticalSection() at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at EkolPY.Integration.Service.IntegrationTaskService.AddTaskLog(TaskLogLevel logLevel, String[] messages) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 154 at EkolPY.Integration.Service.IntegrationTaskService.AddTaskError(Exception ex, String[] msgs) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 146 at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(String message, Exception ex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 246 at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContext context, String queueName, TaskType taskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
#172843	Can not find the target method. Can not change the state to 'Processing': target method was not found. More details...	07/18/2025 09:40:59
#172843	System.IO.FileNotFoundException Could not load file or assembly 'EkolPY.Integration.BackgroundTask, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified. System.IO.FileNotFoundException: Could not load file or assembly 'EkolPY.Integration.BackgroundTask, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified. File name: 'EkolPY.Integration.BackgroundTask, Culture=neutral, PublicKeyToken=null' at System.Reflection.RuntimeAssembly.InternalLoad(ObjectHandleOnStack assemblyName, ObjectHandleOnStack requestingAssembly, StackCrawlMarkHandle stackMark, Boolean throwOnFileNotFound, ObjectHandleOnStack assemblyLoadContext, ObjectHandleOnStack retAssembly) at System.Reflection.Assembly.Load(AssemblyName assemblyRef) at Hangfire.Common.TypeHelper.AssemblyResolver(String assemblyString) at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory) at System.TypeNameParser.ResolveAssembly(String asmName, Func`2 assemblyResolver, Boolean throwOnError, StackCrawlMark& stackMark) at System.TypeNameParser.ConstructType(Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark) at System.TypeNameParser.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark) at System.Type.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError) at Hangfire.Common.TypeHelper.DefaultTypeResolver(String typeName) at Hangfire.Storage.InvocationData.DeserializeJob()
#155498	SendOrderItemIdentityTasks.ProcessRecurringAsync An exception occurred during performance of the job. More details...	07/17/2025 12:08:44
#155498	System.InvalidOperationException A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913. System.InvalidOperationException: A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913. at Microsoft.EntityFrameworkCore.Infrastructure.Internal.ConcurrencyDetector.EnterCriticalSection() at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at EkolPY.Integration.Service.IntegrationTaskService.AddTaskLog(TaskLogLevel logLevel, String[] messages) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 154 at EkolPY.Integration.Service.IntegrationTaskService.AddTaskError(Exception ex, String[] msgs) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 146 at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(String message, Exception ex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 246 at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContext context, String queueName, TaskType taskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
#155120	RetrieveOrdersCreatedPickingTask.ProcessRecurringAsync An exception occurred during processing of a background job. More details...	07/17/2025 11:32:21
#155120	Npgsql.NpgsqlOperationInProgressException The connection is already in state 'Executing' Npgsql.NpgsqlOperationInProgressException (0x80004005): The connection is already in state 'Executing' at Npgsql.Internal.NpgsqlConnector.<StartUserAction>g__DoStartUserAction\|259_0(<>c__DisplayClass259_0& ) at Npgsql.Internal.NpgsqlConnector.StartUserAction(ConnectorState newState, NpgsqlCommand command, CancellationToken cancellationToken, Boolean attemptPgCancellation) at Npgsql.Internal.NpgsqlConnector.Reset(Boolean async) at Npgsql.NpgsqlConnection.CloseAsync(Boolean async) at Npgsql.NpgsqlConnection.Close() at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Close() at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.ClearTransaction() at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.Dispose() at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.ResetState(Boolean disposeDbConnection) at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.Dispose() at Microsoft.EntityFrameworkCore.DbContext.Dispose() at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.Dispose() at Hangfire.JobActivatorScope.Dispose() at Hangfire.Server.CoreBackgroundJobPerformer.Perform(PerformContext context) at Hangfire.Server.BackgroundJobPerformer.<>c__DisplayClass9_0.<PerformJobWithFilters>b__0() at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation) at Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(PerformContext context, IEnumerable`1 filters) at Hangfire.Server.BackgroundJobPerformer.Perform(PerformContext context) at Hangfire.Server.Worker.PerformJob(BackgroundProcessContext context, IStorageConnection connection, String jobId)
#155083	SendOrderItemIdentityTasks.ProcessRecurringAsync An exception occurred during processing of a background job. More details...	07/17/2025 11:30:02
#155083	Npgsql.NpgsqlOperationInProgressException The connection is already in state 'Executing' Npgsql.NpgsqlOperationInProgressException (0x80004005): The connection is already in state 'Executing' at Npgsql.Internal.NpgsqlConnector.<StartUserAction>g__DoStartUserAction\|259_0(<>c__DisplayClass259_0& ) at Npgsql.Internal.NpgsqlConnector.StartUserAction(ConnectorState newState, NpgsqlCommand command, CancellationToken cancellationToken, Boolean attemptPgCancellation) at Npgsql.Internal.NpgsqlConnector.Reset(Boolean async) at Npgsql.NpgsqlConnection.CloseAsync(Boolean async) at Npgsql.NpgsqlConnection.Close() at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Close() at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.ClearTransaction() at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.Dispose() at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.ResetState(Boolean disposeDbConnection) at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.Dispose() at Microsoft.EntityFrameworkCore.DbContext.Dispose() at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.Dispose() at Hangfire.JobActivatorScope.Dispose() at Hangfire.Server.CoreBackgroundJobPerformer.Perform(PerformContext context) at Hangfire.Server.BackgroundJobPerformer.<>c__DisplayClass9_0.<PerformJobWithFilters>b__0() at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation) at Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(PerformContext context, IEnumerable`1 filters) at Hangfire.Server.BackgroundJobPerformer.Perform(PerformContext context) at Hangfire.Server.Worker.PerformJob(BackgroundProcessContext context, IStorageConnection connection, String jobId)

System.InvalidOperationException

System.InvalidOperationException

Npgsql.NpgsqlOperationInProgressException

Npgsql.NpgsqlOperationInProgressException

Npgsql.NpgsqlOperationInProgressException

System.InvalidOperationException

System.IO.FileNotFoundException

System.InvalidOperationException

Npgsql.NpgsqlOperationInProgressException

Npgsql.NpgsqlOperationInProgressException