Activity is randomly restarted after successful completion

[lucascolonna]

I have an orchestrator that calls multiple activities, in one of those it calls and wait for responses of an API service.

On Sept 4th this orchestrator was instantiated at 2025-09-04T21:51:21.5321651z and completed at 2025-09-04T21:53:08.7450701Z without any failures or exceptions.
At 2025-09-04T21:58:05.5321651z a single activity inside this orchestrator was appears to have restarted , the issue was spotted by the downstream system.

There is no logs of the orchestration restarting and no entries of the second run on the HubHistory table, no other activity had this behavior.
Per below, Application Insights clearly shows that the activity in question was successfully completed during the first run.

[AnatoliB]

@lucascolonna Please note that the framework provides "at-least-once" guarantee for activity invocations. By design, the same activity may run two or more times occasionally. If you believe this is happening too often to your app, please consider creating an Azure support request

[lucascolonna]

@AnatoliB reading https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-types-features-overview#activity-functions I don't fully get "The Durable Task Framework guarantees that each called activity function will be executed at least once during an orchestration's execution."
In the scenario of this issue the orchestration was finished and still one activity apparently restarted.

I also imagine that the scenario happens when there is a underline instability on Azure, hence the suggestion to monitor? Similar to the described on https://learn.microsoft.com/en-us/answers/questions/1351212/azure-durable-function-at-least-once-execution-and

[AnatoliB]

@lucascolonna The "at-least-once" guarantee is documented at https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-types-features-overview#activity-functions. This typically happens when a worker executing an activity fails to mark it as completed in the storage, either because of intermittent storage availability issues, or because the worker got terminated for any reason. The underlying infrastructure issues like this are hard to escape in a large enough distributed system, and this is why we strongly recommend either making activities idempotent or have other ways to tolerate and compensate for occasional extra invocations. You can find more on this at https://blog.cgillum.tech/common-pitfalls-with-durable-execution-frameworks-like-durable-functions-or-temporal-eaf635d4a8bb.

I don't fully get "The Durable Task Framework guarantees that each called activity function will be executed at least once during an orchestration's execution."
In the scenario of this issue the orchestration was finished and still one activity apparently restarted.

I do not recommend overphasizing the "during an orchestration's execution" part here, as it will depend on the exact meaning of the "orchestration's execution" boundaries. For practical purposes, here is a simple way to think about it. When an orchestrator schedules an activity invocation, it posts a queue message with the acitivity name and inputs. A worker listening to this queue will eventually pick up this message, invoke the activity code, and, after the activity is complete, remove this message from the queue. However, there are many potential reasons to fail the last step: workers are not absolutely reliable, the storage is not absolutely reliable, and the communication channel is not absolutely reliable, so the message may reappear in the queue, and a worker listening to this queue will assume this activity needs to be executed again. It's hard to prevent this from happening completely.

Having said that, if you feel it's happening way too often for your app, it may deserve a more detailed investigation (for example, perhaps there is a systemic reason for your workers to crash), so please file an Azure support request in this case.

You may also want to read and track this: Azure/azure-functions-durable-extension#1555