Missed triggers in Azure Logic App (Consumption)

Recently, I had to develop a solution for a client to sync data from CRM Dataverse to Salesforce, with the added requirement that only a single instance of the sync process run for a particular entity. The easiest way to achieve this is to force the Logic App workflow that is triggered on the add/update of the entity to run in a singleton mode, i.e., set concurrency to 1.

You can find the above option on the settings for most Azure Logic App trigger.

This design should suffice in most cases.

However, we started observing a lot of update triggers are being missed. We could not find any logical explanation as there was no error logged anywhere. We also raised a ticket with both the MS Logic App Team and the PowerApps (Dataverse) team but could not find anything meaningful to help with our situation. At this point, we knew there were missed webhook calls to the Logic App trigger endpoint but we did not know the reason behind it.

In order to better understand the situation and have some visibilty into the error logs, I tried recreating the scenario but with an HTTP trigger. For this, I created a Parent Logic App workflow that calls a Child workflow with HTTP trigger set to run in a singleton mode (Concurrency Control set to 1). I also added a delay ( 1 min) in the child workflow to mimic some processing logic.

Parent Logic App

Child Logic App

The trigger for Parent Logic app workflow is a Scheduled trigger (could be HTTP or something else). I ran the Parent workflow multiple times ( > 12) and started to see triggers missed, i.e., the Parent workflow execution was not resulting in the child workflow execution. And the best part, now as we are the caller of the webhook trigger, we can now see the error:

Parent Logic App Failed Runs

Child Logic App waiting instances

429 error for child workflow webhook calls

As you can see in the image above, we see 429s as soon as the calls exceed a certain number.

{
  "error": {
    "code": "WorkflowMaximumWaitingRunCountExceeded",
    "message": "The number of waiting workflow runs '11' has exceeded the maximum allowed limit of '11'. Try increasing the value of 'maximumWaitingRuns' in trigger concurrency configuration."
  }
}

The error message here suggests that there is a limit to the number of runs that can wait in the queue for the singleton child workflow. After digging a bit, it seems there is an obscure setting “maximumWaitingRuns” which you do not see while setting the Concurrency on the portal. You can only set this property in the code view.

If you do not set this property explicitly in the code view, it will use a default of 10.

Property for setting MaximumWaitingRuns

Once you set maximumWaitingRuns to a higher value, you should see all the calls to the webhook result in succesful calls to the child workflow waiting in the queue to be processed.

Unfortunately, the maximum value you can set for this property is 200. So, if your source trigger is expected to have upserts in excess of 200 within a short span of time, it would be better to design a queue based solution brokered by Service bus or Storage queue ( Service bus with session enabled if order of messages is important).