Last Thursday, we started having connectivity issues with our data repositories hosted in Microsoft Fabric. We experienced multiple timeouts, were no longer able to connect to the SQL databases, and queries to the Warehouse and Lakehouse also failed. After some initial analysis, we discovered the following error in our logs and IDEs:

Retrieval of MWC token used for accessing storage failed with error ‘0xa (MWC service error: Server responded with error: 400)(DmsPbiServiceUserException: An internal system error has occurred.)’ (7451)

Opening the Warehouse and running a simple SELECT-query resulted in the same error:

We found this quite strange, as there had not been any deployments or changes in either Microsoft Fabric or our Microsoft Azure solution in the past couple of days.

The error was not very obvious to me either. I figured there must have been something wrong with the Workspace, maybe due to suspending and resuming the Capacity on a daily basis. It could be that something was stuck in an erroneous state, or maybe the capacity had reached its limits and throttling was occurring, which would be strange because this is a DEV instance.

What I did to troubleshoot:

1. Suspend and resume the Capacity again.
   No luck
2. Scale up from an F4 to an F8.
   No luck, and based on my monitoring solution we were only using about 50% of the capacity anyway.
3. Changed the permissions of the managed identity connecting to the Workspace (Viewer and Contributor via direct and group assignments).
   No luck. As mentioned, nothing had changed, so everything should still have been working as before.

The solution

You will never guess this, or at least I would not have.
Not based on the exception, nor on my expectations of how a product like Fabric should behave.

As you might know, every resource in Fabric has an Owner assigned to it. By default, this is the account that created the resource. Often, this is a person. If you have a deployment pipeline, it is probably a service principal, which is recommended. For an environment used for development, and sometimes a Test environment too, I think this is most often still a person’s identity. It turns out that if this person leaves the company and the account is removed from Entra ID, the resources in Fabric stop functioning.
The MWC token error is basically saying something like “hey, I can’t create a token for the Owner of this resource, so you can’t use this resource anymore”.
Needless to say, my mind was blown when I learned about this.

To solve the issue, you need to navigate to the resource, open its Settings, and press the Take Over button, as highlighted in the image below.

Obviously, this will happen again when I, or whoever owns a resource at that time, is removed from the company’s Entra ID.

It has taken me quite some time and tokens to figure out how to solve the issue. Hopefully this post will save others from spending the same amount of energy.