Last Thursday, we started having connectivity issues with our data repositories hosted in Microsoft Fabric. We experienced multiple timeouts, were no longer able to connect to the SQL databases, and queries to the Warehouse and Lakehouse also failed. After some initial analysis, we discovered the following error in our logs and IDEs:

Retrieval of MWC token used for accessing storage failed with error ‘0xa (MWC service error: Server responded with error: 400)(DmsPbiServiceUserException: An internal system error has occurred.)’ (7451)

Opening the Warehouse and running a simple SELECT-query resulted in the same error:

We found this quite strange, as there had not been any deployments or changes in either Microsoft Fabric or our Microsoft Azure solution in the past couple of days.

The error was not very obvious to me either. I figured there must have been something wrong with the Workspace, maybe due to suspending and resuming the Capacity on a daily basis. It could be that something was stuck in an erroneous state, or maybe the capacity had reached its limits and throttling was occurring, which would be strange because this is a DEV instance.

What I did to troubleshoot:

1. Suspend and resume the Capacity again.
   No luck
2. Scale up from an F4 to an F8.
   No luck, and based on my monitoring solution we were only using about 50% of the capacity anyway.
3. Changed the permissions of the managed identity connecting to the Workspace (Viewer and Contributor via direct and group assignments).
   No luck. As mentioned, nothing had changed, so everything should still have been working as before.

The solution

You will never guess this, or at least I would not have.
Not based on the exception, nor on my expectations of how a product like Fabric should behave.

I use Microsoft Fabric on a project to store all customer data. Each customer gets their own workspace, so data is isolated. While Microsoft Fabric has its challenges and comes with a hefty price, it does bring quite a lot of useful data solutions under one umbrella. If you only need to store data in a (simple) database there are many more solutions that will fit your use case better. For my project we need to do ingestion, transformation, cleansing, store structured data, store unstructured data, etc. With all of these requirements, using Fabric makes more sense.

During development and testing we ran into capacity issues, which resulted in strange errors when querying or ingesting data.
To keep track of capacity usage you can install the Microsoft Fabric Capacity Metrics app from the store. This app provides useful insights, but it does require users to have a Power BI Pro license. If your users already have this, make sure to install the app. If not, you might want to get this for the users managing the solution, so be sure to discuss that.
I wanted to see if I could create something to deliver the same or similar insights. The Metrics app is based on the available data, so I figured it should be possible to query this myself too. Spoiler: it’s possible, but it does require a bit of setup.