Some time ago the Application Insights became available as a preview in the Azure portal. Application Insights helps you monitor the state of an application, server, clients, etc. As said, it’s still in preview, but it’s rather stable and very easy to use and implement in your applications.

The documentation is still being worked on, but with all the getting started guides on the Microsoft site you can kick start your project with it in a couple of minutes.

The main reason for me to dig into Application Insights is because we still had to implement a proper logging solution for our applications which are migrating to the Azure Cloud. As it happened, Application Insights just became available at the time and because of the tight Azure integration we were really eager to check it out (not saying you can’t use it with non-Azure software of course).

If you want to start using Application Insights, the first thing you will have to do is creating a new Application Insights application. You will be asked what type of application you want to use.

It’s wise to select the proper application type over here, because a lot of settings and measuring graphs will be set for you depending on the choice made over here. For this example I’ll choose the ASP.NET web application. After having waited for a few minutes the initial (empty) dashboard will be shown.

Once you have set up your sharding solution with a fully configured Shard Map Manager, modified your data access layer to use Elastic Scale, added fault handling and running your stuff in production, there will be a time when you are in need to split, merge or move shardlets between shards.

This is where the Elastic Scale Split Merge tool comes in place. The team has created a nice web application which will enable you to do this kind of management. In order to use this tooling, you have to download the latest Nuget package (1.0.0.0 at the moment) into your project. This will create a new folder called splitmerge which contains two subfolders (powershell & service).

The folder containing the PowerShell scripts will give you the power to move, merge and split shards via scripting. A preferred solution for most power users.

The service folder contains a package and configuration template to deploy a web application which is able to do the exact same thing as the PowerShell scripts.

As I don’t fancy big PowerShell scripts for blog posts that much, this post will dig into the service part a bit more.

There is a nice tutorial available in the documentation on Elastic Scale describing on how to use and configure the Split Merge service. The basics come down to specifying a connection string to your newly created database and file storage in the configuration file. Afterwards you can create a new cloud service with the package and configuration. If all works well, you are ready to use the tooling. A best practice is to copy the settings of your deployed configuration file somewhere safe and do incremental updates on it when a new version of the Nuget packge is released.

There are quite a lot of differences between having an on-premise data center and using the cloud. One of these differences is the (guaranteed) uptime and the latency between the different servers. When creating your local on-premise datacenter you will have a pretty stable network connection between the different servers and it’s probably really fast. The cloud can be pretty fast also, especially when you are located in the same datacenter/container. However, you don’t have any real guarantees on where your stuff gets deployed, let alone the latency or connection between the servers.

This might lead to some small problems when you have to hit a different server, like when accessing files from storage or querying a database. This is why the Microsoft Patterns & Practices team has published the Transient Fault Handling Application Block. This application block states there are several transient exceptions which can occur when you try to access a service. These exceptions are known to sometimes be automatically be resolved over a small period of time, therefore retrying within a small period of time might fix the problem.

Basically, all this application block does is helping you add a retry-mechanism so you don’t have to worry much about these transient errors yourself. There are a couple of retrying strategies available out of the box when using the Entity Framework. The SQL team has also provided a new retrying mechanism to be used with the Elastic Scale libraries.

Now that you have configured Elastic Scale for your solution there are still some changes to be made in your application.

At the moment there are 2 sample applications available, one using plain old SQL-queries (ADO.NET) and another one using the Entity Framework. I would suggest checking out these samples before doing any serious Elastic Scale work.

You will probably notice there are some small differences between the ADO.NET sample and the Entity Framework sample. One of them is the Entity Framework not having a sample for doing multi sharded queries. This is because Entity Framework (or any other ORM for that matter) can’t use the multi shard connection objects which are used by the Elastic Scale libraries. I’ve developed a small work around (hack) for this which is shown at the end of this post.

First, let’s step through the basics on how the Elastic Scale library is supposed to be used.

As stated in my earlier posts about this subject, Elastic Scale should be used when you are creating different shards each with subsets of data. In order to connect to the a shard all you need to do is use the Elastic Scale library.

// Looks up the key in the shard map and opens a connection to the shard
using (SqlConnection conn = shardMap.OpenConnectionForKey(customerId, credentialsConnectionString))
{
	// Create a simple command that will insert a new order
	SqlCommand cmd = conn.CreateCommand();

	//Execute your commands, the way you would normally do.
}

As you can see, all you need is the shard map you want to use, the identifier to search for and (the shardlet) and a credentialsConnectionString. The shardlet is specified in the ShardMap of the ShardMapManager. The connection string is something you will have to create yourself. It should only contain the credentials to log in to a database as the Elastic Scale library will find out the server and database by itself. The sample application has a nice helper method for it which you can use of course.

When implementing a sharding solution, you will need something which knows in what shard a specific shardlet exists. This is something you will want to store in a single location, so you know for sure you are always using the most recent information. When using the Elastic Scale libraries this is called the Shard Map Manager. The Shard Map Manager keeps track of the location & state of the shardlets and shards. As you can imagine this is quite an important aspect of the sharding solution.

In a perfect world you will generate the Shard Map Manager (SMM) once, telling it which which shardlets reside in a specific shard and never update it again. Since the Shard Map Manager only exists in one location and hardly ever changes, it’s a great candidate for caching. This is why the Elastic Scale libraries are making sure the content of the Shard Map Manager is cached right after the first call to the database. This way the latency between the SMM and the remote location will only be hit once, after this first call it will be in-memory of the invoking location.

In the real world however, the SMM will get some changes from time to time. For example, if you are sharding by continent you might decide you want to narrow them down a bit by changing US to West US and East US. When sharding with ranges (0..100, 100..200, etc.) you might have to add some new ranges from time to time.

These days we all want to build the next big thing which will be deployed across the world. This of course is all fun and games, but there are also some technical difficulties you have to overcome when creating a software platform which has to be available from everywhere in the world with a responsive interface.

One of these difficulties you will have to face is getting the required data near your customers. Most of the time we are using a database to store this data for us. In the traditional form, this will likely be an on-premise database somewhere in a datacenter (with all disaster recovery aspects in place of course). Note, I’m talking about the traditional relational databases over here, but most of it will also apply on the non-relational databases.

In order to deploy your software globally, the data has to move with it. It doesn’t make much sense to deploy your software solution in a datacenter on the other side of the world when all data still remains far away, because the latency will slow down the experience. There are of course multiple solutions you can think off to solve this problem, one of them is to use sharding in your database.

For our automated deployments we have several Azure Organizational accounts in place. These are created within the Azure Active Directory.

Because these accounts are meant for services, we don’t want them to inherit the default password policy for renewing their passwords every X days. Lucky for us, you can configure this via PowerShell. A short how-to is written on MSDN.

The thing that isn’t written (or referenced) over there is how to run the MSOL cmdlets.

I kept getting the messages The term 'Set-MsolUser' is not recognized. By searching a bit on this error I found a thread on the Office365 community forums where someone mentioned the “Microsoft Online Service Module for Windows PowerShell”. This set me off to searching in the right direction. Apparently you need to install a (new/extra) PowerShell module on your system in order to use the MSOL cmdlets. These cmdlets are part of the Office365 and Exchange Online services. A page with download links is provided by Microsoft Support. They provide a link to the Microsoft Online Service Sign-in Assistant for IT Professionals and the Azure Active Directory Module for Windows PowerShell (32-bit and 64-bit).

Once installed, you are finally able to use the MSOL cmdlets. Keep in mind though, you have to connect to the MSOL services first using the connection cmdlet.

The past couple of days I’ve had the pleasure to start using a Git server as the new version control system at my customer. I’ve already had the pleasure to use GitHub and BitBucket in the past, which works like a charm with tools like GitHub for Windows and SourceTree. Because I’m used to using these tools, I really wanted to use them on my day job also.

Because we chose to use SSH keys as a validation mechanism I had to create one. Once you know how to do this it’s quite easy, but till 2 days ago I didn’t had a clue.

Lucky for me there’s a nice tutorial on how to create SSH keys on the GitHub help pages. Also Atlassian has provided an extensive help document with a couple of steps. In retrospect I think the Atlassian help page is the most useful for helping out with local Git servers. However, these help documents don’t take into account the usage of SourceTree and you will need to do some extra steps to get this working.

For future reference, I’ll describe the steps I had to take below.

First thing you want to do is install Git, the Git Extensions and if you haven’t already, SourceTree.

While creating the PowerShell scripts for automatic deployment of the project’s Azure environment I discovered there are multiple Azure PowerShell modules.

When you want to manage a single resource, such as storage accounts, websites, databases, virtual machines, and media services, you need the (default) Azure module. However, when you need to manage resource groups, you will need the AzureResourceManager module.

This is useful information if you want to deploy new Azure websites with a specific hosting plan, like Basic or Standard. To create such websites the command Get-AzureResourceGroup is necessary. If you use PowerShell ISE you will notice this command isn’t available. In order to make this command available, run the following:

Switch-AzureMode AzureResourceManager

Doing so will activate the AzureResourceManager module and you will have a couple of different commands available.

If you want to see which commands are available within this module, run this command:

Get-Command -Module AzureResourceManager | Get-Help | Format-Table Name, Synopsis

Switching back to the ’normal’ Azure module is also very easy. You just need to switch back to the different AzureMode again.

Switch-AzureMode -Name AzureServiceManagement

After switching back, all your normal commands are back again.

Keep in mind, if you need both modules, you need to switch between the AzureModes in your script also!

There are quite a couple of Azure cmdlets made available by Microsoft. All of this sweetness can be installed on your system via the Web Platform Installer. After installing these modules you can start managing your Azure subscription in PowerShell scripts.

Most of the stuff for managing your Azure subscription is implemented in these Azure cmdlets. One of the things which isn’t implemented (yet) is managing the Service Busses in your subscription. It is possible to add, delete and get a new Service Bus namespace with the New-AzureSBNamespace, Remove-AzureSBNamespace and Get-AzureSBNamespace cmdlets, but that’s all you get. You will probably understand, this isn’t enough if you want to deploy your complete environment via a PowerShell script.

Luckily for us we have the ability to use all of the .NET libraries and assemblies on your system. When you search online you will probably find some articles describing how to create service bus queues in C# by using the NamespaceManager. I’ve written some PowerShell which uses this class and creates queues in your subscription.

#First, create a new service bus namespace. This doesn't return the newly created object
New-AzureSBNamespace -Name $servicebusNamespace -Location $locationWestEUDataCenter -CreateACSNamespace $true
#Get the newly created service bus namespace, so we can do stuff with the information.
$azureServicebus = Get-AzureSBNamespace -Name $servicebusNamespace

#We need a tokenprovider for proper credentials
$tokenProvider = [Microsoft.ServiceBus.TokenProvider]::CreateSharedSecretTokenProvider("owner", $azureServicebus.DefaultKey)
#The uri of the namespace
$namespaceUri = [Microsoft.ServiceBus.ServiceBusEnvironment]::CreateServiceUri("sb", $servicebusNamespace, "");
#Now we can finally crate a NamespaceManger which has the power to create new queues.
$namespaceManager = New-Object Microsoft.ServiceBus.NamespaceManager $namespaceUri,$tokenProvider

Write-Host "Creating the queues" -ForegroundColor Green -BackgroundColor Black
#Creating the queues should work by now.
$namespaceManager.CreateQueue($nameOfTheServiceBusQueue)

If you want to start over, you can just delete the complete namespace and run the script again. This can be done with the following command.