For those that don’t know what I’m going to talk about, here is a prelude.

First, I’m going to go through about 10 slides talking briefly about what is azure.  That is, Azure is basically a family of hosted services that Microsoft provides in the cloud.  I’m going to talk about:

• Compute
• Storage
• Cache
• Service Bus
• SqlAzure

Then, I’m going to got through and actually show all the things I do on a daily basis while I’m doing my development work.  That includes using their development environment (called the dev fabric) as well as how I deploy and test.  The product I developed can be found at http://connectionroad.com if you are interested.  It’s basically Microsoft SyncToy to Cloud Storage.

Bring your questions!  This could be very fun.

OK, I think this is the longest post title I’ve ever made, but if you understand it, you’ll know why it needs to be so long.  I discovered this totally by accident.  I would never ever have pressed F11 (step into) from a client side proxy and expect to get into anything but a bunch of ugly machine generated proxy code (especially with Azure).  To my total surprise, I landed right inside my WCF service as if I had started the Windows Azure Developer fabric in debug mode and set a break point.

So, I’ll step through the process an show screen shots on the outside chance I was dreaming and can’t reproduce it. If I can, now I will have proof so I can do it again.  Sorry for the work in progress code you will see.  The point here is really the debugger and not the code I’m showing so try and ignore that.

OK, here we go.

Let’s assume you have a completely configured azure web role that is hosting a simple WCF service.  When you start that web role and point at the service, you’ll get something like this: (just a note that I started the app fabric by deploying directly from Microsoft Visual Studio 2010 with the start/run.  I am using a debug profile but am NOT running in debug mode.  If I do run the app fabric in debug mode, this does not work.

OK, now let’s run my app that has a WCF client configured to connect to this service.  The configuration is the normal “Add Service Reference” you would expect but I’m showing it any how just to be clear.

Next, we simply Launch Visual Studio 2010 in debug mode and put a break point just before we cross over into the WCF service by calling the above proxy code.  Here is what that screen looks like after the break point is hit and I’m about to step INTO.

F11

and…..

There we are!!! right inside the source code that is my WCF service with everything working as if I had launched the site on it’s own, set a break point and waited.

Truly amazing!!!  I have nothing more to say.

Security patches to the operating system

1. Worrying about what happens if the server goes down and you are not around
2. Database backups if your system crashes unexpectedly
3. Scaling servers to meet high demands
4. Other platform provided features such as Cache

The penalty for this unfortunately is that you have to spend more time up front specifying your configuration exactly so that Azure knows how to exactly boot and provision your web site in the cloud.

The Windows Azure I imagine is one where, from Visual Studio, I simply could say “Publish To Azure” , and as long as I have a Microsoft Live sign on, it would basically publish my web site to the Azure Cloud.  Seems to me that as long as the web site can run in my local development environment with either the built in visual studio web server (aka Cassini) or IIS Express, Azure should be able to make this into a cloud web site with no other steps.  I’m not saying don’t offer all the other configuration options, just give me a choice to make it easy and let me go with the defaults.

Am I wrong?  I totally get that there are a ton of configuration options that can be specified, but it seems to be they should not have to be.  If you feel the same or different, let me know your opinion, post your comment below and let the discussion begin now.

Again, I can’t express how much I like having Azure as a destination for deploying web sites.  Even with the current extra work to deploy, it is a huge benefit and worth it.  I strongly recommend taking a serious look at Azure for your web site if you have not already.

MIX11 In Las Vegas has been a really fun and exciting couple of days.  For me, the sessions are interesting, but my feeling is I can always go back and see them on the internet.  The most interesting things happen during the side conversations with Microsoft Team members and friends I don’t see often.

In this video, I’m having a little Fun with Steve Marx and Tina Dawson of the Windows Azure Team.  I’m not sure of Tina’s role, but Steve is undoubtedly my best advocate on the Windows Azure team.  Over the past 6 months or so, since I’ve been working building my first major Azure product (Soon to be found at http://connectionroad.com), Steve has been an immense help in sorting out tough issues.  Patiently explaining concepts as well as taking my “constructive” feedback to the team for further analysis.

or:   http://www.youtube.com/watch?v=kbF5lwqaLNI

(BTW, just a little background on the wardrobe consulting comment at the beginning.  About a year ago, I discovered, well, maybe someone mentioned to me, that wearing white socks with my dark running shoes and long pants was not exactly a fashion statement.  Noticing Steve in the same condition, obviously landing there as I did, we asked Tina for a consult.  Steve asked Tina if white socks with dark shoes and dark pants were reasonable.  Tina’s response was “Only if you are Michael Jackon”.  Later, she tried to mellow her response however by that point every had heard it her hopes of what is said in Las Vegas stays in Las Vegas were totally dashed.)

So, I tried do a web deploy to my Windows Azure Instance this morning and I got a cryptic error that I blogged about here:  http://social.msdn.microsoft.com/Forums/en-US/windowsazuredevelopment/thread/3b1b6204-2a5f-4fe0-aebc-5245edf46c82

It looks something like this:

Error 1 Web deployment task failed.(Remote agent 
(URL https://xxx.cloudapp.net:8172/MsDeploy.axd?site=Web_IN_0_Web) could not be 
contacted.  Make sure the remote agent service is installed and started on the target computer.)
Make sure the site name, user name, and password are correct. If the issue is not resolved, 
please contact your local or server administrator.
Error details:
Remote agent (URL https://xxx.cloudapp.net:8172/MsDeploy.axd?site=Web_IN_0_Web)
 could not be contacted.  Make sure the remote agent service is 
installed and started on the target computer.
An unsupported response was received. The response header 'MSDeploy.Response' was 
'' but 'v1' was expected.
The remote server returned an error: (401) Unauthorized. 
C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.targets 3847 5 Web

Today, I accidentally figured out a way to stop it.  Simply go to the web project and right mouse button on your project and say “Publish”. You will be immediately presented with an option to stop the deployment!

That’s it.  Simple, but pretty undiscoverable IMHO.

In my scenario, I have the same web project that I want to host in two different Windows Azure Data Centers (BTW, Steve Marx let me know it’s “Windows Azure” and not “Azure” at MVP Summit so I’ll try and keep my terminology right as much as I can).  Each Windows Azure Data Center has it’s own azure account (Azure Credential).  It resolves to a different domain name and as part of that scenario, has different properties in the ServiceConfiguration.csfg file.

What Happens By Default

When you create a new Azure WebRole, the wizard automatically creates a web project for you.  In my case, that is fine for the first datacenter deployment, but what about the second?  In this post (hopefully, as I’m blogging as I’m doing it), I will show you how to create multiple deployments of the same web project.  Below is a screen shot of what my project looks like out of the box.

Adding Another Web Role Pointing At Same Web Project

So, here we go.

First, add a new cloud project / web role as follows:

Add the project:

Add the Web Role

Now, associate this role with the original web project “Web” and notice that a new web site has been created by the previous step. Our plan will be to delete the one that just go created.

That seems to have worked, so now let’s remove the WebRole1 project and then delete it from the directory.

Now, I believe we have what we want as shown below.

The final thing I need to do is simply update my ServiceConfiguration and ServiceDefinition files with what I want that is specific to AzureWebRoleConnectionRoadCom.

My plan is to do this for all my different deployments of the same project.  If things don’t continue to go swimmingly, I’ll add some notes to this post to indicate that maybe this wasn’t the best possible plan, but for now, I’m keeping my fingers crossed that this will work.

As part of the meetup, Robin (@robindotnet) and I both did short presentations on how to  work SqlAzure, basically based around the problem of Azure (by design) dropping connections.  Robin talked about her real world experiences, and I talked about a method for how to deal with the problem elegantly in ado.net.  I did a blog post explaining what I presented here:  http://peterkellner.net/2011/01/21/sqlazure-connection-retry-problem-using-best-practices/

Of course ,what would a meet up be without pictures.  Feel free to add comments to the post about Ward Bell and the others.

That’s it for now!  Looking forward to the next Bruno Meetup

If you’ve started using SqlAzure for your SqlServer with your Azure application, you’ve probably discovered that you get a reasonable number of connection failures.  The advice from the Azure team is add retry logic to all your connections to SqlAzure. There is a long discussion posted by the Azure team here.

The key paragraph states the problem as follows:

The Problem
One of the things that SQL Azure does to deliver high availability is it sometimes closes connections. SQL Azure does some pretty cool stuff under the covers to minimize the impact, but this is a key difference in SQL Azure development vs. SQL Server development.

Basically, what this means is that you must be able to deal with connections failing when you call SqlAzure.  Something that all of probably should have been doing forever, but because most of the time SqlServer is running on your local LAN and the likelihood if a connection failing was next to zero unless something else was going terribly wrong.  Certainly not something we had to do on regular basis.  To emphasize that even more, most of the controls built into asp.net that open connections to sqlserver don’t even do this and that’s from Microsoft itself.

The solution proposed in the thread mentioned above basically has you add tons of code to everyplace you access a connection object.  Personally, I don’t like that because I have hundreds if not thousands of places I open connections and inserting tens of thousands of lines of extra new untested code is a little scary.

So, what to do?

Fortunately, another team at Microsoft, known as the Windows Server AppFabric Customer Advisory Team published a general purpose solution using Extension Methods and some darn clever coding wrote a great article and published code including azure examples that solves this problem very elegantly without requiring a lot of changes to your existing code base.

In this article I plan on giving an example and publishing a sample project that uses this code with SqlAzure to solve the connection retry problem.  My goal here is not to simply restate what they published but to simply have a very simple concrete example of using their library.

Design Goal

We have two goals.

1. Change as little code as possible
2. Log Connection Errors when they happen With Locations
3. Make sure not to trap errors that are NOT connection related such as bad column names

Incorrect Code

So, this is what the original code looks like that will fail because it does not have connection retry logic:

public static int UsersIdFromUserNameNoConnectionRetry(string userName)
{
    var retUsersId = 0;
    const string sql =
        @"SELECT Id FROM Users
            WHERE Username = @Username";

    using (var sqlConnection = new SqlConnection(
            ConfigurationManager.ConnectionStrings["CRStorageWebConnectionString"].
            ConnectionString))
    {
        sqlConnection.Open();
        using (var sqlCommand = new SqlCommand(sql, sqlConnection))
        {
            sqlCommand.Parameters.Add("@Username", SqlDbType.NVarChar).Value = userName;
            using (var reader = sqlCommand.ExecuteReader())
            {
                while (reader.Read())
                {
                    retUsersId = reader.IsDBNull(0) ? 0 : reader.GetInt32(0);
                }
            }
        }
    }
    return retUsersId;
}

So, if there is a connection, an exception will get thrown and will need to be caught, but then the method will not have done it’s job.

Correct Code With Retries

So, now take a look at the revised code after the library is setup and used.  The setup is non-trivial, but you just have to do that once and then you can simply fix all your other code with very few changes.  Below is the new code with connection retry logic built in.

public static int UsersIdFromUserName(string userName)
{
    var retUsersId = 0;
    const string sql =
        @"SELECT Id FROM Users
            WHERE Username = @Username";

    using (var sqlConnection =
        new ReliableSqlConnection(
            ConfigurationManager.ConnectionStrings["CRStorageWebConnectionString"].
            ConnectionString,
            new RetryUtils("", "UsersIdFromUserName").GetRetryPolicy()))
    {
        sqlConnection.Open();
        using (var sqlCommand = new SqlCommand(sql, sqlConnection.Current))
        {
            sqlCommand.Parameters.Add("@Username", SqlDbType.NVarChar).Value = userName;
            using (var reader = sqlCommand.ExecuteReader())
            {
                while (reader.Read())
                {
                    retUsersId = reader.IsDBNull(0) ? 0 : reader.GetInt32(0);
                }
            }
        }
    }
    return retUsersId;
}

1. The first is instead of create an SqlConnection(…) we are creating a ReliableSqlConnection(…).  The ReliableSqlConnection takes an extra parameter which basically wraps the retry logic used, as well as labelling this connection so when it fails, it gets logged.  In the log, there will be the comment “UsersIdFromUserName” so we know what method threw the retry.
2. The second is slight change we when we create the SqlCommand, we have to add the property .Current to it so we know we are talking about the currently executing connection.  There may be a default way to handle this but I could not figure it out.

The Setup Pieces

The project has all kinds of stuff in it that build quite nicely, run and test under vs2010.  All I’m interested in is the ado.net piece and the retry logic around that.  I actually used the 1.2 version, however I now see there is a 1.3 version with some improvements.  I would post my project but I don’t want to post it with old code so I’ll just tell you the steps I went through so you can do the same.

Build the class library

Compile the project and make sure you have the dll from the library lincluded in your actual visual studio project.  The library you want is Microsoft.AppFabricCAT.Samples.Azure.TransientFaultHandling.

Update Your Web.config

Add the config section below to your web.config file.

<configSections>
    <section name="RetryPolicyConfiguration" 
         type="Microsoft.AppFabricCAT.Samples.Azure.TransientFaultHandling.Configuration.RetryPolicyConfigurationSettings,
            Microsoft.AppFabricCAT.Samples.Azure.TransientFaultHandling" />
</configSections>

<RetryPolicyConfiguration defaultPolicy="FixedIntervalDefault" defaultSqlConnectionPolicy="FixedIntervalDefault" 
         defaultSqlCommandPolicy="FixedIntervalDefault" defaultStoragePolicy="IncrementalIntervalDefault" 
         defaultCommunicationPolicy="IncrementalIntervalDefault">
    <add name="FixedIntervalDefault" maxRetryCount="10" retryInterval="100" />
    <add name="IncrementalIntervalDefault" maxRetryCount="10" retryInterval="100" retryIncrement="50" />
    <add name="ExponentialIntervalDefault" maxRetryCount="10" minBackoff="100" maxBackoff="1000" deltaBackoff="100" />
</RetryPolicyConfiguration>

This defines a new section in your web.config, then implements it with several different retry policies include a default one which seems quite reasonable to me.

If you remember, in the implementation section above, we used a public class called RetryUtils.  This class is actually one I invented as a convenience class to minimize the code I have to update on each use of the ReliableConnection Object.  Just to refresh your memory, the implementation is this:

using (var sqlConnection =
                new ReliableSqlConnection(
                    ConfigurationManager.ConnectionStrings["CRStorageWebConnectionString"].
                    ConnectionString,
                    new RetryUtils("", "UsersIdFromUserName").GetRetryPolicy()))

The actual code fo rthe class RetryUtils is below here.  You’ll have to stick this someplace in your project.

namespace Utils
{
   
    public class RetryUtils
    {
        public string Username { get; set; }
        public string CallersName { get; set; }

        public RetryUtils(string username, string callersName)
        {
            Username = username;
            CallersName = callersName;
        }

        public RetryUtils()
        {
            Username = string.Empty;
            CallersName = string.Empty;
        }

        public RetryPolicy<SqlAzureTransientErrorDetectionStrategy> GetRetryPolicy()
        {
            var retryPolicy = new RetryPolicy<SqlAzureTransientErrorDetectionStrategy>
                 (RetryPolicy.DefaultClientRetryCount, RetryPolicy.DefaultMinBackoff,
                  RetryPolicy.DefaultMaxBackoff,
                  RetryPolicy.DefaultClientBackoff);

            retryPolicy.RetryOccurred += RetryPolicyRetryOccurred;

            return retryPolicy;
        }

        void RetryPolicyRetryOccurred(int currentRetryCount, Exception lastException, TimeSpan delay)
        {
            GeneralUtils.GetLog4NetAllDataContext().AddLog4NetAll(new Log4NetAll
                                                    {
                                                        Date = DateTime.UtcNow,
                                                        EllapsedTime = 0,
                                                        ExceptionStackTrace = lastException.StackTrace,
                                                        Message = "RetryCount: " + currentRetryCount + " delay: " + delay.Seconds,
                                                        ExceptionMessage = lastException.Message,
                                                        Logger = "",
                                                        Level = "Error",
                                                        UserName = Username,
                                                        PartitionKey = Username,
                                                        RowKey = Guid.NewGuid().ToString()
                                                    });
        }       
    }
}

Non Connection Related Errors

Conclusions

Two of the sections I feel are particularly well written or the ones that talk about Storage and Tables (chapters 7 and 8).  The application I’ve been working on heavily uses blob storage and I spend lots of times re-reading those sections.  Azure tables is a hard topic to get a sql server programmer like me to get my head wrapped around.  Sriram does a great job of talking directly to someone like me to help me understand how and when to use tables.

Again, I whole heartedly recommend this book.

I’m attaching a screen shot below which shows viewing what is actually in the storage (not a hierarchical false view) as well as a display of all the meta data.  Either of those two features are show starters for me and enough to switch.

If you have not used it, give it  try (and have some patience, it takes a little bit of practice to find the true value).

Good luck!

Just so we now have an example, I’ve put together a simple windows form app that let’s you set a few variables in your app.config to point at your azure storage and container, let you view your app as a tree as well as see the code how it can be done.  I have not commented the code much, just thought it would be good to get it out there.  The running application shows you the data as follows.

So, for the details, I’m pasting below the meet of the code.  Basically, it does what you would expect in terms of iterating through the directories recursively to build the list.  Again, just set your parameters in your app.config as follows:

and you can run it for yourself and see how it goes.

I’m pasting some code below, as well as including the project source code.  The main reason I’m pasting the code is so that the search engines can find it better.  It’s really just easier to download the project and load it into Visual Studio 2010 and run it yourself.

Good Luck, and hope this helps!

1. 3 to 63 Characters
2. Starts With Letter or Number
3. Letters, Numbers, and Dash (-)
4. Every Dash (-) Must Be Immediately Preceded and Followed by a Letter or Number

So, I decided to hunt for a regular expression to do this and I found one on the web. I’m not sure if it’s correct, but here is what I found at http://social.msdn.microsoft.com/Forums/en-GB/windowsazuredata/thread/d364761b-6d9d-4c15-8353-46c6719a3392 from Gaurav Mantri (method that includes the regex).

internal static bool IsBlobContainerNameValid(string name)
    {
      if (name.Equals("$root"))
      { 
        return true; 
      }
      string validBlobContainerNameRegex = @"^([a-z]|\d){1}([a-z]|-|\d){1,61}([a-z]|\d){1}$";
      Regex reg = new Regex(validBlobContainerNameRegex);
      if (reg.IsMatch(name))
      {
        return true;
      }
      return false;
    }

My friend RobinDotNet suggested I look at his article about how to use Visual Studio 2010 to view the blob storage.  Sure enough, in VS, I can set my account and key in server explorer and see the blob storage!  Thanks Robin.

Link to Thread with details below:

Reply:

The billing portal will give you a fairly detailed report on your storage consumption (it’s about 4 hours behind realtime). You can get to this from the Azure portal (should be near the upper-right by your Live ID). This won’t give you per-container breakdowns or anything that detailed though.

This Azure Storage Team blog post gives a fairly detailed breakdown of how consumption is computed. Knowing this, you could create a more detailed storage consumption report than the billing report by enumerating your containers, then enumerating each blob within the containers and reading each blob’s size (via CloudBlob.Properties.Length).

There’s nothing I know of that will give you a bandwidth breakdown for each container. For that, you’d need to create some sort of service tier, where you could then retrieve the blob in question and accumulate your own stats within that service tier.

On to the review..

As I mentioned above, The book starts out with a very general discussion of what is cloud computing.  Currently, the cloud is a pretty over used word and it’s important to understand what is meant by it.  Things like “Application as Service”, “Cloud Storage”, etc. are all important to understand when talking about the cloud.

Following that, the book dives into the details of Azure (and there are a lot of details).  It goes into great details (and specifics) on Azure Blob Storage, Azure Queues, Azure Tables, AppFabric (formerly Velocity), Azure Service Bus, and also discusses deployment issues.

I do recommend this book.  I believe it will help if you want to get into the cloud using Microsoft’s offerings.

Windows Azure Platform (Pro)