So, here is the problem.  I created a CodeFirst implementation of EntityFramework in Visual Studio and created a simple hierarchy in my data model.  That is, without showing all the code, here is what I have:

public class Person
{
    [Key]
    [DatabaseGeneratedAttribute(DatabaseGeneratedOption.Identity)]
    public int Id { get; set; }

    public virtual List<EmailAddressInfo> EmailAddressInfoList { get; set; }
    ..
}

public class EmailAddressInfo
{
    [Key]
    [DatabaseGeneratedAttribute(DatabaseGeneratedOption.Identity)]
    public int Id { get; set; }

    public string EmailAddress { get; set; }

}

public class EmailAccount
{
    [Key]
    [DatabaseGeneratedAttribute(DatabaseGeneratedOption.Identity)]
    public int Id { get; set; }
    ...
}

Now, if you execute a LINQ command like the following:

var recs = (from data in db.Persons
                            select data);
                foreach (var rec in recs)
                {
                    foreach (var email in rec.EmailAddressInfoList)
                    {
                        Console.WriteLine(email);
                    }
                }

You get the error:

Exception Details: System.InvalidOperationException: There is already an open DataReader associated with this Command which must be closed first.

To solve this, you need to tell EntityFramework/CodeFirst to include the EmailAddressInfoList in the query.  All you need to do is change the db.Persons to db.Persons.Include("EmailAddressInfoList") as follows:

var recs = (from data in db.Persons.Include("EmailAddressInfoList")
            select data);

Then, it all works!

Hope this helps.

<system.webServer>
    <validation validateIntegratedModeConfiguration="false" />
    <modules runAllManagedModulesForAllRequests="true" />
    <staticContent>
      <mimeMap fileExtension=".json" mimeType="application/json"  />
    </staticContent>
    ...

This works great as long as the GET verb is used (or just enter the on the url like http://mysite.com/myfile.json).

So, what if you "need” to use the POST keyword.  Say for example, you can not change the JavaScript file to use GET instead of POST to get the file.  I was hoping to find some simple web.config parameter to set but I had no luck.  I even posted to StackOverflow and so far, the answers have been less than helpful.  Who knows, maybe by the time you read this, there will be a better answer then my solution which is to write a simple asp.net handler and register it to type json.

So, here is the simpler handler that does the trick:

public class JSONHandler : IHttpHandler
    {

        public void ProcessRequest(HttpContext context)
        {
            context.Response.ContentType = "application/json";
            string output = System.IO.File.ReadAllText(context.Request.PhysicalPath);
            context.Response.Write(output);

        }

        public bool IsReusable
        {
            get
            {
                return false;
            }
        }
    }

and the associated web.config entries

<httpHandlers>
     <add verb="*" path="*.json" validate="false" type="JSONHandler" />
   </httpHandlers>
 </system.web>

 <system.webServer>
   <validation validateIntegratedModeConfiguration="false" />
   <modules runAllManagedModulesForAllRequests="true" />
   <staticContent>
     <mimeMap fileExtension=".json" mimeType="application/json"  />
   </staticContent>
   <handlers>
     <add verb="*" path="*.json" name="JSONHandler" type="JSONHandler"/>
   </handlers>
 </system.webServer>

I’m assuming there is a simple way, but for now, this works for me.  Please post a better solution and reference it or just tell me in the comments.

I know this is old hat, but after recently having a conversation with one of the ASP.NET Team guys about how easy Web Forms is to use, I thought I’d just post a simple example of that.  Here is the scenario.  You have a simple GridView with just three columns (Select,Id,TagName).  You want to be able to capture what happens when the user pressed the Select command and capture the Id.  It looks like this:

And has an aspx page like this:

<asp:GridView ID="GridViewNotSelected" runat="server" AllowSorting="True" 
        AutoGenerateColumns="False" 
        DataSourceID="SqlDataSourceTagsNotSelected" 
        onrowcommand="GridViewNotSelected_RowCommand">
    <Columns>
        <asp:TemplateField ShowHeader="False">
            <ItemTemplate>
                <asp:LinkButton ID="LinkButton1" runat="server" 
                    CausesValidation="False" CommandArgument='<%# Eval("Id") %>'
                    CommandName="Select" Text="Select"></asp:LinkButton>
            </ItemTemplate>
        </asp:TemplateField>
        <asp:TemplateField 
            HeaderText="&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;">
        </asp:TemplateField>
        <asp:TemplateField>
            <ItemTemplate>
                <asp:Label ID="Label1" Text='<%# Eval("TagName") %>' runat="server"></asp:Label>
            </ItemTemplate>
        </asp:TemplateField>
    </Columns>
</asp:GridView>

Notice that in the TemplateField there is LinkButton with a CommandArgument that simply extracts the TagName with the current Id of the SqlDataSource.  Now, all you need is the RowCommand event and you are done.  Here it is in C#.

protected void GridViewNotSelected_RowCommand(object sender, GridViewCommandEventArgs e)
{
    int tagsId = Convert.ToInt32(e.CommandArgument);
}

And finally, the proof!

Very very simple.  HTH’s.

Introduction and Problem

Converting Data Formats back and forth between browser and back end servers can be a little confusing.  In my case, I have a simple Sencha ExtJS 4.0 form that the user types a date into.  That date gets passed through an ajax request to the Microsoft asp.net controller, then that dates updates the SqlServer Database.  When the Date is displayed on a Grid Panel, again, an ajax request is made that grabs the date from the controller, then formats it using the MS dateFormat from ExtJS and hopefully, the same date the user typed in gets re-displayed.

So, let’s look at the details.

Posting the Date To the Server (View From JavaScript)

First, we need to define an ExtJS Model that has the date in it.  That codes looks like this in my case.

Ext.define('EvMgr.model.SponsorListJobListing', {
    extend: 'Ext.data.Model',
    fields: [
        'Id',
        'SponsorListId',
        'JobName',
        'JobLocation',
        'JobURL',
        'JobBrief',
        'JobTagline',
        'JobButtonToolTip',
        { name: 'EnteredDate', type: 'date', dateFormat: 'MS' },
        'JobCompanyName',
        { name: 'StartRunDate', type: 'date', dateFormat: 'MS' },
        { name: 'EndRunDate', type: 'date', dateFormat: 'MS' },
        { name: 'HideListing', type: 'boolean' },
        'Notes'
    ],
    idProperty: 'Id',

    proxy: {
        type: 'ajax',
        api: {
            read: 'SponsorListJobListing/Get',
            update: 'SponsorListJobListing/Update',
            create: 'SponsorListJobListing/Create',
            destroy: 'SponsorListJobListing/Delete'
        },
        reader: {
            type: 'json',
            root: 'Data',
            successProperty: 'success'
        }
    },


    belongsTo: 'EvMgr.model.SponsorList'
});

Notice the StartRunDate field has dateFormat of ‘MS”.

Now, we have an ExtJS Window that we use to collect the data from the user.  The code is as follows:

Ext.define('EvMgr.view.sponsorlist.ListJobListingsEdit', {
    extend: 'Ext.window.Window',
    alias: 'widget.sponsorlistjoblistingsedit',
    title: 'Edit Job Listing',
    layout: 'fit',
    width: 1000,
    autoShow: true,

    initComponent: function() {

        this.items = [
            {
                xtype: 'form',

                items: [
                    {
                        xtype: 'textfield',
                        name: 'JobName',
                        fieldLabel: 'JobName',
                        width: 850
                    },
                    {
                        xtype: 'textfield',
                        name: 'JobLocation',
                        fieldLabel: 'JobLocation',
                        width: 850
                    },
                     {
                         xtype: 'textfield',
                         name: 'JobURL',
                         fieldLabel: 'JobURL',
                         width: 850
                     },
                     {
                         xtype: 'textfield',
                         name: 'JobBrief',
                         fieldLabel: 'JobBrief',
                         width: 850
                     }, {
                         xtype: 'textfield',
                         name: 'JobTagline',
                         fieldLabel: 'JobTagline',
                         width: 850
                     },
                     {
                         xtype: 'textfield',
                         name: 'JobButtonToolTip',
                         fieldLabel: 'JobButtonToolTip',
                         width: 850
                     },
                     {
                         xtype: 'textfield',
                         name: 'JobCompanyName',
                         fieldLabel: 'JobCompanyName',
                         width: 850
                     },
                     {
                         xtype: 'textfield',  // checkbox makesthe value not come back for some reason
                         name: 'HideListing',
                         fieldLabel: 'HideListing',
                         width: 850
                     },
                     {
                         xtype: 'datefield',
                         name: 'StartRunDate',
                         fieldLabel: 'StartRunDate',
                         width: 850
                     },
                     {
                         xtype: 'datefield',
                         name: 'EndRunDate',
                         fieldLabel: 'EndRunDate',
                         width: 850
                     },
                    {
                        xtype: 'textfield',
                        name: 'SponsorListId',
                        fieldLabel: 'SponsorListId',
                        width: 850
                    }
                ]
            }
        ];

        this.buttons = [
            {
                text: 'Save',
                action: 'save'
            }, {
                text: 'Cancel',
                scope: this,
                handler: this.close
            }
        ];

        this.callParent(arguments);
    }
});

Posting the Date To the Server (View From ASP.NET)

I’m over simplifying a little because it’s calculating UTC based on the Servers time zone, not the browsers.  It would be better to send the timezone of the browser also and then calculate UTC correctly in JavaScript, but I just want to give you the idea here.  In my case, I’m always running this app from California and my server is always inCalifornia so at least things stay consistent.  I do realize I’m simplifying and you need to make sure you think this through.

Now, when I query the server for the returned data using the same ExtJS data model, the data comes down as follows.  In ASP.NET it looks like this:

Which is actually different than what was pushed up.  It seems though that when ExtJS displays this, it converts it to local time.

Our browser Grid shows it like this (as we would expect)

Hope this helps!

Something that may not be obvious is if are creating an asp.net WebForms project and you put a datasource such as SqlDataSource or ObjectDataSource for example on the page, how can you prevent the SqlSelect associated with that datasource from being triggered.

The answer is to set the control’s visible property to false.  That’s it!

The Why

The reason you might want to do this is for a case where you have a public facing web page that might be easily subjected to a denial of service attach.  If that web page is always causing some SqlDataSource to fire, you could easily find your SqlServer overloaded.  At least by checking to see a user is logged in before firing the sql statement, you buy yourself a little bit of protection.

The Code

<%@ Page Language="C#" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<script runat="server">

    protected void 
        GridView1_SelectedIndexChanged(object sender, EventArgs e)
    {

    }

    protected void Page_Init(object sender, EventArgs e)
    {
        if (!Context.User.Identity.IsAuthenticated)
        {
            GridView1.Visible = false;
        }
    }
</script>

<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title></title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
        <asp:SqlDataSource ID="SqlDataSource1" runat="server" 
            ConnectionString="<%$ ConnectionStrings:CodeCampSV06 %>" 
            SelectCommand=
            "SELECT [Id], [Username], [Email] FROM [Attendees] ORDER BY [Id] DESC">
            </asp:SqlDataSource>
        <asp:GridView ID="GridView1" runat="server" AutoGenerateColumns="False" 
            DataKeyNames="Id" DataSourceID="SqlDataSource1" 
            onselectedindexchanged="GridView1_SelectedIndexChanged">
            <Columns>
                <asp:BoundField DataField="Id" HeaderText="Id" InsertVisible="False" 
                    ReadOnly="True" SortExpression="Id" />
                <asp:BoundField DataField="Username" HeaderText="Username" 
                    SortExpression="Username" />
                <asp:BoundField DataField="Email" HeaderText="Email" 
                SortExpression="Email" />
            </Columns>
        </asp:GridView>
    </div>
    </form>
</body>
</html>

Here it is.  Can’t be much simpler I don’t think.

byte[] downloadByteArray = File.ReadAllBytes(imageLocation);

That’s it! hope this helps someone else.

The big difference is that GetTempFileName actually creates a file where as GetTempPath simply tells you where the file is going to be.

Here are some words out of MSDN for Path.GetTempFileName

This method creates a temporary file with a .TMP file extension.

And For Path.GetTempPath

The path to the temporary folder, ending with a backslash.

Subtle, but clear when you read the doc.  If you read the community content, there is a humorous discussion with someone saying “you might as well use the other one, it does the same thing”.  All I can say to that is read the docs!  I got stung today, but never again.

I’m now creating my filename with a string as follows which kind of does the same thing, but not really. 

public static string GetTempFileWithGuid(string filePrefix)
{
    string retFileName = string.Format("{0}{1}{2}.crsync", 
        Path.GetTempPath(), filePrefix, Guid.NewGuid());
    return retFileName;
}

HTH’s.

If you’ve started using SqlAzure for your SqlServer with your Azure application, you’ve probably discovered that you get a reasonable number of connection failures.  The advice from the Azure team is add retry logic to all your connections to SqlAzure. There is a long discussion posted by the Azure team here.

The key paragraph states the problem as follows:

The Problem
One of the things that SQL Azure does to deliver high availability is it sometimes closes connections. SQL Azure does some pretty cool stuff under the covers to minimize the impact, but this is a key difference in SQL Azure development vs. SQL Server development.

Basically, what this means is that you must be able to deal with connections failing when you call SqlAzure.  Something that all of probably should have been doing forever, but because most of the time SqlServer is running on your local LAN and the likelihood if a connection failing was next to zero unless something else was going terribly wrong.  Certainly not something we had to do on regular basis.  To emphasize that even more, most of the controls built into asp.net that open connections to sqlserver don’t even do this and that’s from Microsoft itself.

The solution proposed in the thread mentioned above basically has you add tons of code to everyplace you access a connection object.  Personally, I don’t like that because I have hundreds if not thousands of places I open connections and inserting tens of thousands of lines of extra new untested code is a little scary.

So, what to do?

Fortunately, another team at Microsoft, known as the Windows Server AppFabric Customer Advisory Team published a general purpose solution using Extension Methods and some darn clever coding wrote a great article and published code including azure examples that solves this problem very elegantly without requiring a lot of changes to your existing code base.

In this article I plan on giving an example and publishing a sample project that uses this code with SqlAzure to solve the connection retry problem.  My goal here is not to simply restate what they published but to simply have a very simple concrete example of using their library.

Design Goal

We have two goals.

1. Change as little code as possible
2. Log Connection Errors when they happen With Locations
3. Make sure not to trap errors that are NOT connection related such as bad column names

Incorrect Code

So, this is what the original code looks like that will fail because it does not have connection retry logic:

public static int UsersIdFromUserNameNoConnectionRetry(string userName)
{
    var retUsersId = 0;
    const string sql =
        @"SELECT Id FROM Users
            WHERE Username = @Username";

    using (var sqlConnection = new SqlConnection(
            ConfigurationManager.ConnectionStrings["CRStorageWebConnectionString"].
            ConnectionString))
    {
        sqlConnection.Open();
        using (var sqlCommand = new SqlCommand(sql, sqlConnection))
        {
            sqlCommand.Parameters.Add("@Username", SqlDbType.NVarChar).Value = userName;
            using (var reader = sqlCommand.ExecuteReader())
            {
                while (reader.Read())
                {
                    retUsersId = reader.IsDBNull(0) ? 0 : reader.GetInt32(0);
                }
            }
        }
    }
    return retUsersId;
}

So, if there is a connection, an exception will get thrown and will need to be caught, but then the method will not have done it’s job.

Correct Code With Retries

So, now take a look at the revised code after the library is setup and used.  The setup is non-trivial, but you just have to do that once and then you can simply fix all your other code with very few changes.  Below is the new code with connection retry logic built in.

public static int UsersIdFromUserName(string userName)
{
    var retUsersId = 0;
    const string sql =
        @"SELECT Id FROM Users
            WHERE Username = @Username";

    using (var sqlConnection =
        new ReliableSqlConnection(
            ConfigurationManager.ConnectionStrings["CRStorageWebConnectionString"].
            ConnectionString,
            new RetryUtils("", "UsersIdFromUserName").GetRetryPolicy()))
    {
        sqlConnection.Open();
        using (var sqlCommand = new SqlCommand(sql, sqlConnection.Current))
        {
            sqlCommand.Parameters.Add("@Username", SqlDbType.NVarChar).Value = userName;
            using (var reader = sqlCommand.ExecuteReader())
            {
                while (reader.Read())
                {
                    retUsersId = reader.IsDBNull(0) ? 0 : reader.GetInt32(0);
                }
            }
        }
    }
    return retUsersId;
}

1. The first is instead of create an SqlConnection(…) we are creating a ReliableSqlConnection(…).  The ReliableSqlConnection takes an extra parameter which basically wraps the retry logic used, as well as labelling this connection so when it fails, it gets logged.  In the log, there will be the comment “UsersIdFromUserName” so we know what method threw the retry.
2. The second is slight change we when we create the SqlCommand, we have to add the property .Current to it so we know we are talking about the currently executing connection.  There may be a default way to handle this but I could not figure it out.

The Setup Pieces

The project has all kinds of stuff in it that build quite nicely, run and test under vs2010.  All I’m interested in is the ado.net piece and the retry logic around that.  I actually used the 1.2 version, however I now see there is a 1.3 version with some improvements.  I would post my project but I don’t want to post it with old code so I’ll just tell you the steps I went through so you can do the same.

Build the class library

Compile the project and make sure you have the dll from the library lincluded in your actual visual studio project.  The library you want is Microsoft.AppFabricCAT.Samples.Azure.TransientFaultHandling.

Update Your Web.config

Add the config section below to your web.config file.

<configSections>
    <section name="RetryPolicyConfiguration" 
         type="Microsoft.AppFabricCAT.Samples.Azure.TransientFaultHandling.Configuration.RetryPolicyConfigurationSettings,
            Microsoft.AppFabricCAT.Samples.Azure.TransientFaultHandling" />
</configSections>

<RetryPolicyConfiguration defaultPolicy="FixedIntervalDefault" defaultSqlConnectionPolicy="FixedIntervalDefault" 
         defaultSqlCommandPolicy="FixedIntervalDefault" defaultStoragePolicy="IncrementalIntervalDefault" 
         defaultCommunicationPolicy="IncrementalIntervalDefault">
    <add name="FixedIntervalDefault" maxRetryCount="10" retryInterval="100" />
    <add name="IncrementalIntervalDefault" maxRetryCount="10" retryInterval="100" retryIncrement="50" />
    <add name="ExponentialIntervalDefault" maxRetryCount="10" minBackoff="100" maxBackoff="1000" deltaBackoff="100" />
</RetryPolicyConfiguration>

This defines a new section in your web.config, then implements it with several different retry policies include a default one which seems quite reasonable to me.

If you remember, in the implementation section above, we used a public class called RetryUtils.  This class is actually one I invented as a convenience class to minimize the code I have to update on each use of the ReliableConnection Object.  Just to refresh your memory, the implementation is this:

using (var sqlConnection =
                new ReliableSqlConnection(
                    ConfigurationManager.ConnectionStrings["CRStorageWebConnectionString"].
                    ConnectionString,
                    new RetryUtils("", "UsersIdFromUserName").GetRetryPolicy()))

The actual code fo rthe class RetryUtils is below here.  You’ll have to stick this someplace in your project.

namespace Utils
{
   
    public class RetryUtils
    {
        public string Username { get; set; }
        public string CallersName { get; set; }

        public RetryUtils(string username, string callersName)
        {
            Username = username;
            CallersName = callersName;
        }

        public RetryUtils()
        {
            Username = string.Empty;
            CallersName = string.Empty;
        }

        public RetryPolicy<SqlAzureTransientErrorDetectionStrategy> GetRetryPolicy()
        {
            var retryPolicy = new RetryPolicy<SqlAzureTransientErrorDetectionStrategy>
                 (RetryPolicy.DefaultClientRetryCount, RetryPolicy.DefaultMinBackoff,
                  RetryPolicy.DefaultMaxBackoff,
                  RetryPolicy.DefaultClientBackoff);

            retryPolicy.RetryOccurred += RetryPolicyRetryOccurred;

            return retryPolicy;
        }

        void RetryPolicyRetryOccurred(int currentRetryCount, Exception lastException, TimeSpan delay)
        {
            GeneralUtils.GetLog4NetAllDataContext().AddLog4NetAll(new Log4NetAll
                                                    {
                                                        Date = DateTime.UtcNow,
                                                        EllapsedTime = 0,
                                                        ExceptionStackTrace = lastException.StackTrace,
                                                        Message = "RetryCount: " + currentRetryCount + " delay: " + delay.Seconds,
                                                        ExceptionMessage = lastException.Message,
                                                        Logger = "",
                                                        Level = "Error",
                                                        UserName = Username,
                                                        PartitionKey = Username,
                                                        RowKey = Guid.NewGuid().ToString()
                                                    });
        }       
    }
}

Non Connection Related Errors

Conclusions

This post shows a very simple technique for processing a gzip compression on a background thread using c# with Visual Studio 2010.  What is unique here is we are using no statics to do it.  I’m not totally against using statics, but in general, it is best to avoid them.  I’ve heard the notorious Ward Bell say statics are evil and have had many cases where they have bitten me.  Since I heard Ward say this, I’ve been trying to avoid them where I can.

The Simple Problem

The problem is to simply compress a file to bytes and return to us the compressed, uncompressed and byte array of the result.  We can pass parameters into a thread, however we can not return them (when I say thread, I mean the anonymous method that processes our data).

Some Code

So, to that end, Let’s create a main method as below.  Notice that it creates a very simple anonymous method which executes the code cryptoCopress.CompressFile(…), then simply starts that thread.  Once the thread starts, it simply waits for the thread to end by looping every 50 milliseconds on the thread.IsAlive method.  Finally, when it returns, it simply looks at the cryptCompress object for the results.  No Statics!

private static void Main(string[] args)
{
    var cryptCompress =
        new CryptCompress();
    var thread =
        new Thread(
            () =>
            cryptCompress.CompressFile
                (@"G:\NoBackup\ext-3.2.1\ext-all-debug-w-comments.js"));
    thread.Start();

    int cnt = 0;
    while (thread.IsAlive)
    {
        cnt++;
        Console.WriteLine("Waiting... " + cnt);
        Thread.Sleep(50);
    }
    Console.WriteLine("Before Compression KBytes: {0}",
                        cryptCompress.BeforeCompressionBytes/1000);
    Console.WriteLine("After Compression KBytes: {0}",
                        cryptCompress.AfterCompressionBytes/1000);
}

Now, Lets look at the CryptCompress class.  Notice that it’s basically got one public method (CompressFile) and 3 public properties that will be used to hold the return values.  This way, the main method that started the thread can get the results.  Again, notice that there is no word static any place in this project.

public class CryptCompress
{
public byte[] CompressedBytes { get; set; }
public long BeforeCompressionBytes { get; set; }
public long AfterCompressionBytes { get; set; }

public void CompressFile(string fileName)
{
    using (var fileStream = 
        new FileStream(fileName, FileMode.Open, FileAccess.Read))
    {
        var uncompressedBytes = new byte[fileStream.Length];
        fileStream.Read(uncompressedBytes, 0, 
            (int) fileStream.Length);
        CompressedBytes = CompressGzip(uncompressedBytes);
        BeforeCompressionBytes = fileStream.Length;
        AfterCompressionBytes = CompressedBytes.Length;
        fileStream.Close();
    }
}

/// <summary>
/// Take a simple stream of uncompressed bytes and compress them
/// </summary>
/// <param name="uncompressedBytes"></param>
/// <returns></returns>
public byte[] CompressGzip(byte[] uncompressedBytes)
{
    using (var memory = new MemoryStream())
    {
        using 
            (var gZipStream = 
                new GZipStream(memory, CompressionMode.Compress, true))
        {
            gZipStream.Write
                (uncompressedBytes, 0, uncompressedBytes.Length);
        }
        return memory.ToArray();
    }
}
}

The Results

When we run this, notice that it takes 3 iterations (or 150 milliseconds) to complete. I’m only compressing a small file so no surprise.  The file is actually 2.7 Megabytes and compress to .7 Megabytes.

That’s it for now! Hope this helps.

Background And Conclusion

In my very last post I talked about how using SHA256 seemed to not be affected by whether you use Buffered or Not Buffered streams.  An astute reader (Samuel Jack) referenced an article by one of my favorite ex MIcrosoft employees (Brad Abrams) saying that almost all the .Net streams have buffering built in.

Well, I think SHA256Managed does not.  One of the challenges I’ve been facing lately with my current project is to provide feedback while doing all kinds of byte piping (stream stuff).  While figuring out how to do this, I inadvertently figured out that buffering makes a huge difference.  Briefly, let me show my results first, then talk about the code that went into it.

Notice that when I use a 64MB Buffer, the memory used for the process is 77MB and when I use an 8MB buffer, the memory used is 17MB.  Clearly the buffer allocated matters.  Just for information, I’ve also included the code that does not break the SHA256 hash into blocks, and it has the same results based on the size of the Memory Buffer declared.  That is, small buffer, small use, big buffer, big use.

The Code

The code that I wrote to do this basically does a nice job of splitting up the processing so that you can log progress as it runs. I’m attaching the Visual Studio 2010 project here as well as pasting the relevant source code.  It’s a good base to experiment with.  Feel free to comment on interesting things you find experimenting with it.

Visual Studio 2010 Project:      

/// <summary>
/// Used so we can get MD5Hash and get progress on calculation
/// </summary>
/// <param name="streamIn"></param>
/// <returns></returns>
public static string GetSha512Buffered(Stream streamIn)
{
    Process process = Process.GetCurrentProcess();
    const int bufferSizeForMd5Hash = 1024*1024*8; // 8MB
    Console.WriteLine("-----------------------------------");
    Console.WriteLine("Memory Buffer Size {0} MB", bufferSizeForMd5Hash/1000);
    Console.WriteLine("-----------------------------------");
    string hashString;
    using (var md5Prov = new SHA256Managed())
    {
        int readCount;
        long bytesTransfered = 0;
        var buffer = new byte[bufferSizeForMd5Hash];
        while ((readCount = streamIn.Read(buffer, 0, buffer.Length)) != 0)
        {
            // Need to figure out if this is final block
            if (bytesTransfered + readCount == streamIn.Length)
            {
                md5Prov.TransformFinalBlock(buffer, 0, readCount);
            }
            else
            {
                md5Prov.TransformBlock(buffer, 0, bufferSizeForMd5Hash, buffer, 0);
            }
            bytesTransfered += readCount;
            Console.WriteLine("GetSha512Buffered:{0}MB/{1}MB.   Memory Used: {2}MB",
                                bytesTransfered/1000000,
                                streamIn.Length/1000000,
                                process.PrivateMemorySize64/1000000);
        }
        hashString = BitConverter.ToString(md5Prov.Hash).Replace("-", String.Empty);
        md5Prov.Clear();
    }
    return hashString;
}

I’m building a file synchronization with the cloud application and I want to store a checksum with the file so that I can verify later that the file is what I think it is. I’m not a crypto guy so I searched around the internet for a solution.   I found lots of examples and settled on this one that uses SHA256 for the job.  I also found some comments saying that it would be more efficient to wrap it in a BufferedStream rather than processing the entire file at once.

Example Links: http://efreedom.com/Question/1-1345851/MD5-File-Processing ; http://stackoverflow.com/questions/1177607/what-is-the-fastest-way-to-create-a-checksum-for-large-files-in-c/1177744#1177744

My intention for this post was to show how much more efficient it would be to use BufferedStream, however my results don’t show that.  I’m guessing that somehow, the efficiency is happening under the covers in a place I don’t see.

If anyone knows this space well, please feel free to comment and suggest a better method.  I’m publishing my source below for the test and my surprisingly similar results whether I used buffering or not.

Looking forward to the responses.

using System;
using System.Diagnostics;
using System.IO;
using System.Security.Cryptography;

namespace ConsoleApplicationMD5test
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            //const string fileName = @"g:\tempjunk\BlobSyncClient.zip";
            const string fileName = 
            @"g:\msdn\en_visual_studio_2008_service_pack_1_x86_dvd_x15-12962.iso";

            var stopwatch1 = new Stopwatch();
            stopwatch1.Start();
            string str1 = ""; // GetChecksum(fileName);
            stopwatch1.Stop();

            var stopwatch2 = new Stopwatch();
            stopwatch2.Start();
            var fileStream = new FileStream(fileName, FileMode.OpenOrCreate,
                FileAccess.Read);
            string str2 = GetChecksumBuffered(fileStream);
            stopwatch2.Stop();

            Console.WriteLine(str1 + " " + stopwatch1.ElapsedMilliseconds);
            Console.WriteLine(str2 + " " + stopwatch2.ElapsedMilliseconds);

            Console.ReadLine();
        }

        private static string GetChecksum(string file)
        {
            using (FileStream stream = File.OpenRead(file))
            {
                var sha = new SHA256Managed();
                byte[] checksum = sha.ComputeHash(stream);
                return BitConverter.ToString(checksum).Replace("-", String.Empty);
            }
        }

        private static string GetChecksumBuffered(Stream stream)
        {
            using (var bufferedStream = new BufferedStream(stream, 1024 * 32))
            {
                var sha = new SHA256Managed();
                byte[] checksum = sha.ComputeHash(bufferedStream);
                return BitConverter.ToString(checksum).Replace("-", String.Empty);
            }
        }
    }
}

Just so we now have an example, I’ve put together a simple windows form app that let’s you set a few variables in your app.config to point at your azure storage and container, let you view your app as a tree as well as see the code how it can be done.  I have not commented the code much, just thought it would be good to get it out there.  The running application shows you the data as follows.

So, for the details, I’m pasting below the meet of the code.  Basically, it does what you would expect in terms of iterating through the directories recursively to build the list.  Again, just set your parameters in your app.config as follows:

and you can run it for yourself and see how it goes.

I’m pasting some code below, as well as including the project source code.  The main reason I’m pasting the code is so that the search engines can find it better.  It’s really just easier to download the project and load it into Visual Studio 2010 and run it yourself.

Good Luck, and hope this helps!

That is, if you have code like this:

private void backgroundWorker1_DoWork(object sender, DoWorkEventArgs e)
{
    backgroundWorker1.ReportProgress(0,"starting...");
    for (int i = 0; i < 10;i++ )
    {
        if (i > 5)
        {
            throw new ApplicationException("any error here");
        }
        backgroundWorker1.ReportProgress(i, "working...");
    }
}

and you run it in the debugger, it will not stop by default when the exception is thrown (like it would in single threaded code).

So, I recommend doing the following and setting a break point to see the error.

Another thing you can do is bump up where debug’s stops uisng the dialog:

goes to:

Then, you will get an error you expect.

HTH’s!

<appSettings>
  <add key="AccountStorageURL" value="http://testit.blob.core.windows.net/"  />

public string GetStorageURLOld(string username, string password)
{
   var accountStorageURL = string.Empty;
   if (ConfigurationManager.AppSettings["AccountStorageURL"] != null)
   {
       accountStorageURL = ConfigurationManager.AppSettings.Get("AccountStorageURL");
   }
   return accountStorageURL;
}

I should have realized a long time ago that this code is equivalent to:

public string GetStorageURL(string username, string password)
{
  return ConfigurationManager.AppSettings["AccountStorageURL"] ?? string.Empty;
}

So, here is an example class that does that.

public class DbProgressReport
{
    public string ScopeNameProgress { get; set; }
    public string SourceOrDestination { get; set; }
    public string TableName { get; set; }
    public int TotalRecords { get; set; }
    public int ChangesApplied { get; set; }
    public int ChangesFailed { get; set; }
    public int ChangesPending { get; set; }
    public int Deletes { get; set; }
    public int Inserts { get; set; }
    public int TotalChanges { get; set; }
    public int Updates { get; set; }
    public DateTime LastChangeDate { get; set; }

    public override string ToString()
    {
        string readableDbProgressReport =
            string.Format(
                "{0} ScopeName {1},Table {2},TotalRecords {3}"
                SourceOrDestination, ScopeNameProgress, TableName);

        return readableDbProgressReport;
    }
    ...

Now, all you have to do when accessing this class is use the ToString property.

That is:

DbProgressReport dbProgressReport = 
  new DbProgressReport("source","scopename","tablename");
string str = dbProgressReport.ToString();

and the output will be:

source ScopeName scopename,  Table tablename

Simple as that!

Hope this helps.

So, say you have code like this:

var sessionAttendeeOds = 
  new SessionAttendeeODS();
listSessionAttendees = 
  sessionAttendeeOds.GetByUsername(Context.User.Identity.Name);

When I chose “InlineTemp”, the code changes to the following, make it easier to read.

listSessionAttendees = 
                new SessionAttendeeODS().GetByUsername(Context.User.Identity.Name);

Again, this is not a huge deal, but small fixes like this will add up over time and make my code cleaner and more readable.  CodeRush makes it so easy that I’m sure I’ll do this kind of thing more and more.

HTH’s.

In my original code, I had the following code that called the method GetLoadResultsFromInterval.

List<LoadResult> loadResultsForInterval =
    GetLoadResultsForInterval(rec.TimePoint,
                              rec.IntervalLengthInMs);

Then, I simply added one more parameter (loadFtpArrivalTimeDictionary) and ReSharper prompted me with the following:

If I choose the first “Add Parameter, I get what I would expect, which is just the method with an extra parameter.

/// </summary>
/// <param name="timePoint"></param>
/// <param name="intervalLengthInMs"></param>
/// <param name="loadFtpArrivalTimeDictionary"></param>
/// <returns></returns>
private static List<LoadResult> 
   GetLoadResultsForInterval(DateTime timePoint, 
   int intervalLengthInMs, 
   Dictionary<string, DateTime> loadFtpArrivalTimeDictionary)
{
   throw new NotImplementedException();
}

It even adds the param xml definition automatically.

However, I’m not quite as pleased with the second refactoring, that is, “Create Overload”.  It does almost everything I would expect, but does not implement the method, which I think it could.  Or, at least it could give it a try.  What it does implement is the following:

I’m actually including a picture of it rather than the actual code because it created an error.  It should have said DateTime instead of loadFtpArrivalTimeDictionary.  Two other things of interest.

First, it did not add xml comments (which is probably because of the syntax error), and second, I would have expected it to have called the overloaded method as follows instead of what it did.

private static List<LoadResult> GetLoadResultsForInterval(DateTime timePoint,
    int intervalLengthInMs, 
    Dictionary<string, DateTime> loadFtpArrivalTimeDictionary)
{
    // - put additional code here
    return GetLoadResultsForInterval(timePoint, intervalLengthInMs);
}

I began this post to brag about yet another awesome ReSharper refactoring because I really just wanted to add a parameter to an existing method call and it worked perfect.  It was not until I decided to show the other refactoring for this blog post did I run into the problem. 

I’m going to post this to the refactor dev’s and see if they can let me know where either I’ve gone astray, or there is a problem in their code.  At any rate, I’m running this on vs2008 and have not tried it on vs2010.

Hope this helps!

So, the issue this blog addresses is that if you form a URI with the parameter $top={anything}, your data will automatically be sorted.  The documentation for OData on top basically says that, but it could be clearer.  It says the following:

“If the data service URI contains a $top query option, but does not contain a $orderby option, then the Entries in the set needs to first be fully ordered by the data service.”

What actually happens is when you use the orderby clause, the data will be sorted 100% of the time for you, whether you do it or not.

I put a small example together that shows that.  I’ll briefly step through the parts of the code attached to this post that show that happening.  First, the results:

Here is the actual Visual Studio 2010 RC project you can run yourself:

Notice that when $top=5 is called, the data returned is sorted by the primary key.

Now, here are the details of the project that shows this happening.

Below is the complete code for the WcfDataService.svc.  Notice that the data is created in the SampleDataSource constructor and is created in non-key order.

using System;

using System.Collections.Generic;

using System.Data.Services;

using System.Data.Services.Common;

using System.Linq;

using System.ServiceModel.Web;

using System.Web;

namespace WebApp

{

    [EntityPropertyMappingAttribute(

        "Title", SyndicationItemProperty.Title,

        SyndicationTextContentKind.Plaintext, true)]

    [DataServiceKey("Id")]

    public class SampleInfo

    {

        // The DataServiceKey attribute is necessary unless you name 

        // this ID

        public int Id { get; set; } 

        public string Title { get; set; }

    }

    public class SampleDataSource

    {

        private readonly List<SampleInfo> _sampleInfoList;

        public SampleDataSource()

        {

            _sampleInfoList =

                new List<SampleInfo>()

                    {

                        new SampleInfo {Id = 3, Title = "3"},

                        new SampleInfo {Id = 4, Title = "4"},

                        new SampleInfo {Id = 1, Title = "1"},

                        new SampleInfo {Id = 5, Title = "5"},

                        new SampleInfo {Id = 2, Title = "2"}

                    };

        }

        public IQueryable<SampleInfo> SampleInfoData

        {

            get { return _sampleInfoList.AsQueryable(); }

        }

    }

    public class WcfDataService : DataService<SampleDataSource>

    {

        // This method is called only once to initialize service-wide policies.

        public static void InitializeService(DataServiceConfiguration config)

        {

            config.SetEntitySetAccessRule("*", EntitySetRights.AllRead);

            config.SetServiceOperationAccessRule("*", ServiceOperationRights.All);

            config.DataServiceBehavior.MaxProtocolVersion = 

                DataServiceProtocolVersion.V2;

        }

    }

}

<head runat="server">

    <title>Demonstration of OData Sort Based on Using $top</title>

     <script src="http://ajax.microsoft.com/ajax/jquery/jquery-1.3.2.min.js"    

        type="text/javascript"></script>  

    <script type="text/javascript" >

        // Display the data with and without $top with OData

        $(document).ready(function () {

            loadData("/WcfDataService.svc/SampleInfoData","#resultTableNoODataParms");

            loadData("/WcfDataService.svc/SampleInfoData?$top=5", "#resultTableWithODataParms");

        });

        // Called after DOM is ready to call service and display table data

        function loadData(urlString,divName) {

            var url = urlString; // "/WcfDataService.svc/SampleInfoData";

            $.ajax({

                type: "GET",

                url: url,

                contentType: "application/json; charset=utf-8",

                dataType: "json",

                success: function (msg) {

                    loadTable(msg.d,divName);

                }

            });

        }

        // display data at a given ID

        function loadTable(results,divName) {

            var table = '<table border=1><tbody><tr>';

            for (var post in results) {

                var row = '';

                row += '<td>' + results[post].Id + '</td>';

                table += row;

            }

            table += '</tr></tbody></table>';

            $(divName).html(table);

        }

    </script>

</head>

<body>

    <form id="form1" runat="server">

    <div>

    <h2>/WcfDataService.svc/SampleInfoData</h2>

    <div id="resultTableNoODataParms" />

    </div>

    <hr />

     <div>

    <h2>/WcfDataService.svc/SampleInfoData?$top=5</h2>

    <div id="resultTableWithODataParms" />

    </div>

    </form>

</body>

</html>

As you can see, adding the $top=5 causes the data to be sorted.

Hope this helps!

 List<string> strings = new List<string>
 
	{
		"a","b","c"

	};

To something that looks like:

“a:b:c:”

Here is the trick using the foreach syntax:

 var stringBuilder = new StringBuilder();
 strings.ForEach(a=>stringBuilder.AppendFormat("{0}:",a));

This article is very short.  In the actual presentation, there was not much time to talk about this so a brief overview was done.  Basically, it’s all standard WCF stuff.  The idea is that the Silverlight Business Template adds logging in and supports Authentication just like an asp.net application does.  It uses the DomainDataSource to do the bridging between the silverlight clientside app, and the web application.  Authorization is built into the login module also so that you can assign attributes to your domainservice classes to restrict access.

That is, if you only want people who are logged in able to access one of your methods, you simply tag it as follows with the [RequiresAuthentication] attribute.

If you only want a certain logged in user role to be able to access a certain method, you tag it as follows

This of course solve the server side only.  On the client side, you need to call methods that are exposed through the domain data.  http://www.silverlightshow.net/items/Silverlight-3-with-Ria-Authentication-service.aspx discusses much of this on the client side.