Using SHA256Managed to Generate SHA256 Hash with Microsoft .Net Cryptography Namespace / Memory Usage Implications

 

Background And Conclusion

In my very last post I talked about how using SHA256 seemed to not be affected by whether you use Buffered or Not Buffered streams.  An astute reader (Samuel Jack) referenced an article by one of my favorite ex MIcrosoft employees (Brad Abrams) saying that almost all the .Net streams have buffering built in.

Well, I think SHA256Managed does not.  One of the challenges I’ve been facing lately with my current project is to provide feedback while doing all kinds of byte piping (stream stuff).  While figuring out how to do this, I inadvertently figured out that buffering makes a huge difference.  Briefly, let me show my results first, then talk about the code that went into it.

Notice that when I use a 64MB Buffer, the memory used for the process is 77MB and when I use an 8MB buffer, the memory used is 17MB.  Clearly the buffer allocated matters.  Just for information, I’ve also included the code that does not break the SHA256 hash into blocks, and it has the same results based on the size of the Memory Buffer declared.  That is, small buffer, small use, big buffer, big use.

image

image

The Code

The code that I wrote to do this basically does a nice job of splitting up the processing so that you can log progress as it runs. I’m attaching the Visual Studio 2010 project here as well as pasting the relevant source code.  It’s a good base to experiment with.  Feel free to comment on interesting things you find experimenting with it.

Visual Studio 2010 Project:      

 

/// <summary>
/// Used so we can get MD5Hash and get progress on calculation
/// </summary>
/// <param name="streamIn"></param>
/// <returns></returns>
public static string GetSha512Buffered(Stream streamIn)
{
Process process = Process.GetCurrentProcess();
const int bufferSizeForMd5Hash = 1024*1024*8; // 8MB
Console.WriteLine("-----------------------------------");
Console.WriteLine("Memory Buffer Size {0} MB", bufferSizeForMd5Hash/1000);
Console.WriteLine("-----------------------------------");
string hashString;
using (var md5Prov = new SHA256Managed())
{
int readCount;
long bytesTransfered = 0;
var buffer = new byte[bufferSizeForMd5Hash];
while ((readCount = streamIn.Read(buffer, 0, buffer.Length)) != 0)
{
// Need to figure out if this is final block
if (bytesTransfered + readCount == streamIn.Length)
{
md5Prov.TransformFinalBlock(buffer, 0, readCount);
}
else
{
md5Prov.TransformBlock(buffer, 0, bufferSizeForMd5Hash, buffer, 0);
}
bytesTransfered += readCount;
Console.WriteLine("GetSha512Buffered:{0}MB/{1}MB. Memory Used: {2}MB",
bytesTransfered/1000000,
streamIn.Length/1000000,
process.PrivateMemorySize64/1000000);
}
hashString = BitConverter.ToString(md5Prov.Hash).Replace("-", String.Empty);
md5Prov.Clear();
}
return hashString;
}

About Peter Kellner

Peter is a software professional specializing in mobile and web technologies. He has also been a Microsoft MVP for the past 7 years. To read more about Peter Kellner and his experience click here. For information about how Peter Kellner might be able to help you with your project click here.

Follow me:


Trackbacks

  1. […] This post was mentioned on Twitter by pkellner, Connection Road. Connection Road said: RT @pkellner: Just Posted On How to Generate SHA256 Hash In Chunks With Progress. (Tested Thoroughly!) http://bit.ly/eTqgqY […]

  2. […] * I just added another post on similar topic and added more details on performance and a case where buffered memory usage really did matter:  http://peterkellner.net/2010/12/03/using-sha256managed-to-generate-sha256-hash/ […]

Follow

Get every new post delivered to your Inbox

Join other followers: