Background And Conclusion
In my very last post I talked about how using SHA256 seemed to not be affected by whether you use Buffered or Not Buffered streams. An astute reader (Samuel Jack) referenced an article by one of my favorite ex MIcrosoft employees (Brad Abrams) saying that almost all the .Net streams have buffering built in.
Well, I think SHA256Managed does not. One of the challenges I’ve been facing lately with my current project is to provide feedback while doing all kinds of byte piping (stream stuff). While figuring out how to do this, I inadvertently figured out that buffering makes a huge difference. Briefly, let me show my results first, then talk about the code that went into it.
Notice that when I use a 64MB Buffer, the memory used for the process is 77MB and when I use an 8MB buffer, the memory used is 17MB. Clearly the buffer allocated matters. Just for information, I’ve also included the code that does not break the SHA256 hash into blocks, and it has the same results based on the size of the Memory Buffer declared. That is, small buffer, small use, big buffer, big use.
The Code
The code that I wrote to do this basically does a nice job of splitting up the processing so that you can log progress as it runs. I’m attaching the Visual Studio 2010 project here as well as pasting the relevant source code. It’s a good base to experiment with. Feel free to comment on interesting things you find experimenting with it.
Visual Studio 2010 Project:
/// <summary>
/// Used so we can get MD5Hash and get progress on calculation
/// </summary>
/// <param name="streamIn"></param>
/// <returns></returns>
public static string GetSha512Buffered(Stream streamIn)
{
Process process = Process.GetCurrentProcess();
const int bufferSizeForMd5Hash = 1024*1024*8; // 8MB
Console.WriteLine("-----------------------------------");
Console.WriteLine("Memory Buffer Size {0} MB", bufferSizeForMd5Hash/1000);
Console.WriteLine("-----------------------------------");
string hashString;
using (var md5Prov = new SHA256Managed())
{
int readCount;
long bytesTransfered = 0;
var buffer = new byte[bufferSizeForMd5Hash];
while ((readCount = streamIn.Read(buffer, 0, buffer.Length)) != 0)
{
// Need to figure out if this is final block
if (bytesTransfered + readCount == streamIn.Length)
{
md5Prov.TransformFinalBlock(buffer, 0, readCount);
}
else
{
md5Prov.TransformBlock(buffer, 0, bufferSizeForMd5Hash, buffer, 0);
}
bytesTransfered += readCount;
Console.WriteLine("GetSha512Buffered:{0}MB/{1}MB. Memory Used: {2}MB",
bytesTransfered/1000000,
streamIn.Length/1000000,
process.PrivateMemorySize64/1000000);
}
hashString = BitConverter.ToString(md5Prov.Hash).Replace("-", String.Empty);
md5Prov.Clear();
}
return hashString;
}