In May of 2009 I discovered some significant performance problems that I blogged about.  In summary, I had tracked it down to the issue of LINQ2SQ having to create a full expression tree on every instantiation of a LINQ2SQL query.  I’m not a compiler write kind of guy but do respect the complexity of that and doing things like building expression trees, but still this really sucked.  Using the compile syntax in LINQ2SQL is very awkward, and IMHO takes all the fun out of using LINQ2SQL.  If you don’t remember my post, here is the graph showing the evil happening.

So, what is a Microsoft MVP to do?  I complained to everyone at Microsoft I knew.  I made this my mission for about a year.  I was told time after time that the problem was hugely complicated and no one was working on making it any faster.  I just could not believe it because I felt this was such a barrier to success to LINQ2SQL type technology (now Entity Framework really) that Microsoft just could not ignore my findings.  I was actually surprised that I was the only one screaming about this.

Well, quietly, Microsoft has announced that in .Net Framework 4.5, Entity Framework will include “Auto-Compiled LINQ Queries”.  This is awesome!   When it comes out, I’ll be testing and it giving feedback.

For now, I’m a very happy camper.

The technology pairing I’ve chosen is Microsoft’s .Net platform with MVC on the server, and ExtJS on the client.  Though it’s possible to still use ExtJS with standard html/aspx pages, I’ve found the best combination is to use 100% JavaScript on the client (ExtJS) and have all the server side technology be 100% service based.  I’ve used LINQ2SQL extensively as well as Entity Framework in the latest Visual Studio 2010 release.

The learning curve was quite steep to actually be able to efficiently build highly flexible, highly scalable applications using these technologies, but now that I know it, I wouldn’t have it any other way.

I’m considering putting together a series of 4 Day Classes around the country (or even world) that would basically teach people the methods and patterns I’ve learned and essentially leap frog a development team into being able to quickly do what it has taken me years to figure out.  I’ve been fortunate enough to know the top 1% instructors and I’m sure with the right incentive, can get them to join me in both putting together these classes as well as teaching them.

So, my question is:

I really appreciate your feedback.  If I go down this path, it will be a lot of work and it would be nice to know if people are really interested.  I know for myself, I’ would have gone to something like this 2 years ago and it would have saved me a lot of time.

Thanks for your feedback.

So, in my C# code, here is what you need to convert to the Linq.Binary:

Linq.Binary carrierMatrix =
   new ASCIIEncoding().GetBytes(myString);

And, to go back from Linq.Binary:

string carrierMatrixString = Encoding.ASCII.GetString(carrierMatrix.ToArray());

Hope this help!

It occurred to me I could do it in LINQ, then grab out the SQL it generated.  The tool?  LINQPad of course.  So, here is what I typed into LINQPad:

(from data in Attendees
let dateNoTime = 
    new DateTime
     (data.CreationDate.Value.Year,
      data.CreationDate.Value.Month,
      data.CreationDate.Value.Day)
orderby dateNoTime
group data by dateNoTime into g
where g.Count() > 0
select new 
    {
    g.Key,
    Number_Registered = g.Count() 
    }).OrderByDescending(f=>f.Key)

Then, After pasting that into LINQPad, choosing the SQL tab on the bottom, it gives me this:

SELECT [t2].[value3] AS [Key], [t2].[value2] AS [Number_Registered]
FROM (
SELECT COUNT(*) AS [value], COUNT(*) AS [value2], [t1].[value] 
          AS [value3]
FROM (
    SELECT CONVERT
         (DATETIME, CONVERT(NCHAR(2), DATEPART(Month, [t0].[CreationDate])) 
           + ('/' + (CONVERT(NCHAR(2), DATEPART(Day, [t0].[CreationDate])) +
              ('/' + CONVERT(NCHAR(4), DATEPART(Year, [t0].[CreationDate]))))), 101) 
                    AS [value]
        FROM [Attendees] AS [t0]
        ) AS [t1]
    GROUP BY [t1].[value]
    ) AS [t2]
WHERE [t2].[value] > @p0
ORDER BY [t2].[value3] DESC

And, the nice screen shot of LINQPad

And, it works perfect!  Even the TSql generated looks pretty nice.  Group By, Convert, DatePart and everything. Somehow, I feel like I’ve cheated, but I’m good with that in this case.

Hope this helps!

So, How to do this?  Well, first, here is the answer:

   1:         /// <summary>

   2:          /// Returns a list of days that have a loadCount > loadCount

   3:          /// </summary>

   4:          /// <param name="days"></param>

   5:          /// <param name="loadCount"></param>

   6:          /// <returns></returns>

   7:          public List<DateTime> GetLastDaysWithLoadsGreaterThan(int days,int loadCount)

   8:          {

   9:              var dateGroups = (from load in _meta.Load

  10:                                orderby load.DateCreated

  11:                                group load by load.DateCreated

  12:                                into g

  13:                                    where g.Count() > loadCount

  14:                                    select new

  15:                                               {

  16:                                                   g.Key,

  17:                                                   cnt = g.Count()

  18:                                               }).Take(days).OrderByDescending(f => f.Key);

  19:              

  20:              var listDates = dateGroups.Select(a => a.Key).ToList();

  21:              

  22:              return listDates;

  23:          }

And now, a short explanation.

First, the obvious I thought would have been if I did orderby load.DateCreated descending, that I would not have had to reverse the list at the end.  It did not work and I’m not sure why. Luckily, I can sort the list at the very end which is a small penalty since my lists are not that large.

Notice the groupby and the where clause are doing the work.  The resulting anonymous method solves the problem of getting the data out.

Line 20 is one of my favorite tricks to avoid 4 or 5 lines of code.

And of course, the SQL is a lot simpler, but not type safe.

   1:  SELECT datecreated,

   2:         count(*)

   3:  FROM dbo.[Load]

   4:  group by datecreated

   5:  having count(*) > 200

   6:  order by datecreated desc

   7:    

    public partial class CodeGenTestQuery : QueryBase

    {

        public int? Id { get; set; }

        public List<int> Ids { get; set; }

        public bool? IsStarred { get; set; }

        ...

If I pass an empty list of int’s into Ids, I have a line of code that build the query as follows:

if (query.Ids != null) baseQuery = baseQuery.Where(a => query.Ids.Contains(a.Id));

This works fine as long as a pass into ids either null, or a list that contains more than one object. However, if I pass in an empty list like this:

new CodeGenTestQuery
    {
         Ids = new List<int>() { }, MyAge = 2
     }

Then, the result of the LINQ query is it creates a where clause that says something like this:

SELECT [t0].[Id],

       [t0].[IsStarred]

FROM [dbo].[CodeGenTest] AS [t0]

WHERE ([t0].[MyAge] = 2) AND

      (0 = 1)

Which means you will not get the MyAge == 2 record ever

So, I changed my c# as follows and now I get what I expect.

if (query.Ids != null)
            {
                if (query.Ids.Count > 0)
                {
                    baseQuery = baseQuery.Where(a => query.Ids.Contains(a.Id));
                }
}

I can see this kind of thinking on the LINQ2SQL team, but it’s a little insidious if you ask me.

At anyrate, I’m now following Jim’s Twitter feed and he recently mentioned he included all the examples from his book in LINQPAD.  I just downloaded them.  This is just too cool!  He wrote a blog post about it here:  http://www.thinqlinq.com/Default/LINQ-In-Action-Samples-available-in-LINQPad.aspx  Here is what it looks like in LINQPAD

Hope this helps!!!  It’s huge for me.

Here are some assumptions:

• My Notebook Computer Simulates a real world web server (Lenovo W500, Vista64, IIS7)
• All Web Pages do 7 LINQ2SQL executions (Heading speak at  VSLive in Vegas so 7 seemed right)
• All Web Requests Do Only LINQ2SQL, no images, css, JavaScript or anything else

OK, maybe my assumptions are not so good, but I believe overall I’m making a good point so I’m going to keep going rather than just say “take my word for it, trust me”.  So, I’m attaching the web project here that let’s you recreate all this so you can do your own tests.  I think you’ll need Visual Studio Team Suite to do these tests.  If you don’t have that, you can at least look at the code.  The guts are in LINQUtils.cs.  I’m going to show the relevant code below.

But before that, here is the Visual Studio 2008 Project Code if you want to try it yourself:

First, The Results

The code is very straight forward.  Below is the code for the LINQ2SQL Not Compiled Test:

  1: public string ProcessLinqQueryNotCompiled()


  2: {


  3:     if (meta == null)


  4:     {


  5:         meta = new DataClassesDataContext();


  6:     }


  7: 


  8:     Stopwatch stopWatch = new Stopwatch();


  9:     stopWatch.Start();


 10: 


 11:     int retCnt = 0;


 12:     for (int i = 0; i < iterations; i++)


 13:     {


 14:         const int companyTypeIdToFind = 1;


 15:         var q3 = from dataCo in meta.Companies


 16:                  where (from dataCt in meta.CompanyCompanyTypes


 17:                         where dataCt.CompanyTypeId == companyTypeIdToFind


 18:                         select dataCt.CompanyId).Contains(dataCo.Id)


 19:                  select dataCo;


 20: 


 21:         var newList = q3.ToList();


 22:         retCnt = newList.Count;


 23:     }


 24:     stopWatch.Stop();


 25:     double milliSecondsPerIteration = Convert.ToDouble(stopWatch.ElapsedMilliseconds)/Convert.ToDouble(iterations);


 26:     return string.Format("Return Cnt: {0} LINQ Executions: {1}      Milliseconds Per Iteration: {2}", retCnt, iterations,


 27:                          String.Format("{0:0.00}", milliSecondsPerIteration));


 28: }

And, the Compiled LINQ2SQL Code:

  1: public string ProcessLinqQueryCompiled()


  2: {


  3:     var stopWatch = new Stopwatch();


  4:     stopWatch.Start();


  5: 


  6:     int retCnt = 0;


  7:     for (int i = 0; i < iterations; i++)


  8:     {


  9:         const int companyTypeIdToFind = 1;


 10: 


 11:         if (meta == null)


 12:         {


 13:             meta = new DataClassesDataContext();


 14:         }


 15: 


 16:         if (_compiledQuery == null)


 17:         {


 18:             _compiledQuery =


 19:                 CompiledQuery.Compile((DataClassesDataContext metax, int myQuery) =>


 20:                                       (from dataCo in metax.Companies


 21:                                        where (from dataCt in metax.CompanyCompanyTypes


 22:                                               where dataCt.CompanyTypeId == companyTypeIdToFind


 23:                                               select dataCt.CompanyId).Contains(dataCo.Id)


 24:                                        select dataCo)


 25:                     );


 26:         }


 27: 


 28:         


 29:         var query =


 30:             (IOrderedQueryable<Company>)


 31:             _compiledQuery(meta, 1);


 32:         var newList = query.ToList();


 33:         retCnt = newList.Count;


 34:     }


 35:     stopWatch.Stop();


 36:     double milliSecondsPerIteration = Convert.ToDouble(stopWatch.ElapsedMilliseconds) / Convert.ToDouble(iterations);


 37:     return string.Format("Return Cnt: {0} LINQ Executions: {1}      Milliseconds Per Iteration: {2}", retCnt, iterations,


 38:                          String.Format("{0:0.00}", milliSecondsPerIteration));


 39: 


 40: }

Bottom line, You need to compile your LINQ2SQL!

Today, I discovered you can actually put c# right into the editor!  I’m attaching a screen shot of how it helped me solve a simple LINQ command that involved an integer list.  I reviewed it in a little more detail on a previous post here:  http://peterkellner.net/2008/09/20/linqpad-sqlserver-linq-interpreter-nutshell/

Hope you find it useful also!

Well, my fears are put aside.  Though LINQ is somewhat slower, for 100,000 iterations, .052 seconds is pretty good.  (compared with processing a single not to complicated LINQ2SQL statement for 100,000 iterations would take about 20,000 seconds or 333 hours.  Quite a difference to .052 seconds!

Here is the code I ran for my test.  Hope this help!

using System;

using System.Collections.Generic;

using System.Diagnostics;

using System.Linq;

using System.Text;

namespace LINQTimingInMemory

{

     class Program

     {

         static void Main(string[] args)

         {

             for (int i=0;i<2;i++)

             {

                 List<int> myList = new List<int>() {1,2,3,4,5,6,7,8,9,6,5,5,5,5,5,5};

                 Stopwatch stopWatch = new Stopwatch();

                 stopWatch.Start();

                 int iterations = 100000;

                 for (int t = 0; t < iterations; t++)

                 {

                     if (i == 0)

                     {

                         var ids = myList.Where(a => a > 5).ToList();

                     }

                     else

                     {

                         var ids = new List<int>();

                         foreach (var v in myList)

                         {

                             if (v > 5)

                             {

                                 ids.Add(v);

                             }

                         }

                     }

                 }

                 stopWatch.Stop();

                 int ti = Convert.ToInt32(stopWatch.ElapsedMilliseconds);

                 Console.WriteLine("time for " + i.ToString() + " " + ti/iterations + " " + ti);

                 stopWatch.Reset();

             }

         }

     }

 }

At my company, we have been running a workflow process that is taking 6 hours to complete.  This is a new project so we’ve had the good fortune to be able to use the latest Microsoft technologies.  We are a Microsoft shop, so that means Sql Server 2008, C#, ASP.NET, ExtJS, LINQ to SQL (hoping to move to Entity Framework someday) and others.  After spending all day running performance analysis tools and basically performing a full audit of the 6 hour process, we sadly concluded that our time was being eaten up by LINQ to SQL processing.  My experience has always been that anything you do on the compiled code side is usually overwhelmed by database access times, usually at least ten to one.  Well, I learned a lot yesterday.

The Problem

Since I already know the answer, I put together a very trivial problem to solve.  What I want to do is execute one simple SQL statement over and over.   I’m going to query just one table that actually has no records in it just to make sure the database really has nothing to do.  I’m going to make this connection on my blazingly fast Lenovo W500 with a SATA SSD running Sql Server 2008  locally.  I’m going to run that simple query first calling just ADO.NET (bare bones using best practices), Then using LINQ to SQL in the way I’ve always used it (and I would say 99% of the other developers out there), then I’m going to compile that LINQ to SQL and run it again.  Actually, I’m going to iterate 2000 times for each of the three conditions.

So, here is the simple SQL I’m executing:

SELECT 
  Id,
  FileName,
  FileSize,
  UploadDate,
  DownloadDate,
  ProcessDate
FROM
  LinqTest
WHERE 
  dbo.LinqTest.FileName = 'MyVal'

The Shocking Results

Or, for those graphically challenged (like search engines), Here is the actual data.

In English, this says that

• LINQ to SQL is  37 times slower than running a raw ADO.NET Sql query
• LINQ to SQL compiled 1.8 times slower than running a raw ADO.NET Sql query

I’ve read in several places that compiling your LINQ will help, but I have never heard anyone say how drastic the speed improvement can be.  For example, in one of my favorite books (LINQ in Action) by Fabrice Marguerie and others, he quotes on page 296  a blog post by Rico Mariani titled DLINQ (Linq to SQL Performance (Part 1) as saying using a compiled query offers nearly twice the performanced of a non-compiled query, and goes on to say that it brings the performance to within 93% of using a raw data reader.   Well, suffice it to say I never ran the test myself.  I could have lived with twice, but not 37 times.

Here is the Project File Source:

How to Compile you SQL (seems like a duh kind of thing)

It’s actually not very hard.  I’m attaching the Visual Studio 2008 project that I ran this test with to this post so you can run it for yourself as well as see an example of how to write the compiled LINQ code.  Below is the method that actually does the work.  I won’t include in this article the actual ADO.NET and LINQ to SQL not compiled code, but you can see those for yourself in the attached solution.

   1:  /// <summary>

   2:  /// This method compiles the LINQ to SQL query and 

   3:  /// then executes it the number of iterations passed

   4:  /// in.  

   5:  /// </summary>

   6:  /// <param name="iterations">Number of iterations</param>

   7:  /// <returns>time in seconds of execution</returns>

   8:  private static double 

   9:      TestDataAccessSpeedLinq2SqlCompiled(int iterations)

  10:  {

  11:      Func<DataClassesDataContext, string, IQueryable<LinqTest>> compiledQuery =

  12:          CompiledQuery.Compile((DataClassesDataContext meta,string fileNameForSearch) =>

  13:                                (from myData in meta.LinqTests

  14:                                 orderby myData.Id

  15:                                 where myData.FileName.Equals(fileNameForSearch)

  16:                                 select myData));

  17:   

  18:      var metaNew = new DataClassesDataContext();

  19:      DateTime startTime = DateTime.Now;

  20:      for (int i = 0; i < iterations; i++)

  21:      {

  22:          IOrderedQueryable<LinqTest> query = 

  23:              (IOrderedQueryable<LinqTest>) 

  24:              compiledQuery(metaNew,string.Format("abcde{0}", i));

  25:          List<LinqTest> newList = query.ToList();

  26:      }

  27:      return 

  28:          DateTime.Now.Subtract(startTime).Duration().TotalSeconds;

  29:  }

Essentially, line 11 is compiling the code into an instance variable called compileQuery.  Using this instance variable, you can now execute the  LINQ query in a compiled form while still passing in variable data such as fileNameToSearch.  Again, the important thing to note is that one line 24 compiledQuery is already compiled so the IOrderedQueryable result is obtained without having to recompile the LINQ statement.

Conclusions

So, from this, it seems that you should always compile your LINQ to SQL queries.  Well, that’s not quite true.  What I’m recommending is that if you have a reason to execute the same query over and over you should strongly consider compiling.  If for example, you are just making a LINQ to SQL call once, there is no benefit because you have to compile it anyway.  Call it ten times?  Well, you will have to decide for yourself.

Forewarned is forearmed!  good luck and hope this helps.