A TiVo Disk Space Viewer, It’s all about LINQ to XML!

Peter Kellner — Sat, 19 Jan 2008 02:16:40 +0000

Tivo Space Viewer Source Code VS2008 Project Complete

Abstract

Do you ever want to know exactly how much space each TV show on your TiVo is taking on your hard drive? Do you ever wonder how much space is consumed by high definition programs verses standard definition programs? For the answer, download the program associated with this article and you to can know those answers. Below area pictures of those graphs. Chances are though, if you are reading my blog, you really want to know about Microsoft technology, and specifically, this article is all about parsing the webservice data that comes from your TiVo. At the end of the article, there is a brief section on how to configure this program so if all you care about is running the program, skip all the LINQ stuff and just go to the Setup below. The exe and dll file necessary are attached there.

Almost lastly, the application produced uses Windows Forms Programming. This article makes no attempt to discuss any of the windows forms programming issues. The code provided in Forms is simply a wrapper to look at all the fun LINQ commands. In addition, the graphs are done with the Open Source Project Zedgraph.

And lastly, this work is heavily based on the TiVo gadget beautifully done by Microsoft’s Mike Swanson. I won’t spend much time talking about all the issues regarding setup and security. Mike has done a great job of that in his article. If you are running Windows Vista, I suggest you download his awesome sidebar gadget.

(This shows TiVo Disk Usage by Recording Type (In Progress, Suggestions, etc.)

(This compares High Definition verses Standard Definition by Recording Type)

(And a very simple of what is stored including size and title)

The Basics

Assuming you have a TiVo that exposes a web feed (works with TiVo Desktop) you’ve got a chance of this program working for you. I’m not at all an expert on TiVo, but I do have a series 3 unit and following the directions written by Mike Swanson to install his Vista Gadget, I got his program to work, and I use a similar mechanism to access the web feed. So, I suggest first thing to do is to download Mike’s Vista Sidebar Gadget and make that work.

Once you have that working, you can unzip the file attached to this article, build it with Visual Studio 2008, then enter your TiVo service ID and Media Access Key into the running application, press "Save Settings" and your graphs should appear in the tabs. Here is what the running program looks like:

The TiVo Web Service Interface

Without going into all the details about how the xml data is actually retrieved from the TiVo (you can checkout the code yourself), basically what happens is the TiVo provides to the application a full list of all the shows it has stored in XML. Each show is wrapped in an element. Below is one item for your reference.

   1:  <Item>

   2:  <Details>

   3:  <ContentType>video/x-tivo-raw-ttsContentType>

   4:  <SourceFormat>video/x-tivo-raw-ttsSourceFormat>

   5:  <Title>Cold CaseTitle>

   6:  <SourceSize>8105492480SourceSize>

   7:  <Duration>3601000Duration>

   8:  <CaptureDate>0x4781B1CECaptureDate>

   9:  <EpisodeTitle>SabotageEpisodeTitle>

  10:  <Description>The team searches for a serial ...Description>

  11:  <SourceChannel>705SourceChannel>

  12:  <SourceStation>KPIXDTSourceStation>

  13:  <HighDefinition>YesHighDefinition>

  14:  <ProgramId>EP5927270107ProgramId>

  15:  <SeriesId>SH592727SeriesId>

  16:  <ByteOffset>0ByteOffset>

  17:  Details>

  18:  <Links>

  19:  <Content>

  20:  <Url>http://652-0001-5555-4444:80/downloa.T...Url>

  21:  <ContentType>video/x-tivo-raw-ttsContentType>

  22:  Content>

  23:  <CustomIcon>

  24:  <Url>urn:tivo:image:expired-recordingUrl>

  25:  <ContentType>image/*ContentType>

  26:  <AcceptsParams>NoAcceptsParams>

  27:  CustomIcon>

  28:  <TiVoVideoDetails>

  29:  <Url>https://4444-0001-6565-d...146267Url>

  30:  <ContentType>text/xmlContentType>

  31:  <AcceptsParams>NoAcceptsParams>

  32:  TiVoVideoDetails>

  33:  Links>

  34:  Item>

  35:  <Item>

The data we are actually interested in is Duration (line 7), HighDefinition (line 13), Url (line 20) and Title (line 5). The entire purpose of using LINQ to XML is to extract this information from the repeating tag and have available for plotting. The next several sections explains in details how XML is used to extract this data.

Create a Generic List of Objects for Further Processing

To figure out how much space is allocated to each recording type, we essentially need to do a sort and sum of type of recording (called url in our xml). The url tag is actually buried under the and tags (see lines 18-20 above) so this is a little tricky. I’ll outline the steps we need to follow.

The first thing we do is create a simple type called TivoShowInfo. This type (really a class definition) is defined as follows:

   1:  class TivoShowInfo

   2:      {

   3:         ///

   4:         /// Initializes a new instance of the

   5:         /// TivoShowInfo class.

   6:         ///

   7:         ///

   8:         ///

   9:         ///

  10:         ///

  11:         public TivoShowInfo(string title, int duration,

  12:         Int64 numberBytes, string showType,

  13:         bool highDefinition)

  14:         {

  15:         _Title = title;

  16:         _Duration = duration;

  17:         _NumberBytes = numberBytes;

  18:         _ShowType = showType;

  19:         _HighDefinition = highDefinition;

  20:  } ...

The reason we do this will become apparent in the next section as we build the list. So, let’s assume we have an XML file represented by a string (xmlString). First thing we have to do is create an XNamespace declaration to help us process the xml. The reason we need this is because the namespace is actually defined in the xml file itself so we need to reference it in our queries.

Here is what the top of the XML looks like that comes from the TiVo:

   1:  xml version="1.0" encoding="utf-8"?>

   2:  <TiVoContainer

   3:  xmlns="http://www.tivo.com/developer/calypso-protocol-1.6/">

   4:  <Details>

Notice on line 3 the xmlns tag. In our code, we can reference it with the c# code:

   1:  XNamespace ns1 = http://www.tivo.com/developer/calypso-protocol-1.6/;

Now we need to create a LINQ query to process the elements.

Here is what that Query looks like:

1: var query =

2:   from item in root.Descendants(ns1 + "Item")

3:   let details = item.Element(ns1 + "Details")

4:   let links = item.Element(ns1 + "Links")

5:   where links.HasElements ==

6:   true && links.Elements("CustomIcon") != null

7: select new

8: {

9:   Title = (String)details.Element(ns1 + "Title"),

10:   Duration = (Int32)details.Element(ns1 + "Duration"),

11:   Bytes = (Int64)details.Element(ns1 + "SourceSize"),

12:   HighDefinition =

13: (String)details.Element(ns1 + "HighDefinition"),

14: IconUrl =

15: (XElement)links.Element(ns1 + "CustomIcon")

16: };

What this says is create an anonymous type called query (line 1). This object query is going to process the descendents of the Item tag. That is, what is between the and tags. To do that, line 2 says "from item in root.Descendants(..)". this means item is basically a variable and we are going to iterate through all the root.Descendants(..) of "Item". The rest is just syntax. The goal is to create the "var query" which is really an enumeration of XML item elements. Our next step will be to use a simple foreach loop to run through that iteration. Before that, however, let’s look at the rest of the LINQ above and try to understand what it is doing.

Line 3 and line 4 are let statements that assign the

and to temporary variables. Line 5 says only include items where the element actually exists. This is necessary because there are some extra elements we want to skip over in the xm file.

Finally, lines 7 through 16 are what creates the anonymous type which we can reference in our foreach statement when we iterate through query (which if you remember is enumerable). Basically, we are creating an anonymous type here that has the attributes: Title;Duration;Bytes;HighDefinition and IconUrl. These are dynamically created attributes of all different types (notice the casting to String,Int32,Int64 and XElement).

Finally, we need to actually process this enumeration called query. The obvious way to process any enumeration is foreach. So, here goes:

   1:  foreach (var item in query)

   2:  {

   3:      if (item.IconUrl != null)

   4:      {

   5:          foreach (XElement item1 in item.IconUrl.Nodes())

   6:          {

   7:              if (item1.Name.LocalName.Equals("Url"))

   8:              {

   9:                  tivoShowInfoList.Add(

  10:                  new TivoShowInfo(

  11:                      item.Title,

  12:                      item.Duration,

  13:                      item.Bytes,

  14:                      item1.Value,

  15:                      item.HighDefinition.Equals("Yes")

  16:                      ? true : false)

  17:                  );

  18:              }

  19:          }

  20:      }

  21:  }

What we are doing here is iterating through the query and pulling apart the anonymous type attributes. Remembering that IconUrl was of type XElement, we are checking to see if that element is not null. If it is not null, we using another LINQ statement to iterate through it’s nodes, and if we find the node Url, we simply add to our generic collection of TivoShowInfo elements. That’s not a LINQ thing, but then we can operate on that generic collection with LINQ again to make up our graphs. The next several sections use this generic collection we just created.

LINQ for Disk Usage By Recording Type

Getting space by category uses the following LINQ to generate a generic list of KeyValuePairs. That is, we want a list that looks like: ShowCategory1;Percentage1 / ShowCategory2;Percentage2, etc. Here is the code that creates that followed by an explanation.

1:  public static List<KeyValuePair<string, Int64>> 2:   GetSpaceByCategory(string xmlString)3:  {4:   List<TivoShowInfo> tivoShowInfoList = 5:   GetTivoShowList(xmlString);

6:   Int64 totalBytes = 7:   (from t in tivoShowInfoList select t.NumberBytes).Sum();

8:   var categories =9:   from s in tivoShowInfoList10:   group s by s.ShowType into g11:   select12:          new13:          {14:   ShowType1 = g.Key,15:   TotalBytes = g.Sum(s => s.NumberBytes)16:   };

17:   List<KeyValuePair<string, Int64>> kvpList = 18:   new List<KeyValuePair<string, Int64>>();19:   foreach (var c in categories)20:   {21:   Int64 val = (Int64)c.TotalBytes;

22:   KeyValuePair<string, Int64> kvp = 23:   new KeyValuePair<string, Int64>(c.ShowType1, val);24:   kvpList.Add(kvp);25:   }26:   return kvpList;27:  }

What is happening here is we are using the Generic collection (List) of TivoShowInfo to act as our LINQ source. We get this by calling the method GetTivoShowList on line 4 above.

The first thing we do (which is not really related to the big sort) is to simply get the total bytes stored on the TiVo. We do this by simply iterating over the generic List (tivoShowInfoList) with the command on line 7 (from t in tivoShowInfoList). The select creates an anonymous return type of t.NumberBytes. Essentially, behind the scenes, an anonyomous delegate is created that takes in t (the tivoShowInfoList list item) and returns an integer (t.NumberBytes) which is the size of the show in bytes. Then, by putting that all in ()’s creates an enumerated type. Since the enumerated type has an extension method associated with it called Sum(), we can call this and the return value will essentially Sum the values of NumberBytes for each entry.

Next, we need to perform a GroupBy function to get separate totals for each ShowType. to do this, we again iterate over the generic List in line 8. This time, we add line 10 which creates a group s by the show type. The anonymous delegate that is created has the key of the group by and also the number of bytes. The syntax s => s.NumberBytes on line 15 simply means create a delegate that takes in the parameter s (which is showtype group) and return the number of bytes of the item. Sum (in line 15) of course totals those bytes.

Finally, we iterate through the collection on lines 19 to create the KeyValuePair we will be returning.

LINQ for Disk Usage By High Def Verses Low Def

The method GraphBarByHighDef is used to get this data. Below is the LINQ code used for this.

1:  var query = (from tivoList in listTivoShowInfo2:   select new3:               {4:   tivoList.ShowType5:   }).Distinct();

6:  List<string> showTypeStringList = new List<string>();7:  foreach (var val in query)8:  {9:   //string str = ShortenContentType(val.ShowType);10:      showTypeStringList.Add(val.ShowType);11:  }

12:  string[] contentTypeList = new string[showTypeStringList.Count];13:  double[] hdSize = new double[showTypeStringList.Count];14:  double[] sdSize = new double[showTypeStringList.Count];

15:  int cnt = 0;16:  foreach (string s in showTypeStringList)17:  {18:   contentTypeList[cnt] = ShortenContentType(s);19:   var totalHDList = from tivoList in listTivoShowInfo20:   where tivoList.ShowType.Equals(s) &&21:   tivoList.HighDefinition22:   select new23:                        {24:   Bytes = tivoList.NumberBytes25:   };

26:   hdSize[cnt] = 27:   Convert.ToInt32(Convert.ToInt64(totalHDList.Sum(p => p.Bytes))28:   / INT_Constant_Billion);29:   var totalSDList = from tivoList in listTivoShowInfo30:   where tivoList.ShowType.Equals(s) && 31:   !tivoList.HighDefinition32:   select new33:                        {34:   Bytes = tivoList.NumberBytes35:   };

36:   sdSize[cnt] = 37:   Convert.ToInt32(Convert.ToInt64(totalSDList.Sum(p => p.Bytes))38:   / INT_Constant_Billion);39:   cnt++;40:  }

The strategy here is to first generate a unique list of ShowTypes. Just like for the sum operation in the previous section, we want to create an enumeration of ShowType’s and from that, we want to call the Distinct (line 5) function to get the unique list. Again, the enumeration created is basically a full list of just ShowType’s for all the TiVo listings, and Distinct just gives us those that are unique. We need this because we want to create one bar for each ShowType. Our foreach (line 7 to 10) simply creates a generic List of ShowTypes we can iterate through (which we do on line 16).

Our objective is to create a string array of ShowType’s, a corresponding double array of High Def Sum’s and a double array of standard def Sums’s. We do this one ShowType at a time (hence the foreach loop on line 16. Line 18 extracts a nice short name of the ShowType (that is, urn:tivo:image:expired-recording turns into Expired). Then, line 19 simply says iterate through the entire collection of shows and only look at show’s of the type we are interested in (line 20) and return just the NumberBytes of that show (line 24). Line 26-27 simply totals that collection and stores the value in the array of doubles for High Definition. Lines 29-38 does the same thing for standard definition. Notice on lines 27 and 37, the expression p=>p.Bytes is the new syntax for create an anonymous delegate that takes p in as a parameter and returns an integer p.Bytes.

LINQ for Simple Listing of Shows

Creating a listing is the simplest thing of all to do with link. Here is the LINQ code.

1:  /// 2:  /// Generate a simple listing of all shows with space3:  /// 
4:  private void ShowTivoListings(string xmlString)5:  {6:   List<TivoShowInfo> listTivoShowInfo = 7:   TivoUtils.GetTivoShowList(xmlString);8:   var query = from tivoList in listTivoShowInfo9:   orderby tivoList.Title10:   select tivoList;

11:   foreach (var item in query)12:   {13:   double gigabytes = 14:   Convert.ToDouble(15:   Convert.ToDouble16:   (item.NumberBytes / 17:   Convert.ToDouble(INT_Constant_Billion)));

18:   string displayString = 19:   String.Format("Size: {0:n} Status: {1} Title: {2}",20:   gigabytes,21:   ShortenContentType(item.ShowType),22:   item.Title);

23:   listBoxTivoListings.Items.Add(displayString);24:   }25:   SetSize();26:  }

Lines 6 and 7 get the Generic Collection just as in the two other displays. Lines 8 to 10 create the LINQ enumeration query which is ordered by Title and simply selected the entire record. lines 11 through 24 just iterate over the entire collection returned from the above LINQ expression.

Remarks

As I mentioned in the start, my only intent in this article is to explain the LINQ in this application. The forms program used to create this program, for me, is just a holder of LINQ objects. It would be great to see someone take what I’ve done and convert it to a WPF application. Including drill downs and better graphics whould be awesome. Who knows, this may be the start of an open source project.

Thanks

First, thanks to Mike Swanson for inspiring this application and doing the hard work of figuring out what is really going on with the TiVo on the local net. Without that, this would not have been possible. Second, thanks to Fabrice Marguerie, author of Manning’s book "LINQ in Action". Fabrice tirelessly answered several of my very naive questions I posted on the internet forums.

Amazon.com: LINQ in Action

Appendix: How To Install and Run this TiVo Program

If you just want to run the program and don’t care about all this LINQ stuff, here is what you have to do.

First, you need to get three things about your TiVo

1. It’s IP address. This can be found by navigating your TiVo to Message & Settings, then Settings, then Phone and Network.

2. Your Media Access Key: Navigate to Message and Settings, then Account and System Information, then Media Access Key

3. Your TiVo Service Number: Navigate to Message and Settings, then Account & System Information.

After you’ve done this, copy the files from the attached zip file

Tivo Space Viewer Executable Files Tivo Space Viewer Exe

to your desktop (Just drag and drop them to your windows or XP desktop.

The final thing to do is to add your Service number to your hosts file located at %SystemRoot%\system32\drivers\etc\hosts.

add a line for your TiVo Service number and IP as is shown in the following picture.

TiVo Setup Screen on XP

That’s it! click on TivoSummaries on your desktop. If this doesn’t work, I suggest getting Mike’s gadget working, then trying this procedure again.

PeterKellner.net » TiVo

Just the Exe’s for the Tivo Summaries Executables!