Jesse Liberty

 Turing Project Page: [Novice: 7]    FAQ    Table of Contents

  What is this and where do I start?

Linq – More Powerful Than A Locomotive

Linq was introduced into .NET with .NET 3.5, and is fully supported in C# 3 and VB9.  It has rapidly become the database language of choice among .NET developers, providing an efficient and “natural” alternative to making calls out to stored procedures or to T-SQL.

You can use Linq for retrieving, updating, removing or adding data to and from SQL-server or any database or from any other collection such as a list<T>, an XML file, etc. An important benefit of this is that the same logic used to retrieve specified data from an in-memory collection may be applied to your data storage layer. That is, rather than using an enumerator on your list but ADO.NET on your database, you can use a Linq query against both.

The following example violates my normal practice of keeping things as simple as possible; I’m doing so consciously to illustrate a fully developed in-memory Linq example that I will walk through in detail. The next posting will demonstrate Linq To SQL, and will give us an opportunity to explore a bit more of Linq.

Let’s start by declaring a class that we can then create a collection of:

   1: using System.Collections.Generic;

   2:  

   3: namespace Linq1

   4: {

   5:   public class Blog

   6:   {

   7:     public string BlogName { get; private set; }

   8:     public string BlogOwner { get; private set; }

   9:     public string BlogUrl { get; private set; }

  10:     public int BlogRanking { get; private set; }

  11:  

  12:     public override string ToString()

  13:     {

  14:       return string.Format(

  15:         "{0} ( {1} ), ranked {2} is owned by {3}",

  16:         BlogName, BlogUrl, BlogRanking, BlogOwner );

  17:     }

  18:   } // end class

  19: }   // end namespace

  20:  

(full source code will be available, so I’m going to indulge in using line numbers)

The Blog Class

The class consists of just four properties (using C#’s automatic properties) and an override of ToString to make it easy to examine what we have. Each instance (blog)  is assigned a ranking so that I can order the blogs in favorite to least favorite.

I will want to generate a collection of these objects, and since I’ll only want to do so for debugging and testing, I’ll incorporate a method to generate that data as a static member of the class (the following code is inserted after the override of ToString and before the brace closing the class:

   1: public static List<Blog> GenerateBlogs()

   2: {

   3:   var blogs = new List<Blog>

   4:               {

   5:                 new Blog

   6:                 {

   7:                   BlogName = "Silverlight Geek",

   8:                   BlogOwner = "Jesse Liberty",

   9:                   BlogUrl = "http://www.SilverlightGeek.me",

  10:                   BlogRanking = 5

  11:                 },

  12:                 new Blog

  13:                 {

  14:                   BlogName = "Mike Harsh's Blog",

  15:                   BlogOwner = "Mike Harsh",

  16:                   BlogUrl = "http://blogs.msdn.com/mharsh/",

  17:                   BlogRanking = 4

  18:                 },

  19:                 new Blog

  20:                 {

  21:                   BlogName = "Brad Abrams",

  22:                   BlogOwner = "Brad Abrams",

  23:                   BlogUrl = "http://blogs.msdn.com/brada",

  24:                   BlogRanking = 3

  25:                 },

  26:                 new Blog

  27:                 {

  28:                   BlogName = "Computer Zen",

  29:                   BlogOwner = "Scott Hanselman",

  30:                   BlogUrl = "http://blogs.msdn.com/hanselman.com/",

  31:                   BlogRanking = 2

  32:                 },

  33:                 new Blog

  34:                 {

  35:                   BlogName = "Scott Gu's Blog",

  36:                   BlogOwner = "Scott Guthrie",

  37:                   BlogUrl = "http://weblogs.asp.net/scottgu",

  38:                   BlogRanking = 1

  39:                 }

  40:               }; // end initialization of collection

  41:   return blogs;

  42: } // end static method

On line 3 we declare blogs using the new var keyword (implicit type) and assign to blogs a new List<Blog>, which we proceed to initialize using object initialization  The net effect is that we end with a list of five blogs, fully initialized, which we can return to any method that needs such a list.

Let’s look at the UI now, which will serve as a spec for building the (simple) application, allowing us to take a look at how Linq can make querying for a subset of the blogs easier (this is a posting about Linq!)

— live demo —

— end live demo —

Notice that you can use the spinner to choose to list between 1 and 5 blogs and as you’ll see they are listed in “rank” order.

The Xaml Behind the Display

The data is of course hard wired, but let’s look at how the data gets from the Blogs class into the data grid.

Step 1 is to create the data grid, the prompt and the spinner.

The trick to using a datagrid is to drag it from the toolbox onto the markup so that Visual Studio will include the right libraries and create the right namespace for you. You can of course, add the reference yourself,

and then add a namespace/alias at the top of your file

xmlns:data="clr-namespace:System.Windows.Controls;assembly=System.Windows.Controls.Data"

In any case, with that in place, you are ready to add the DataGrid which you can do in the mark up or programmatically. I’ll do so in the markup, telling the DataGrid not to generate the columns, but rather to use the columns I’ll describe:

   1: <data:DataGrid AutoGenerateColumns="False"

   2:                x:Name="BlogGrid"

   3:                Grid.Column="0"

   4:                Grid.Row="1"

   5:                Grid.ColumnSpan="3"

   6:                RowBackground="AntiqueWhite"

   7:                AlternatingRowBackground="LightGreen"

   8:                >

   9:     <data:DataGrid.Columns>

  10:         <data:DataGridTextColumn Header="Blog"

  11:                                  Width="SizeToCells"

  12:                                  Binding="{Binding BlogName}"

  13:                                  FontFamily="Georgia"

  14:                                  FontSize="18" />

  15:         <data:DataGridTextColumn Header="Owner"

  16:                                  Width="SizeToCells"

  17:                                  Binding="{Binding BlogOwner}"

  18:                                  FontFamily="Georgia"

  19:                                  FontSize="18" />

  20:         <data:DataGridTextColumn Header="URL"

  21:                                  Width="Auto"

  22:                                  Binding="{Binding BlogUrl}"

  23:                                  FontFamily="Georgia"

  24:                                  FontSize="18" /> 

  25:     </data:DataGrid.Columns>

  26: </data:DataGrid>

Note that each column is bound to a property in the Blog object; we’ll get the collection of Blog objects in the code behind. Before we do, though, let’s be sure to add the prompt and spinner for the top row:

   1: <TextBlock Text="How many?"

   2:            Grid.Column="0"

   3:            Grid.Row="0"

   4:            FontSize="14"

   5:            FontFamily="Georgia"

   6:            VerticalAlignment="Bottom"

   7:            HorizontalAlignment="Right"

   8:            Margin="5"/>

   9: <inputToolkit:NumericUpDown x:Name="num"

  10:      VerticalAlignment="Bottom"

  11:      HorizontalAlignment="Left"

  12:      Margin="5"

  13:      Width="50"

  14:      DecimalPlaces="0"

  15:      Minimum="1"

  16:      Value="1"

  17:      Grid.Row="0"

  18:      Grid.Column="1" />

Notice that the NumericUpDown comes from the Toolkit. You can read all about the Silverlight Toolkit here, and we have videos on Toolkit controls here.  The NumericUpDown control is handsomely demonstrated in the Toolkit Samples

We make fairly simple use of it, setting its minimum value and its current value to 1.

Implementing The Event Handlers

Turning to the code behind we begin by creating a private member, blogs of type List<Blog>.

In the constructor, we assign to that member variable the result of calling the static method we created in the Blog class. Being extremely careful to make sure that we have data to work with, we then check to ensure that the member variable blogs is no longer null. (lines 8-11)

   1: public partial class MainPage

   2:  {

   3:    private readonly List< Blog > blogs;

   4:    public MainPage()

   5:    {

   6:      InitializeComponent();

   7:      blogs = Blog.GenerateBlogs();

   8:      if ( blogs == null)

   9:      {

  10:        throw new ArgumentNullException( "GenerateBlogs returned null!" );

  11:      }

  12:      num.Maximum = blogs.Count;

  13:      num.ValueChanged += ( (sender, e) => NumValueChanged(sender, e) );

  14:      SetGrid();

  15:    }

  On line 12 we examine how many entries there are in the List<Blog> and set the Maximum value for the NumericUpDown to that number.

We set an event handler for when the user clicks on the NumericUpDown. The syntax may strike you as a bit odd; this is a Lamda Expression (an integral part of Linq as it turns out) and says that we are registering an event whose delegate will match a method that takes two objects and given those two objects will return the result of invoking NumValueChanged and passing in those objects.

Finally, we call the private method SetGrid, which would normally be the contents of the method NumValueChanged, but is factored out so that we can also call it from the constructor.

   1: void NumValueChanged( object sender, System.Windows.RoutedPropertyChangedEventArgs<double> e )

   2: {

   3:   SetGrid();

   4: }

The incredibly astute reader will have noticed that we’ve not assigned an ItemsSource for the DataGrid. That is done on purpose as we’re going to create the ItemsSource out of a selected subset of the member collection blogs.  All of that work happens in SetGrid:

Linq (At Last!)

The job of SetGrid is to retrieve the value from the NumericUpDown and to retrieve that many Blog entries from the collection blogs, based on the BlogRanking that we previously assigned to each entry.

   1: private void SetGrid() 

   2: {

   3:   var result =

   4:     from eachBlog in blogs

   5:     where eachBlog.BlogRanking <= num.Value

   6:     select eachBlog;

   7:  

   8:   BlogGrid.ItemsSource = result;

   9: }

It accomplishes this task by using a Linq query against blogs, the result of which is placed in the anonymous type variable result.  The actual type of that variable, however, is determined by what the Linq statement returns, and that is an object of type IEnumerable<Blog>. 

A quick check of the documentation will show that the ItemsSource property for the DataGrid is looking for an IEnumerable,

And the last line of SetGrid assigns the variable result to the ItemsSource property, causing the DataGrid to be populated.

Parsing The Linq Query

The Linq query will be quite familiar to anyone who has worked with SQL; the most noticeable difference is that the select statement comes at the end rather than the beginning.

We begin with from eachBlog in blogs  — blogs of course is our collection obtained in the constructor, and eachBlog is a temporary variable of type Blog that is created implicitly just by using it in this way (there is no earlier statement such as

Blog eachBlog;   // not needed

The where statement filters for all the entries where the BlogRanking property of the Blog is less than or equal to the Value currently stored in the NumericUpDown. Thus, if the user sets the NumericUpDown to 3, we’ll get the entries whose ranking (that is, whose property BlogRanking) is 1,2 and 3, as set in the initialization of the Blog entries in the static method.

The select statement adds each of these entries to the IEnumerable result and then, finally result is set as the ItemsSource for the DataGrid. 

In order to get into more advanced aspects of Linq it will be helpful to have a more complex data source; the next posting will take a look at Linq to Sql.

 Turing Project Page: [Novice: 6, Advanced: 4]    FAQ    Table of Contents

  What is this and where do I start?

In the previous entry, we created the design for the database. Over the next few postings, we’ll be working with that Data in five distinct ways:

1. Gather data from the bloggers about themselves, their blog and each entry
2. Store that data in a database
3. Extract that data as needed from the database
4. Transmit the data to a Silverlight application
5. Display the data in a Silverlight application

Of these five, we’ve made simplifying interim decisions about numbers 1,2 and 5:

Gathering Data

We’ve decided, for now, to gather the data from the bloggers by using a form, though over time we’ll want to migrate to other, better options.

Storing Data

We’ve also decided for now to use SQL Server as our database, though the truth is we can use almost any data store.

Displaying The Data

Finally, we’ve decided to hold off on the Display decisions for now, making do with a very simple data grid  just to check that we’re getting the data we think we’re getting.

Extracting and Transmitting Data – Too Many Choices?

Emperor Joseph II: My dear young man, don’t take it too hard.
Your work is ingenious. It’s quality work.
And there are simply too many notes, that’s all.
Just cut a few and it will be perfect.

Mozart: Which few did you have in mind, Majesty?
Amadeus 1984

It is possible to complain that Microsoft has provided too many choices for retrieving and transmitting data. Over the past decade we’ve seen RDO, DAO, ADO, COM, DCOM,  ADO.Net, ADO.NET Data Services, Entity Framework, LINQ, ASMX, WCF, RIA Services…

But that would be querulous; after all, each new option (with the possible exception of COM!) has achieved a higher level of abstraction for the programmer; handling more of the plumbing and requiring less repeated, error prone and annoying code in our application.  But yes, it is hard to keep up.

One suggestion that I’ve found helpful,  is to separate the Data Access Layer (DAL), the transport layer and the Business layer, when deciding on which technologies to employ.  That doesn’t necessarily mean three separate sets of classes, but it does mean three distinct concepts. The picture I keep in my head looks (crudely) like this:

Data Access Layer

For the DAL layer your realistic choices today are

• ADO.NET
• Linq to Sql
• Entity Frameworks

ADO.NET Is Not ADO

ADO at one time stood for ActiveX Data Objects, and the original ADOwas the successor to Remote Data Objects (RDO) and Data Access Objects (DAO).  All of this is back in the 20th Century, and is of interest only to those of us who lived through the Dark Ages (the time of COM).

ADO.NET is not named thus because it is the illegitimate child of ADO in any real technical evolutionary sense, but rather to send the message that it is the .NET replacement for ADO; and it has been a hearty replacement indeed.

Impedance Mismatch

When electrical engineers refer to an Impedance Mismatch they mean the need to match up the input impedance of an electrical load to a fixed impedance (most often, the signal they are working with).  This became an irresistible analogy for the problem that object oriented programmers have trying to match up their object model with the relational model most often used in successful databases. (This metaphor has become so entrenched that Wikipedia’s page for Impedance mismatch lists both meanings!)

The key problem, in short, is that relational databases are very (very!) good at storing, updating and retrieving data, but they achieve their terrifying performance at the cost of insisting of thinking about data as consisting of records and fields (rows and columns). 

While programmers have had enormous success thinking about objects and their relationships

In the case shown, the mapping from object to table appears deceptively straight forward, but as your application grows the complexity transcends the fact that properties of an object may not map to a single table: with many applications there simply isn’t an obvious way to build a given object from the provided tables without serious manipulation or compromise.

ADO.NET As A First Layer of Abstraction

ADO.NET was a great first attempt at solving this problem and made life significantly easier. That said, ADO.NET does stay pretty close to the relational model, and the objects in ADO.NET include

• DataReader
• DataSet
• Connection
• Command
• DataAdapter
• DataTable

and so forth.  You certainly can accomplish everything you want; and there may be times you’ll go to ADO.NET specifically because it maps so cleanly to your database, but there are alternatives today that do much of the drudge work of ADO.NET for you. 

In  a future posting, I’ll come back to this exercise and implement it with ADO.NET.

Linq To SQL

LINQ was one of the most significant additions to .NET 3.5. 

Linq to Sql is specifically designed to use LINQ to interact with SQL Server. That is both its power and its Achilles Heel as it is very coupled to the particulars of SQL Server technology.  While Linq To Sql would certainly do the job for us, I’d prefer to use something more flexible, and a bit more decoupled from the database and frankly, something with a bit more object encapsulation

On To Entity Framework

This brings us to Entity Framework; a topic worthy of a full posting, and that will be next in this series on the Advanced track; the Novice track will take a short digression to review Linq, which we’ll be using quite a bit in this project.

(click on the image to see other Mini-tutorials)

[Novice: Page 5, Advanced: Page 3]  FAQ  Table of Contents

This project will explore, among other things, getting data into and out of a SQL Database. In fact, we’ll explore a number of ways of doing so. To get started, however, we need to create a database that is sufficient for the project and sufficiently illustrative of working with relational databases, without being overly complex.

Fortunately, the project requirements are of about the right complexity.  Returning to the spec, (such as it is) we can see that we need to capture the following information each time an entry is made in a participating blog:

• The name of the blog
• The title of the entry
• The URL of the entry
• Info about the blogger (first name, last name, email & phone)

It would also be good to know when the blog entry was first created and when it was last modified.

Flat or Normalized?

You certainly could create a flat file that has all the information we need:

The advantage to this approach is that it is quick, easy, simple to understand. Putting in a few records, however, quickly reveals why this kind of flat-file database is now reserved for non-programmers creating simple projects such as an inventory of their music:

Even with just a few records, you can see that a great deal of data is duplicated in every record,

This duplication can make the database very hard to maintain and subject to corruption. To prevent this, databases are ‘normalized’ – a complex subject that boils down to eliminating the duplication by factoring out common elements into separate tables and then reaching into those tables by using unique values (foreign keys).  Thus, without bogging ourselves down in db theory, we can quickly redesign this flat database into three tables:

In this normalized database, each entry has only the information that is unique to the particular BlogEntry:

• Title
• URL
• Date Created
• Date Modified
• Short Description

The entry also has the ID of the Blogger who wrote the entry and the ID of the Blog that the entry belongs to. 

A second table holds the information about each Blog, and a third table holds the information about each Blogger.  Thus, a given Bloggers first and last name, alias (email address at Microsoft.com) and phone number are entered only once for each blogger, and referenced in each entry by ID (known as a foreign key). 

The diagram also shows that the two foreign-key relationships are named

• FK_Blog_Entries_Blogs
• FK_BlogEntries_Bloggers

These simple names indicate that there is a foreign key relationship from BlogEntries to Blogs and another from BlogEntries to Bloggers.  The data is now much cleaner:

Notice that the BlogEntries table now does not duplicate the first and last name and phone number for each Blogger, but rather just refers to that Blogger’s ID. For example, entry #3

was written by Blogger 4 (see ID in the middle image) and was placed in Blog 3 (see ID  in the top image)

Creating A Join Query (T-SQL)

You can easily write a query that recreates the complete set of information by “joining” the tables:

select BlogName, FirstName, LastName, Title, Description from BlogEntries be

join Bloggers bb on be.Blogger = bb.ID

join Blogs b on be.Blog = b.ID

This statement says to select the listed fields (it isn’t necessary to identify the table explicitly as there is no duplication). It then tells SqlServer where to find them in the “from” clause. The first table is BlogEntries which we assign a short-name of be.

We then we join to that tables the Bloggers table, but we match up the rows in BlogEntries with the Rows in Blogger where the value in BlogEtries.Blogger is equal to the ID in the table Bloggers.

You can imagine that SQL Server thus looks at the two tables and joins the matching rows, making a new (temporary) wider table:

It then looks at the Blog table and joins the Blog value to the ID in that table:

Giving us the ability to create a virtual flat table without the duplicated values.

(click on the image to see other Mini-tutorials)

 Turing Entry: [Novice: 3]      FAQ     Table of Contents    

The Turing Project design calls for a four component system:

• Data Entry
• Data Storage
• Data Service
• Data Display

Eventually Data Entry will be via email or web, Data Storage will probably be SqlServer, Data Service will probably be a WCF application and Data Display will be an increasingly spectacular Silverlight application.

Corpus omne perseverare in statu suo quiescendi

Expanding the Spec – Reports

In discussion of this project, one feature that surfaced right away was the desire to have a module that displays statistics about blogs, authors and topics in various ways.  I imagine that we would certainly want to have a treemap for topics and other charts that show, for example, the pace of blogging for a given contributor.

A treemap is an increasingly popular control for displaying trends in data because you can put a lot of information in a small space and still make it relatively easily understood. There are nice write-ups of the treemap here and here

(Colors indicate categories, saturation of color indicates quantity)

Successive Approximation

One of the difficulties in the “get it working and keep it working/ successive approximation” approach to coding is that you have to be able to see a hut and imagine a mansion.  The advantage is that it is easier to build a hovel and turn it into a hut and from there to a ranch, to a tudor, to a center-hall colonial and on to a mansion (at least it is for me; others disagree).

The second big obstacle many of us face is Newton’s First Law: “Every body persists in its state of being at rest… except insofar as it is compelled to change its state by force impressed.”  That is, it is hard to get started.

Overcoming Inertia

I find that the best way to  overcome inertia is to start coding. (I also find that the best way to spend my day).  Let’s begin with Data Entry (it happens to be first on the list). 

In time, I imagine that adding a blog entry to the db will be very low-friction; the user will be able to send an email or, even better, we’ll write a plug-in for LiveWriter that takes care of the notification automagically.  For now, however, we’ll get 90% of the functionality by writing a small application that gathers the data in a form and prepares the data for distribution to Data Storage.

Form

To create the form, we first need to know what data we need.  As a first cut I’ve identified the following useful data:

Once again, in time we’ll divide these into those fields which we only need to collect once (e.g., Full Name, URL of Blog) and those which are specific to a given blog entry.