(untagged)

LINQ Part 3: An Introduction to IQueryable

Eric Lynch

0.00/5 (No votes)

20 Apr 2018

Part 3 in the LINQ series, this provides an introduction to IQueryable, IQueryProvider, and LINQ expression trees.

Introduction

In this series, we've explored IEnumerable and taken a look at the standard methods that extend this interface. Together these form a small, but crucial part of LINQ, known as LINQ to Objects.

Another important aspect of LINQ is its ability to query other data sources (e.g. databases) where the information does not reside in local memory. In fact, it may even exist on an entirely different machine.

To support this very different requirement, LINQ introduces the concepts of query provider (System.Linq.IQueryProvider), expression tree (System.Linq.Expressions.Expression), and queryable sequence (System.Linq.IQueryable). This article will explore these concepts.

Background

With LINQ to Objects, LINQ can simply provide standard methods that operate directly on the sequences that are queried. With other data sources, this would be inefficient.

For example, for a database, we want to make as few round-trips to the server as possible. Also, we want to request as little information as possible. Finally, we want to take advantage of all of the "smarts" of the database product.

To facilitate this, LINQ introduces the following new concepts:

Query provider (IQueryProvider) - This is specialized software that can interpret a query, so that it efficiently utilizes the underlying resources.
Expression tree (Expression) - A tree is formed from elements of a query. This tree is later analyzed by the query provider. This should be familiar to anyone who has ever written a parser or compiler.
Queryable sequence (IQueryable) - This is the approximate equivalent to IEnumerable in LINQ to Objects. It simply pairs a query provider with an expression tree.

As with IEnumerable, LINQ provides a set of standard methods, defined in the System.Linq.Queryable class. These methods all extend IQueryable. They all share identical names and near identical syntax with their counterparts in System.Linq.Enumerable.

While this reduces the learning curve for developers, it is also deceptive. While conceptually similar, these methods accomplish their goal in an entirely different manner: they build the expression tree.

This is the third in a series of articles on LINQ. Links to other articles in this series are as follows:

LINQ Part 1: A Deep Dive into IEnumerable
LINQ Part 2: Standard Methods - Tools in the Toolbox
LINQ Part 3: An Introduction to IQueryable
LINQ Part 4: A Deep Dive Into a Queryable Extension Method

LINQ Expression Trees

Most of the methods in System.Linq.Enumerable simply create a new instance of IEnumerable that wraps the one on which it operates. Since the underlying sequence generally exists in memory, or is easily acquired, there is no real concern about the mechanism by which it is fetched.

The methods in System.Linq.Queryable operate in a different manner. Each consumes an IQueryable, which is simply a pairing of an expression tree (Expression) with a query provider (IQueryProvider). Each method produces a new IQueryable, where the expression tree has been altered to include additional elements of the query.

Let's consider a simple query:

Source.Students.Where(student => student.LastName == "Marx")

The first IQueryable we need to consider is Source.Students. In this case, the expression tree (IQueryable.Expression) consists entirely of a single node (ConstantExpression). The value of this node (ConstantExpression.Value) is simply the queryable itself.

The next IQueryable we need to consider is the one returned by the Where method. Here things get a lot more complex. In this case, the expression tree (IQueryable.Expression) consists of many nodes. Visually, it appears as follows:

The nodes in the tree are as follows:

The call to the Where method, which takes two arguments, each of which appears in the tree.
The original IQueryable upon which the Where method operates.
A container for the Lambda expression that provides the predicate for the Where method. Note: It seems odd (even to me) that this node is required.
The Lambda expression (student => student.LastName == "Marx") that provides the predicate for the Where method.
The parameter for the Lambda expression (student).
The "equals" comparison that is evaluated within the Lambda expression (student.LastName == "Marx").
The left hand operand for the "equals" comparison, which accesses a property/member (LastName).
The right hand operand for the "equals" comparison, the constant "Marx".
The parameter for the property/member access (student). This is the specific instance where the property value is found.

What Good is an Expression Tree?

So, now we have a nifty expression tree. What use is it?

This is where the query provider (IQueryProvider) comes into the picture. It is responsible for "executing" the query described by the expression tree.

For example, in the case of a database provider, it may do quite a bit. First, it needs to translate the expression tree into a SQL query. So, in our previous example, this may be something like:

SELECT * FROM Students WHERE LastName = 'Marx'

It might then need to execute the query on the database server, fetch the results, instantiate objects for each of the rows, and provide an enumerable of these objects.

A Quick Review

From this article, we've learned the following:

The extension methods in System.Linq.Queryable operate on instances of IQueryable, by building an expression tree (Expression).
IQueryable simply pairs the expression tree (Expression) with a query provider (IQueryProvider).
The query provider (IQueryProvider) is responsible for interpreting the expression tree, executing the query, and fetching the results.

The key word here is "interpreting". Because the query provider (and its underlying resource) is likely more limited than the C# language, there are limitations to what can appear within the query.

In general, the Lambda expressions are limited to fairly vanilla operations: comparisons, string manipulations, and projection (creation of new instances from existing property values). The exact limitations are dependent on the query provider itself.

If you're merely consuming a LINQ queryable, what you've learned so far, is about all you need to understand.

The remainder of this article dives a bit deeper. Feel free to learn more, or skip it, depending on your specific needs.

Exploring More Complex Queries

So far, we've only looked at a very simple query. As the query grows in complexity, the expression tree becomes rather large. To facilitate exploring more complex queries, an application (QueryableFun) is included with this article.

Mostly, I recommend simply playing with the application. It allows the user to run some pre-defined queries. These are selected from the "Queries" drop down list, which appears in the upper right hand corner of the window, in the tool strip.

To execute a query, simply click the "Run" button ().

The application consists of two panels: the expression tree (left) and a tab control (right). Within the tab control, you'll find four tabs. There are as follows:

Expression - Enter or view C# LINQ expressions here.
Properties - When you click on a node in the expression tree, this will display the properties for that node.
Results - A data grid with the results of the query.
Errors - Any compilation errors that occur while evaluating the expression.

A screen capture of the application appears below. Please understand that this is simply a learning application. It is not intended for any production use. As such its quality is lower than what I would normally produce.

For serious explorations of LINQ, I highly recommend LINQPad. I have zero association with the company. I simply relied on this, quite heavily, when I was teaching myself LINQ. The application comes in both free and paid versions.

IQueryable Members

While this is the most important interface, it is also the simplest. Its primary purpose is to pair an expression tree with a query provider. In fact, it is so simple that a single implementation could satisfy most query providers.

It has only four members, which are as follows:

Member Name	Description
`Expression`	This is the expression tree for this queryable.
`ElementType`	The type of element that constitutes the sequence.
`Provider`	The query provider responsible for interpreting and executing the query.
`GetEnumerator<TElement>`	Gets a strongly-typed enumerator (`IEnumerator<TElement>`) for this queryable. This usually calls the query provider to interpret and execute the query.
`GetEnumerator`	Gets a weakly-typed enumerator (`IEnumerator`) for this queryable. Often, this simply invokes `GetEnumerator<TElement>()`.

IQueryProvider Members

While the implementation of this interface is generally quite complex, the interface itself is quite simple. It allows a consumer to create queryables and execute expressions. It has only four members:

Member Name	Description
`CreateQuery()`	Creates a weakly-typed queryable (`IQueryable`) for the specified expression.
`CreateQuery<TElement>()`	Creates a strongly-typed queryable (`IQueryable<TElement>`) for the specified expression.
`Execute()`	Executes the specified expression and returns a weakly-typed result.
`Execute<TResult>()`	Executes the specified expression and returns a strongly-typed result.

QueryableFun Modules

The QueryableFun application is mostly intended to provide a flavor of what expression trees look like. While it is not a very serious effort, it does demonstrate a few clever tricks.

Roslyn - It provides an example of compiling and executing C# script using Microsoft's Roslyn technology.
IQueryable - It provides an example of a very simple IQueryable implementation. Instead of the complexity of evaluating and executing a query against an external resource, this implementation simply wraps an underlying collection.
IQueryProvider - It provides an example of a very simple IQueryProvider implementation. This implementation simply alters the expression tree to replace the original queryable with a queryable for an underlying collection. It then tricks LambdaExpression.Compile() into doing all the heavy lifting.
Toy data - It provides some "toy" data for LINQ queries, so that no database or more complex resource is required.

The modules in this application are as follows:

Module Name	Description
`EnumerableExtensions`	Extensions to `IEnumerable` that simplify finding getting the element type for the enumerable.
`ExpressionTreeBuilder`	An `ExpressionVisitor` that populates a Windows Form `TreeView`.
`ExpressionTreeRemediator`	An `ExpressionVisitor` that swaps out nodes containing the original `IQueryable` for a new `IQueryable` for the underlying collection.
`InterestingQueries`	A set of pre-defined queries that demonstrate some common use cases.
`MainAboutBox`	A fairly standard "Help About..." dialog.
`MainForm`	The User Interface for the application.
`Program`	The main launching point for the application.
`QueryableEnumerable`	An implementation of `IQueryable` that simply wraps an underlying collection.
`QueryableEnumerableProvider`	An implementation of `IQueryProvider` that uses `LambdaExpression.Compile()` to do all the heavy lifting.
`Source`	A static source for LINQ queries, which can be easily referenced in user-provided expressions

In the Transeric.Queryable project, which is also a part of this application, "toy" data is provided in the form of simple POCO classes and collections based upon them.

History

4/20/2018 - The original version was uploaded
4/21/2018 - Added links to other articles in this series
4/23/2018 - Updated exe and src to handle singleton LINQ methods
4/24/2018 - A couple of minor cosmetic and grammar corrections
4/25/2018 - Added link to fourth article in series

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here