Designing LINQ operators

http://beach.goodnanoav.com/

crurneturfink — Mon, 29 Jun 2009 18:46:24 GMT

Questions and answers about Beach hotel southern california and On the beach poster http://beach.goodnanoav.com/

re: Designing LINQ operators

skeet — Wed, 25 Feb 2009 21:43:16 GMT

@Mike Rosenblum: The only reason to compile the expression is to execute it. If you pass in the parameter as well as the expression, you don't need to execute it.

re: Designing LINQ operators

Mike Rosenblum — Mon, 23 Feb 2009 13:19:43 GMT

Jan,

You might have seen it in Jon Fuller's article here: jonfuller.codingtomusic.com/.../static-reflection-method-guards

But how would you do this so that you would not need to make use of .Compile()? I don't quite follow you on that one. Could you show an example?

Weekly Web Nuggets #49

Code Monkey Labs — Mon, 23 Feb 2009 03:30:40 GMT

Pick of the week: The Sad Tragedy of Micro-Optimization Theater General Designing LINQ Operators : Jon Skeet has some good tips on writing LINQ operators. Mark a C# Class Data Member as ‘readonly’ When It’s Read Only : Antonio Bello

re: Designing LINQ operators

Jan de Wit — Thu, 05 Feb 2009 10:43:34 GMT

Here's a cool trick for getting the parameter name without duplicating it and making it visible to refactoring tools. It's not mine, but I really can't remember where I saw it...

using System.Linq.Expressions;

public static void ThrowIfNull<T>(Expression<Func<T>> expr) {

Func<T> f = expr.Compile();

if (f() == null) {

var body = (MemberExpression) expr.Body;

throw new ArgumentException(body.Member.Name + " is null");

}

Example:

void test(string param) {

ThrowIfNull(() => param);

// do stuff with a non-null param

}

For efficiency, an overload could be added which takes the parameter, so we don't need to compile the expression.

Weekly Web Nuggets #49

Code Monkey Labs — Fri, 30 Jan 2009 21:39:05 GMT

re: Designing LINQ operators

Stephen Cleary — Mon, 26 Jan 2009 20:02:34 GMT

I agree that mutating operations should not be supported. C++ iterator categories did allow for this, but I never considered it a clean solution.

As for lazily evaluating lists, I do it all the time. Yes, it's more work than an iterator block, but with a common base class it's not too bad. PowerCollections has one very similar to the one I use.

I think I view LINQ as having more gaps than many people who have worked with it since it first came out. If LINQ is a way of saying "run this query on this input stream", then it's sufficient. But for me, I see LINQ as "execute these algorithms as the data streams through", and when using that perspective, the gaps (both algorithm gaps and performance gaps) become more apparent.

I've been using these types of algorithms for ~10 years. Back then, I called them "pipe algorithms" after the Unix shell pipe (|), because it represented a similar concept: running a stream of data through a series of algorithms. C++ iterator categories and Python generators discovered the same concept, followed now by LINQ.

All of these eventually run into what I call the "tee" problem; Unix developed process substitution with redirection to named pipes, I wrote my own rather complex system for C++ algorithms... and you're developing "Push" LINQ.

re: Designing LINQ operators

skeet — Mon, 26 Jan 2009 19:21:12 GMT

@Stephen: Interesting ideas. Would you be thinking of a "Listable" then?

The main problem I see with that is either a) it becomes a mutating operation (which is nasty - I didn't include "don't mutate the input" in the guidance above, because with IEnumerable you really don't have to) or b) you have to return a whole list. I suppose that *could* be lazily evaluated, but that's quite tricky - it certainly couldn't be done with anything as simple as an iterator block.

Of course, *some* LINQ query operators notice if they're working with an IList - certainly Count() does. I'm not sure beyond that.

Your point 2 could certainly be done quite easily though - an "OrderedIntersect" or something like that which streams both input sequences, and throws if it ever sees anything out of order. (That does require an IComparer rather than an IEqualityComparer, of course.)

As for the "need" for it, I'm not sure. I've found quite a few little gaps, mostly in terms of "I want to apply a projection for the sake of finding the max, but I want the original value rather than the projected one" but I've never really felt the need for the kind of things you're mentioning. That's not to say they wouldn't be useful though - I'm not the only developer using LINQ :)

Thanks very much for your input - food for thought, certainly.

re: Designing LINQ operators

Stephen Cleary — Mon, 26 Jan 2009 18:55:17 GMT

Hi, Jon!

I've been a fan of your blog for some time, but thought I'd comment regarding your note on LINQ buffering.

I have many years of background in C++ iterator/algorithm details as well as Python generators, and so I find LINQ to be particularly interesting as the C# equivalent of these concepts. In particular, I've given some thought to how buffering can be improved for the initial set of LINQ operations.

There are two main problems for LINQ, as I see it:

1) There's no way to distinguish between sequential and random-access. You can kind of say that IEnumerable would imply sequential, while IList implies random-access; but this raises two other problems:

1a) The random-access nature is not preserved through LINQ operators. i.e., Enumerable.Reverse does not return IList, even when applied on an IList.

1b) LINQ operators do not look for random-access input sequences, so they choose less efficient algorithms. i.e., Reverse requires buffering even when passed an array.

2) Some algorithms are not present in LINQ. e.g., an Intersect (or any other set-related operation) on already-ordered input sequences would be much more efficient.

I believe (2) is largely the result of LINQ being done to provide "SQL in C#" rather than a general-purpose sequence algorithm framework.

I've played around with the idea of adding a bunch of LINQ operators in a library that would help to fill these gaps, and I believe it would require a new "LINQ namespace" (for lack of a better name - I'm referring to the facility provided by AsQueryable, AsParallel for PLINQ, etc). So, a new set of operators would work on IList interfaces.

What do you think about the need for this?

Designing LINQ operators - Jon Skeet: Coding Blog

DotNetShoutout — Mon, 26 Jan 2009 06:43:25 GMT

Thank you for submitting this cool story - Trackback from DotNetShoutout

re: Designing LINQ operators

skeet — Sat, 24 Jan 2009 21:14:21 GMT

@Vincent: Oh yes. Reflector has to do a lot of clever work to guess at the source code. Did you use a lambda expression? An anonymous method? How do you tell the difference between for and foreach over an array? I bet there's a ton of heuristics in there.

re: Designing LINQ operators

Vincent Croquette — Sat, 24 Jan 2009 21:11:44 GMT

Thanks Jon

It's in these moments I realise that creating a tool such as Reflector involves more than reflection skills

re: Designing LINQ operators

skeet — Sat, 24 Jan 2009 19:29:49 GMT

@Vincent: Yes, I know that would be nice - but unless you did it in some post-compile step it's basically impossible as far as I can tell.

re: Designing LINQ operators

Vincent Croquette — Sat, 24 Jan 2009 18:55:44 GMT

Yes, but I'd like to have

x.ThrowIfNull()

And the variable name "x" to be retrieved automatically for display purposes

re: Designing LINQ operators

skeet — Sat, 24 Jan 2009 09:40:17 GMT

@Vincent: In fact in MiscUtil I have a ThrowIfNull extension method (or something like that) so I'd write:

x.ThrowIfNull("x");

re: Designing LINQ operators

Vincent Croquette — Sat, 24 Jan 2009 09:08:07 GMT

When .net newbies start using IEnumerable, they may think that Count is part of IEnumerable whithout realizing it is an extension method in the first place

If you stick to the IEnumerable members, you should immediately realise you're doing something odd

BTW Jon, not related to this post but... I often write code like yours:

if (source == null)

{

throw new ArgumentNullException("source");

}

especially when implementing WCF operations where I want to make sure the caller provides a valid data contract parameter

I've always wanted to be able to do something like:

// Operation DoSomething

public void DoSomething(ADataContract parameter)

{

CheckOperationParameter(parameter);

...

}

private void CheckOperationParameter(object parameter)

{

if (parameter == null)

{

string localVariableName = ...

throw new FaultException<InvalidArgument>(localVariableName);

}

Theoritically it should be possible to retrieve the local variable's name using reflection but I haven't found an easy way of doing it yet...