"Push" LINQ revisited - next attempt at an explanation

Efficient "vote counting" with LINQ to Objects - and the value of nothing

Jon Skeet: Coding Blog — Sun, 20 Sep 2009 20:48:27 GMT

A long time ago, I'm pretty sure I started writing Push LINQ in response to a request to effectively

re: "Push" LINQ revisited - next attempt at an explanation

skeet — Thu, 23 Jul 2009 19:44:19 GMT

@Jeff:

I now have DataProducer.PumpProduceAndEnd; it's probably easier to look at the code (in MiscUtil) than explain. Basically it takes a "final results" enumerable and yields any values produced by that, pumping more data into the pipeline when there's nothing being yielded. Unfortunately this introduces buffering, but it's relatively minimal.

re: "Push" LINQ revisited - next attempt at an explanation

Jeff — Thu, 23 Jul 2009 19:25:44 GMT

Tremendous!

Could you pass 'query' into the ProduceAndEnd() method, have it create 'results' (calling query.AsEnumerable()), then return results from ProduceAndEnd()?

re: "Push" LINQ revisited - next attempt at an explanation

Marc Gravell — Fri, 26 Jun 2009 22:50:17 GMT

@Keith - a bit overdue now (I was just passing), but the observers can take themselves out of the loop fairly easily - the First/Take etc implementations do exactly this.

More importantly, if they have something chained watching data from *them*, they send the "all done" signal at this early time - so the result of your .Where(...).Take(10).Sum() can be available long before the "real" data feed has finished... as soon as we've seen 10 items, we can (and do) sum them.

re: "Push" LINQ revisited - next attempt at an explanation

Keith J. Farmer — Tue, 14 Apr 2009 07:52:00 GMT

I've seen this pattern before, even recently. It's well-worth the study.

The decision to make is who controls when to halt the data. The "Push" model places that decision at the data source, whereas the "Pull" model, naturally, places it with the requester. Both are valid choices to make, based on the nature of the two ends involved (eg, is one end a serial port, or are you wanting to truncate the data after finding the first Foo?).

re: "Push" LINQ revisited - next attempt at an explanation

skeet — Fri, 19 Dec 2008 12:00:39 GMT

@David Cater: Just download MiscUtil (pobox.com/.../miscutil) - it's all in there.

re: "Push" LINQ revisited - next attempt at an explanation

David Cater — Fri, 19 Dec 2008 11:53:06 GMT

Very nice. Any chance that you'll be posting the DataProducer source code, or do you just plan on publishing that as a binary?

Thanks for the article!

a digital aesthetic — Push vs Pull Linq

a digital aesthetic — Push vs Pull Linq — Fri, 21 Nov 2008 17:14:53 GMT

Pingback from a digital aesthetic — Push vs Pull Linq

re: "Push" LINQ revisited - next attempt at an explanation

skeet — Fri, 08 Aug 2008 08:07:51 GMT

Will have to get my head back into Push LINQ to properly understand your suggestion, but I'll look at it :)

re: "Push" LINQ revisited - next attempt at an explanation

Fredrik Kalseth — Fri, 08 Aug 2008 08:02:05 GMT

Very cool stuff! :)

A thought: why not have a factory method on the DataProducer so that to create a query (as in the last example in your post) you do

var query = voters.Query( from voter in voters.... );

This method could then hide the AsEnumerable() 'hack', and it would sit nicely next to the IFuture<> methods on there aswell.

Data pipelines as a conceptual stepping stone to higher order functions

Jon Skeet: Coding Blog — Thu, 28 Feb 2008 00:08:42 GMT

I was explaining data pipelines in LINQ to Objects to a colleague yesterday, partly as the next step

re: "Push" LINQ revisited - next attempt at an explanation

Dennis van der Stelt — Mon, 04 Feb 2008 19:10:52 GMT

Finally found the time for some reading. Excellent explanation, I love the pictures!

Be sure to blog about this more in the future!

Random Reading: Push Linq « Tales from a Trading Desk

Random Reading: Push Linq « Tales from a Trading Desk — Tue, 15 Jan 2008 18:51:48 GMT

Pingback from Random Reading: Push Linq « Tales from a Trading Desk

re: "Push" LINQ revisited - next attempt at an explanation

Luke Marshall — Fri, 11 Jan 2008 03:18:03 GMT

Awesome idea, excellent explanation!

Keep it up!

re: "Push" LINQ revisited - next attempt at an explanation

skeet — Thu, 10 Jan 2008 20:45:11 GMT

Ara, it's not clear whether you mean "push" LINQ or LINQ in general. Furthermore, I certainly haven't seen LINQ taking an order of magnitude out of performance, when used properly. Evidence?

re: "Push" LINQ revisited - next attempt at an explanation

Ara — Thu, 10 Jan 2008 20:31:58 GMT

Too slow, too bloated and killing a machine by further order of magnitude (another 0 added).

LINQ finally out of hype, try and attempt the same with imperative code.

Random Reading: F#, Linq, ToyScript « Tales from a Trading Desk

Random Reading: F#, Linq, ToyScript « Tales from a Trading Desk — Wed, 09 Jan 2008 23:48:36 GMT

Pingback from Random Reading: F#, Linq, ToyScript « Tales from a Trading Desk