Writing your own LINQ provider

http://aashishkoirala.wordpress.com/2014/03/10/linq-provider-1/

by aashishkoirala • 12 years ago

64 76 12 years ago

aashishkoirala.wordpress.com

Awesome stuff. As a side note, anyone else see a rise in .NET/Microsoft related posts recently?

skrowl2• 12 years ago

I'm actually GLAD to see this rise in .NET stuff. I feel like us enterprise developers are left out on [Y] sometimes. Not everyone rolls python / ruby / node / etc.

thirsteh6• 12 years ago

I will take .NET posts over Node.js ones any day of the week. I feel like I am taking crazy pills whenever somebody celebrates discovering "async performance" in Node.js.

mattgreenrocks6• 12 years ago

It's a shame that .NET is considered 'just' enterprise.

C# is a great language, and there are several libraries with fantastic design aesthetic; one that is much better than most of Ruby/JS.

Pxtl2• 12 years ago

I think the big problem is that MS has a lot of catch-up to do in terms of growing a nice open-source community around the language. .NET suffers from too much "Cathedral" and not enough "Bazaar" so too many developers suffer Microsoft's bloated and broken enterprise tools instead of sensible, rapidly-released libs.

I mean, C# is a fantastic language, but a lot of the libs MS has bundled into it are absolutely horrible and people use them anyways because they're the blessed first-party product. And rather than fixing the problems, Microsoft simply creates a new replacement lib with a new approach. I mean, how many kicks at the can did they get with remoting? DB access? Serialization?

Microsoft has gotten a lot better recently, taking an open-source-style development approach to Entity Framework and MVC, but they're very late to this particular party.

sdegutis2• 12 years ago

Yep, I've noticed more of them. For me at least, it's because I'm beginning to throw out whatever political ideologies are stopping me from being open to using whatever technologies suit my problems best. In this case, it means I'm now open to writing GUI programs for Windows (and I'm even considering using Windows as my new development platform) if it proves to be technologically more innovative and fruitful than its competitors. Same with Windows phones. And since C# with .NET happens to be the language of the platform, I've been toying with that with an open mind. I don't know if other people have the same reason.

gum_ina_package3• 12 years ago

That's actually really interesting - what makes you excited about Win Phone? Is it just that the other platforms are getting stale/boring and WP is the "final frontier"?

Guillaume863• 12 years ago

Nice post, if you plan a writing a full featured provider I recommend using the ReLINQ framework: http://relinq.codeplex.com/

I regret I didn't knew about it when I wrote my first LINQ provider.

aashishkoirala7• 12 years ago

Thanks for the info - I just looked at the home page but have yet to try it out. Looks like it cuts down on a lot of the boilerplate associated with it. Nice.

jdaley• 12 years ago

Although it's quite a bit older than re-linq, I'll throw a recommendation in for IQToolkit: http://blogs.msdn.com/b/mattwar/archive/2008/11/18/linq-link...

I used it to build a LINQ provider for a legacy mainframe database system, abstracting away the incomprehensible table and column naming, weird date systems, EBCDIC, and other nastiness involved in querying that system using straight-up SQL.

One thing I found is that implementing a custom LINQ provider gives you a better understanding of how Entity Framework runs your queries. You get to see how queries get split up into the parts executed in the CLR versus the parts that get translated, and how expressions get rearranged and rewritten.

aashishkoirala7• 12 years ago

I second the comment about getting a better understanding of how EF, etc. work after writing your own provider. Definitely.

thesz4• 12 years ago

You can get away with providing your own classes and extension methods. I did that, my colleagues did that (floowing my attempt). We needed a LINQ access to a non-SQL (hypergraph) database, so basic LINQ wouldn't do it.

You need to know how to parse function bodies and I strongly suggest algebraic types approach, e.g., using SpecificType v = genericTypedVar as SpecificType and checking if v != null. You have to write your own parametrized type supporting Where, Select, etc.

And you're good to go.

It took me three days to write first LINQ-like thing.

aashishkoirala7• 12 years ago

That is true as well, especially if you need to support operations not part of out-of-the-box LINQ. The real meat there is in the Expressions API. The actual LINQ methods, if you look at them, don't do much except provide a way to delegate to the underlying IQueryProvider.

That said, I think for stuff that is supported, better to stay within the "standard approach".

But I get your point.

sidmkp96• 12 years ago

LINQ one of the most beautiful things in C# after extension methods!

corresation10• 12 years ago

I would say that LINQ is one of the worst things to ever happen to .NET. On one team I had to make a dictum that barred LINQ from even being used, except in exceptional circumstances.

I do not mean to raise the ire of defensive .NET programmers by saying this: as with most things, used right it is a beautiful thing, and there is nothing fundamentally wrong with LINQ. But what it was used for in practice was almost always grotesquely inefficient.

If someone had to make a loop to iterate objects to find a member on every function call, pretty soon they'll realize they should using a System.Collections.Generic.Dictionary for instance, or some other appropriate data structure (e.g. Need sorted data? Then use a sorted tree, for instance). When it's a simple LINQ query, though, it's one seemingly harmless little line, so it tends to get a pass.

Pretty soon profiles were just hundreds (if not thousands) of different cases where LINQ was used as a cure-all, destroying performance under any load at all.

LINQ is not magical. It is doing the same dirty work that you would be doing yourself if you needed to filter / sort / recast, but provides syntactical sugar to do so. The compiler and runtime will happily generate horrifically inefficient code if you request it, with no warning or complaints.

The tldr; is that it is a fantastic set abstraction that is more often than not grossly abused.

aashishkoirala7• 12 years ago

I've seen that happen as well, but you can't blame LINQ for that. Shitty developers will make a mess out of any technology.

louthy5• 12 years ago

I would say that LINQ is one of the worst things to ever happen to .NET. On one team I had to make a dictum that barred LINQ from even being used, except in exceptional circumstances.

I find it incredible when I see comments like this. What you should have done is instigated a mentoring programme for the programmers who aren't yet up to speed. LINQ is one of the best things to happen to the language and you just go and kill it.

jmcqk6• 12 years ago

I would change this a little bit. LINQ creates much more readable code, which is always a good thing.

I think the real problem is Cargo Cult programming. People are using linq (especially in combination with EF) and not understanding what's actually going on. So they end up getting things like n+1 selects on collections with thousands of items. They don't understand delayed evaluation. Or how sorting effects a Where() call.

The problem is programmers who think they can take some code from stack overflow and use it without outstanding it. That problem is not unique to LINQ or .NET.

Guillaume863• 12 years ago

Sounds like you'd prefer to forbid LINQ than actually teach how to properly use it, IMO it's absurd.

LINQ is really one of the best things in .NET. Just like any tool, you need to learn it before using it properly.

nightski2• 12 years ago

I think your team is one of the worst things to happen to .NET. No offense, but that sounds like hell.

avenger1232• 12 years ago

Not all of us using .NET use LINQ without any oversight or thought. It's a great tool and needs to be used wisely.

For example, all queries that are generated for database calls via LINQ are reviewed and analyzed to make sure its what is expected.

You are right, LINQ is not magical and was never meant to be. I don't think using LINQ is an automatic ticket to leave your brain at the door.

In your case, it's easier to blame the tech and stop using it than to take responsibility for training the team to be better developers.

I would also imagine that on this team the use of var was also restricted?

profquail3• 12 years ago

Counter-argument: maybe by completely banning LINQ you're prematurely optimizing? If you can describe some query or aggregation with LINQ, it may not always be super-fast, but it's also less error-prone than hand-writing equivalent, faster code. It seems like it'd be better to allow the usage of LINQ, at least for the purposes of rapid prototyping and implementing behavioral/unit tests; once you're happy the system works as expected, you can hit it with a performance/memory profiler to identify specific parts of the code which need to be optimized.

Also, check out LinqOptimizer -- it analyzes your LINQ queries and can optimize them for better performance (and it's free!): https://github.com/nessos/LinqOptimizer

phaedrus• 12 years ago

Yes but LINQ also provides GroupBy() and ToDictionary() which can be used together to replace Where() or First() in subexpressions and restore performance. Simply banning a useful tool's use altogether just means you weren't any more willing to learn how to use the tool correctly than the programmers who were misusing it.

platz3• 12 years ago

Extension methods aren't so great. Odersky resisted putting them into Scala and they went for implicits instead.

aashishkoirala7• 12 years ago

OP here. I wanted to comment on the discussion around ".NET vs. other things" coversation that has started brewing here. I'm not one to hate on any environment or ecosystem. I do make fun of PHP or Java at times, but it's all good-natured. Not to be too PC, but yes, every thing has its pros and cons.

Personally, however, I think C# as a language strikes the right balance in terms of power, ease-of-use, learnability and code readability. Even if you have a problem with Microsoft, I think it is wrong to let your "MS-hate" translate into "C#-hate". It is a beautiful language and the open-source community around it is growing rapidly. The progress made by platform-independent endeavors (case in point Xamarin) is admirable.

In the unlikely scenario that all of programming were to converge into one language, I would want it to be an open-source, platform-independent version of C# (with maybe just a tad more constructs borrowed from the functional programming world).

P.S. I also think it is a mistake to lump C# and Java in the same bucket. Ten or more years ago, maybe. Today, they are VERY different.

doorhammer2• 12 years ago

I do C# dev in my day to day and I really enjoy it, though my personal preference for a "one language" scenario would probably be F#, if it were to come from the MS camp. (I don't mean this as a comparison of languages; just casually throwing out a personal preference)

Was wondering what, in your opinion, are the largest differences between Java and C#? It's been a long time since I've done any Java, and while I can definitely name a bunch, I thought it'd be interesting to get your take on the most impactful or meaningful differences.

Guillaume863• 12 years ago

Well, LINQ is probably one of the most impactful differences.

aashishkoirala7• 12 years ago

Yes, I've heard a lot of praise around F# and its power. I've been meaning to get into it. Coming into it cold, though, I found the syntax a bit cryptic, but that's probably just me. I do plan to power through, though.

In terms of differences with Java (and limiting this to just the language and the syntax), LINQ is a good example of it :) I guess you could classify a lot of the other differences as synctatic sugar, but those little things add up and make a real difference in terms of code brevity, readability and developer productivity. To name a few (and I know that Java, especially with v8 has started adding support for some of these): Properties (and auto-properties), Delegates, Events, Lambdas/Anonymous Functions/First Class Functions, Extension methods, "using" for IDisposable (although Java 7 has something similar in AutoClosable), the lack of forced checked exceptions, the newly added async/await keywords, object and collection initializers and anonymous types, etc.

rip747• 12 years ago

LINQ makes moving from CFML to .NET a lot easier. in CFML you're mostly working with queries and using LINQ makes you feel right at home.

aashishkoirala7• 12 years ago

And just as we were talking about .NET in other environments and Mono and Xamarin, we hear the news about Microsoft in talks to acquire Xamarin. Interesting.

Writing your own LINQ provider

Related Stories