Show HN: LINQ for Go

https://github.com/ahmetalpbalkan/go-linq

by aabalkan • 13 years ago

84 51 13 years ago

github.com

krakensden2• 13 years ago

From the documentation: > So if you have 1M elements as input to a plinq function, you will end up 1M goroutines.

Theoretically, there is not much wrong with that.

Understatement in the finest tradition of man pages and RFCs.

aabalkan8• 13 years ago

I wrote that statement thinking about the experiments people are making using millions of goroutines and still end up low cpu utilization when goroutines mainly depend on IO. Go has a neat scheduling for that. Please correct me if I am wrong.

kevingadd• 13 years ago

pretty much any task/job/thread scheduler on the planet is going to pay some absurd overhead costs to schedule and run a million tasks compared with a reasonable number (like 10, or 100), without a measurable improvement in performance. a very smart scheduler can minimize these costs but they will still be there; the state has to be stored and updated.

RyanZAG• 13 years ago

Does it really sound alright to you? I mean it's not the end of the world or anything, but it's adding extra overhead which simply isn't needed when you're not gaining anything from it in this case.

I assume people want to use golang for the speed advantages of servicing many connections at once. If each connection is spawning hundreds of goroutines it's not going to work so well.

nathanwdavis• 13 years ago

Here's a gist I created a couple months ago that kind of explains the performance characteristics of goroutine per loop vs. partitioning with goroutines

https://gist.github.com/nathanwdavis/6290428

Basically, for big lists a goroutine per loop is very slow in comparison.

JeanPierre• 13 years ago

Just for reference, a single goroutine has a 8 kb stack size. Having one million of these things in memory at once would require 8 Gb of memory. Obviously, the scheduler kicks in, but 8 kb chunks of memory still has to be assigned and freed for every routine started and stopped.

krakensden2• 13 years ago

Not intended as a criticism- no one writes every possible feature immediately! It was me expressing enjoyment at the dry way you documented this implementation detail.

kmontrose• 13 years ago

Another approach using code generation is gen: http://clipperhouse.github.io/gen/

(Disclaimer, I work with the guy behind it)

Personally these sorts of libraries just strengthen my opinion that golang really needs proper generics. However, if you're already gonna use Go they're pretty neat.

aabalkan8• 13 years ago

Yes I know that. I have been in touch with Matt several times about this. I really look for ways to improve it, even if it is one small step at a time. Just posted a thread to golang-nuts about this: https://groups.google.com/forum/#!topic/golang-nuts/nLwWkoS7... noting Go really needs generics and maybe even lambdas some day.

Just leaving this project here. Hopefully, I will maintain it as bugs etc arises. Who knows, maybe someday Go will have these features and there will be many apps instrumenting Go for their application logic and use it.

burntsushi• 13 years ago

Go really needs ... lambdas some day

Eh? Go has lambdas:

    func(x int) int { return x * x }(5)

overgard• 13 years ago

Neat. I guess Go can't do type inference on the function types though? Those anonymous function signatures are all kinds of gross.

midas007• 13 years ago

And the dots. Yuck.

masklinn2• 13 years ago

I guess Go can't do type inference on the function types though?

It can't, and there are no generics which is why the input has to be cast back to a Student* in every function. Also why the Sum implementation and the min, max and order (sort) situations are absolutely gross.

mratzloff2• 13 years ago

This is one of two main complaints I have with Go[0], which I've used professionally for the last six months or so. There have been several times when I've thought "if only I had generics..."

Instead I either have to use interfaces and casts (run-time when I would prefer compile-time) or write separate functions for each type (I wrote a Cartesian product library that abstracts the math involved to indexes and lengths, then requires a simple type-specific wrapper simply because of this silly language gap).

Despite that, I still really love Go.

[0] The other being that 1.2 randomly segfaults on my code base during compilation whereas 1.1.2 works fine. For this reason I haven't yet been able to upgrade until I determine the cause.

DanWaterworth2• 13 years ago

yuck, I don't understand how go people can put up without parametric polymorphism.

dsymonds2• 13 years ago

http://commandcenter.blogspot.com.au/2011/12/esmereldas-imag...

drdaeman• 13 years ago

"Can't imagine/understand" is a figure of speech, not literal statement to be taken by its exact meaning.

Personally, I can imagine how someone could live without $conceptname at all. It's as trivial as just never introducing that concept to to the extent they learn it. Or, maybe, that someone knows the concept but unable to use it due to insuperable force, like a lack of necessary tools for target platform(s) or corporate standard dictating what's permitted and what's not.

The point is, this is an unpleasant imagination one don't want to accept. Or something like that. But this is really getting off-topic...

MichaelGG3• 13 years ago

The real language part of LINQ isn't the simple sequence functions (.NET had them before LINQ), it's the LINQ monad embedded into the language.

While personally I don't think I've ever used the LINQ monad syntax, and wish it had been implemented as a generic feature, not every language with support for anonymous functions or closures or function pointers is "LINQ".

olmo• 13 years ago

Some problems with the Go implementation:

Looks like go does not have something similar to extension methods, so a result() call should be put at the end to unwrap the collection.

The lambda syntax is even worst than C++ and no query expressions (aka monad syntax)

Most important: No IQueryable<T> Expression<Func<...>> so doesn't work with remote data sources (SQL, mongo,...)

But the library looks ok given the possibilities

fleitz• 13 years ago

Go pretty much only has extension methods.

aabalkan8• 13 years ago

Great comments. I have thought about creating an interface so people can just implement their own query evaluation methods and, as you said, do querying from remote data sources. That's still a long-term plan, though, I probably need to see somebody using it before digging that deep.

aabalkan8• 13 years ago

I am afraid that's actually not 100% accurate. Microsoft also uses term "LINQ" for method-based query syntax. See here: http://msdn.microsoft.com/en-us/library/bb896339(v=vs.110).a...

MichaelGG3• 13 years ago

Yes, I know Microsoft put "LINQ" over the entire thing, but calling map, filter, fold etc. "LINQ" seems to be a huge stretch. Method-based query options were around in .NET and C# before the LINQ name existed.

It's not really "Language INtegrated" if there's no real language aspect, is it?

virtualwhys• 13 years ago

Nice job but sorry, that is some hideous syntax require to make LINQ Go ;-)

Was expecting the concision and elegance of LINQ-to-SQL.

codereflection• 13 years ago

Can't say I'm a fan of the syntax, but nice work nonetheless.

bad_user4• 13 years ago

In LINQ, the extension methods defined on IEnumerable are useful, but not that interesting, because they are standard functionality in most modern languages. I also think the naming used (e.g. Where, Select) is unfortunate and it would be better to go with the naming used by other languages/frameworks that are much more standard, like filter() or map(). This implementation is also missing SelectMany (e.g. flatMap or bind), which is much more interesting than Select and Where (i.e. you can implement Select and Where in terms of SelectMany).

What makes LINQ special is IQueryable, e.g. the ability to transform a C# expression at runtime into something else, like a database query. Whenever I see article titles like "LINQ in X", I somehow know that it doesn't refer to IQueryable, which is a pity.

Anyway, these functions should be standard in any modern programming language and because Go is missing such standard functionality is one reason for why I don't like Go. Lack of generics has much to do with it, but also Go's interfaces are weaker than type-classes, combined with Google's disregard for everything we've learned about type-systems in the past 30 years (behold Dart's covariant generics) and so I do have reasons for not holding my breath waiting for a fix. Of course, Go is what many people want, so Go is what many people receive.

V-24• 13 years ago

However, most languages in common use are not "modern" by this definition (if by any) ;) I've switched from C# to Java and the clunkiness of it drives me crazy. I know that Java 8 brought a new LINQ-like API, but it won't be available for Android development for a while (not to mention there's existing codebase to maintain etc.).

noelwelsh• 13 years ago

This is very true -- most languages in common use are 20 or more years old and were not cutting edge when they were created. However you don't have to accept the status quo. Scala is viable on the JVM, F# on .Net, and Haskell (and increasingly Rust) if you want to avoid a VM.

_random_• 13 years ago

Or you could pay for Xamarin and avoid having to deal with Java.

bad_user4• 13 years ago

The problem with Java 8 on Android is JDK 7, as it introduced invokeDynamic that will have to be supported by Dalvik, plus I think the additions to the standard library are also a work in progress. It will happen though.

For Android you can use Scala and many people do it. There's a SBT plugin for painless setup and you can work with IntelliJ IDEA as your IDE. Some people even go for Clojure. For iOS, there's RoboVM [1] and you can also use that in combination with Scala - it's very promising.

Java the language sucks, but the JVM and the ecosystem are great and you've got mature languages designed to run on top of it, languages with big communities behind them. Of course, learning a new language and finding the best stuff in a huge ecosystem where choices abound takes time and patience, but I think you'll learn to appreciate it - or if you like C# so much, there's always Xamarin, though it's a little pricey.

[1] http://www.robovm.org/

[2] http://xamarin.com/

aabalkan8• 13 years ago

I would not blame Google in particular. Go team operates like they are not part of Google most of the time. You are right about the abilities of other languages to make things easier for LINQ-like scenarios. But in Go, that's as much as I could get. I don't know what else to do.

I am planning to add SelectMany, ToDictionary, GroupBy and Zip methods soon. I cannot agree you more, I also am not happy about the result I ended up with. Anyway, this was my first project in Go, I will maintain and keep adding stuff to make things easy for other people who might like to use this. But Go is 'really' not for this, I know.

bad_user4• 13 years ago

Sorry for my rant on the language. It's interesting what you did and definitely helpful for other people, especially if you implement the other missing parts you mentioned.

masklinn2• 13 years ago

much more interesting than Select and Where (i.e. you can implement Select and Where in terms of SelectMany).

Well it doesn't provide a folding method either, and just about anything can be implemented using a fold.

kreeben• 13 years ago

What is a folding method?

bad_user4• 13 years ago

True, but if you're into FP, then folding is for those cases in which you really need the flexibility of iterating over the collection but without the side-effects.

This has consequences. For example you can't distribute a fold over multiple threads. I'm not very keen on parallel extensions to collections, haven't found a concrete usage for them where it matters, but this does tell you something about folding - it's a low-level operation with which you're back to specifying how instead of what you want. And for collections, at least you've got the flexibility of doing a fold or a foreach painlessly, but for other container types you're out of luck.

SelectMany/flatMap/bind is the defining operation of a Monad, which is a general purpose design pattern that has many uses (e.g. Option, State, Cont, Future, Iteratee, Rx.Observable, etc). It's cool having it, because then you can painlessly filter and transform the values in those containers without pulling them out of their context. To quote Erik Meijer: "Monads are return types that guide you through the happy path".

frowaway001• 13 years ago

I think we should invent a term for "LINQ, except for all the interesting parts".

shadowmint• 13 years ago

Worth noting that the syntax example is slightly incorrect.

Since go auto-postpends ; to lines, you must have a trailing . on the line to allow the compiler to consider multiline statements like this as multiline statements instead of multiple (syntactically incorrect) single statements.

I presume this is just done for readability in the docs, but it's a little misleading. Correctly, it would be:

    over18Names, err := From(students).  // <--- Notice the trailing . to prevent ;
    Where(func (s T) (bool,error){
        return s.(*Student).age >= 18, nil
    }).
    Select(func (s T) (T,error){
        return s.(*Student).name, nil
    }).
    Results()

It's argued now and then that this is one of the reasons go isn't useful for chain expression libraries like this; it becomes hard to tell at a glance the difference between two consecutive statements and one chained long statement.

(The other reason typically given is it breaks error return expectations, but whatever. I've never really understood what's so hard about simple having the .Final() or .Resolve() or .Results() statement at the end of a chain to get the return code and/or error details shrug)

Show HN: LINQ for Go

Related Stories