Q – A Data Language

http://www.q-lang.io

by miloshadzic • 12 years ago

220 81 12 years ago

q-lang.io

koloron• 12 years ago

That's at least the third programming language named Q:

https://en.wikipedia.org/wiki/Q_(programming_language_from_K...

https://en.wikipedia.org/wiki/Q_(equational_programming_lang...

egeozcan• 12 years ago

There's also a tool named q to query text files: https://github.com/harelba/q

mercurial3• 12 years ago

And a promise library: https://github.com/kriskowal/q

michaelfeathers• 12 years ago

I vote for the APL derivative.

gruseom4• 12 years ago

It is probably the most astonishing combination of simplicity, efficiency, and beauty that I've seen in computing.

Unfortunately, it's so out-there that nearly everyone dismisses it as unreadable and daft. In a better world, it would be taught and studied as an exemplar of design. (On the other hand, they've succeeded financially so how cool is that.)

ww2• 12 years ago

I found that APL/Q/J and vim share the same required mindset. If someone loves one, it is likely he/she will like another.

mathattack• 12 years ago

That's what I thought clicking through here.

Kurtz79• 12 years ago

Also, naming a programming language with a single letter is not very search-friendly.

baldfat• 12 years ago

As someone learning and using R I agree 100%

psychometry2• 12 years ago

Non-rhetorical question: Why do people continue to do this? The number of unique names still available for a programming language is basically infinite, yet we get stuck with Go and R and C#.

Google is pretty smart about categorizing searches for languages like Ruby and Java (both common words), but with R especially it's often quite difficult to get relevant results.

squigs25• 12 years ago

Kx's Q is used widely in finance. Pretty funky, but I used it a bit in my last job.

EGreg4• 12 years ago

And also we are releasing a platform soon called Q

http://github.com/EGreg/Q

EGreg4• 12 years ago

Why downvoted?

psychometry2• 12 years ago

Then it's not too late to change the name to something that people will actually be able to find information about. Be glad you read all this feedback today and not the day of your release.

msutherl• 12 years ago

Q has a successor called Pure, which is a very interesting language: http://purelang.bitbucket.org/.

anonu2• 12 years ago

I got excited at first - I thought they were referring to the Q language from KDB. This reminds me quite a bit of Google's protocol buffers.

aet• 12 years ago

Same here. I'm interested in figuring out q/kdb, but it seems a bit opaque.

cangencer• 12 years ago

As someone who had to teach myself kdb/Q recently, I can recommend the Q for Mortals book: http://code.kx.com/wiki/JB:QforMortals2/contents

anonu2• 12 years ago

It is opaque - but somehow big banks and other clients dish out millions of dollars in licensing fees to bring this in... and the reason is that it works really well as a time-series tick-data store ...

mercurial3• 12 years ago

A "data language", in this context, is a language for representing and validating data. I'm not clear on why it's called "Q", or why it calls marshalling/unmarshalling undressing/dressing. The language itself looks alright, it's dependently typed. It doesn't say anything about the speed of the existing JS/Ruby implementations.

hgh• 12 years ago

Perhaps the name is based off the R language (which came from S), a major data/stats language. Data representation comes before analysis, hence Q?

Renaming marshaling, couldn't tell you.

skrebbel• 12 years ago

I really like this. In general, a well-supported non-crappy schema language for modern data exchange formats is long overdue (e.g. JSON Schema is horribly un-human-friendly). The fact that they aim higher and allow for a much more detailed set of validations is great, IMO.

I hope that this becomes a success and gets implementations in many programming languages.

FeloniousHam• 12 years ago

For "human-friendliness" (eg. hand-edited configs), I'm all in with YAML. AFAIC, JSON is strictly for machine to machine data transmission.

While I think the idea is neat, a data language should not include processing logic. It's a wash where the constraints get specified, and in my experience, a single point of implementation is far more maintainable.

nevi-me3• 12 years ago

I really like this. In general, a well-supported non-crappy schema language for modern data exchange formats is long overdue (e.g. JSON Schema is horribly un-human-friendly).

I think I'm a bit confused, is Q the actual schema or a validator of the schema? I can also validate a JSON document using JavaScript and respond with a true or false based on whether it meets certain criteria. So I think that there's something amiss with your statement above, specifically that "JSON Schema is horribly un-human-friendly". Doesn't Q take JSON and let the user know if it's correct or incorrect? WOuldn't JavaScript, or any other language at that, do the same?

rpedela• 12 years ago

JSON Schema refers to http://json-schema.org/

I don't personally understand what is so "un-human-friendly" about it. It is easy to use and understand in my opinion.

TheZenPsycho2• 12 years ago

have you seen orderly?

http://orderly-json.org

vithlani2• 12 years ago

For data exchange, what is wrong with using an ASN1 specified, BER encoded file? Genuine question...

spacemanmatt2• 12 years ago

The binary encodings are cumbersome and difficult to perfect.

tinco2• 12 years ago

It's interesting, but is it powerful enough? There's a great talk by Zed Shaw about implementing authorization code. I think the same issue holds here.

A language might work for simple and more complex type constraints, but at some point the constraints might become so weird that you really need a full blown turing complete language to define them.

And that point might be sooner than you'd like. Why not instead of using expressions in this constrained language, use something like Haskell?

It's got the super powerful typesystem, and you're certain that it will be able to express any constraint you throw at it.

That said, this language does look nice, and I haven't really tried to find out if there's any obviously important constraints you wouldn't be able to build in it.

Bjoern2• 12 years ago

This one?

http://vimeo.com/2723800

tinco2• 12 years ago

Yes that's the one :)

mercurial3• 12 years ago

It's got the super powerful typesystem, and you're certain that it will be able to express any constraint you throw at it.

You can go pretty far with, eg, Java, if you're ready to pay the cost in verbosity, by making types for everything and having read-only objects. Haskell is not dependently typed, you can't use the type system to make sure a value is in a given range. You could use phantom types to make sure your type has been validated before, but this kind of control is not really first-class.

masklinn2• 12 years ago

Why not instead of using expressions in this constrained language, use something like Haskell?

Haskell types can't depend on values so there are limitations. If you go that way, you might as well go whole hog and use a dependently typed language (e.g. Idris)

tel3• 12 years ago

(a) As others have said, Haskell's type system is certainly now powerful enough to express any constraint you throw at it.

(b) Almost directly related to that, the less powerful your constraint language the more analyzable it is. I don't necessarily know that Q has these properties, but by constraining how complex the restrictions it encodes are it becomes more amenable to analysis. At first blush, it looks like you'd probably be able to throw a SMT solver at it quite easily.

masklinn2• 12 years ago

is certainly now powerful enough

Small typo there.

frezik• 12 years ago

You might want to embed this inside a larger language. That wouldn't necessarily preclude using a Turing Complete language, but you wouldn't want it to be a complicated one.

Bjoern2• 12 years ago

Interesting... here is a tool, which is also going twoards data oriented approach.

"Drake - a kind of make for data"

http://blog.factual.com/introducing-drake-a-kind-of-make-for...

tel3• 12 years ago

It's strange to me how dependent Q becomes on its host language. Instead of defining its own types in order to sit between various serializations and host systems, Q seems to simply augment the Host with a few extra types.

bokchoi• 12 years ago

I agree. This is something that XMLSchema got both right and wrong. The XMLSchema basic datatypes [0] are well defined and for the most part fit nicely in host languages. However, they really screwed up with the complex types making it difficult to map onto programming languages for multiple reasons (sequence/choice can't be easily modeled in most programming languages without intermediate types, extension versus restriction, derivation versus substitution groups.) For a good write up of the design philosophy behind XMLBeans, see:

http://davidbau.com/archives/2003/11/14/the_design_of_xmlbea...

[0] http://www.w3.org/TR/xmlschema-2/

bjourne2• 12 years ago

Given that you use a powerful rdbms, such as postgres, you don't need anything more than sql constraints. For example, the constraint to check the temperature would be something like CHECK(temp between 33.0 and 45.0). To check that no two user accounts share the same alias, a simple UNIQUE(user_alias) will do. More complex constraints can be enforced using plpgsql.

Then name your constraints. Then when the sql layer throws them, it's trivial to remap them to user readable error messages. It works especially well in newer postgres versions where the error reporting is much more detailed and the messages easily machine parsable.

nicksardo2• 12 years ago

SQL constraints are a given; always use them. However, it's best practice to also validate your data before it gets to the database for several reasons.

A) Depending on a database check constraint failure is analogous to using a try/catch when a pre-emptive check would work; it's dirty.

B) You're putting unnecessary load on the hardest system to scale.

C) Users/systems will get instant feedback with a client-side validation. Depending on the data, this could be an enormous bandwidth savings.

spacemanmatt2• 12 years ago

I am not sure I would say "best practice" but I would rank it as "often useful." I'll explain.

I disagree with your characterization of using exceptions when a check condition behave identically. It's not dirty, it's exception oriented.

I disagree with your characterization of the database as the hardest system to scale. What's hard is fixing a performance problem in the database after an organization has effectively put the schema in a cast by developing 30 apps/scripts directly coupled to the core of the (broken!) schema.

I DO however agree that client-side feedback is important. Validation on the client might include a lot of data used for prompting, or perform validation only the user-interactive application can do. This exceeds the role of the data constraints you might implement in your database for obvious reasons.

I'm not sure Q does more than duplicate the constraints I would put in my database to start with. There's a lot more to collecting input than enforcing closed-form rules.

bjourne2• 12 years ago

In real world situations you already need to go to the database to do validation. For example to check that a username is not already taken. And because of the race condition, you still need to be prepared to handle a duplicate-key integrity error when actually trying to create the user row.

islon• 12 years ago

In Clojure you can use schema (https://github.com/Prismatic/schema) which does the same thing without needing a separate language.

blambeau2• 12 years ago

The point here is to have a separate language. Data interoperability is about having multiple languages, not having only one to rule the world.

emidln• 12 years ago

I've done (limited) schema validation in Python. Schema exports reasonably well to EDN (it is just clojure after all). One can read EDN pretty well. I don't have a full-blown Clojure interpreter that would be necessary for arbitrary Schema, but I can handle the core types as well in Python (or random other language) as I could in Clojure/Clojurescript.

jdc0589• 12 years ago

I can't count the number of times I have wanted to set up re-usable model definition and validation, but haven't been able to find a good portable library that isn't part of something much larger I don't need. This definitely addresses a space in need of more options.

My answer was modlr (https://github.com/jdc0589/modlr), which I have been ignoring, but Q looks like it could do well. I'm not super pumped about the name though, conflicts with super popular promise library lots of people already use.

blambeau2• 12 years ago

I'm not very good at naming things. I'm aware of the name clash with many other tools and would really like to find another name. If you have any better idea, just let me know.

rkrzr• 12 years ago

How about Scheme! ...er, no wait... Or Grammeme, the well-known combination of grammar and meme (https://en.wikipedia.org/wiki/Grammeme)?

bane• 12 years ago

Oh dear, this might conflict with a SQL variant we cooked up at my old company called Q-Lang. Where you could write a SQL statement, leave bits of the WHERE clause "blank" and it would generate a GUI interface with drop downs and calendars and everything to let people fill in those bits without too much effort. Nothing terribly fancy, but it let you go from a working example SQL statement to an integrated query interface in the app in about 10 minutes.

It had lots of limitations, but most of the time it absolutely annihilated some of our competition's multi-month integration engagements.

hipsters_unite• 12 years ago

This is the second project named 'Q' that's been on HN in the last week or so. All cool ideas, but some Googling first by all parties would probably ease confusion.[0]

[0]https://news.ycombinator.com/item?id=7290655

lkrubner• 12 years ago

I am surprised that EDN is not making more progress as a data interchange format, especially now that it has fantastic validators and coercion libraries, such as Prismatic's Schema:

https://github.com/prismatic/schema

This is, by far, the most intelligent data exchange system I have ever seen. It offers a richer language of types than what is offered by JSON, it is as readable as YAML, it is far more concise than XML. And yet it seems unable to break out of the world of Clojure. But it deserves to be taken seriously as a data interchange language.

jared314• 12 years ago

EDN [1] has one flaw. It is not JSON. I will just repost the response [2] I received when I mentioned wanting EDN support in PostgreSQL.

So the same argument as (say) YAML, Lua tables or TNetStrings. Or, if you include binary representations, stuff like Thrift, XDR and ASN.1. You can embed a JVM in PostgreSQL and write the functions yourself (or use PL/Scheme I suppose), but I would be basically amazed if anyone decides to do it for you. JSON gets the nod because it's understood by billions of systems. Other formats are going to struggle.

While I do not agree with his position, it is shared by many people.

[1] https://github.com/edn-format/edn

[2] https://news.ycombinator.com/item?id=5548404

dj-wonk2• 12 years ago

Yes, JSON is "understood" in the sense that it is easy to parse, yes. That doesn't mean actually working with the results is any easier. (1) Without a schema, doing so can actually be harder. (2) JSON leaves out many useful data types and it is not extensible. So, perhaps JSON has "succeeded" by lowering expectations of data interchange. :)

nevi-me3• 12 years ago

Am I missing something fundamental here? I think in the home page Q shouldn't be compared to just JSON as they are in principle two different things. If I were to write a 'data language' or to extend an existing language to cater for data, wouldn't I write my validators in that said language? If I am correct, then I was expecting Q to be compared to the closest other 'data language', not be compared with a document/schema.

For example, if I was doing it in JS, I'd say: var validDoc = true; var Temp = function (t) {return t >= 33.5 && t <= 45.0};

then when I get the doc, I loop through my conditions, as such:

validDoc = Temp(doc.temperature);

// reject if !validDoc

From here, the benefit of Q becomes that it's much easier to do validations, instead of writing them from scratch, considering that my example above only returns a true || false, as it doesn't have the error handling to let me know what went wrong and where.

One could also achieve some of what Q is doing by using an ORM if said data is going to a database/document store.

So, HN, am I missing something here?

dj-wonk2• 12 years ago

Q is trying to do both data and validation, which are very different problem spaces. I think designing for both involves serious tradeoffs for the individual parts.

mrcactu5• 12 years ago

where is all the theory behind q-lang coming from? are there theoretical principles behind this new schema language?

snorkel• 12 years ago

I like it because you can practically use this to define a grammar for parsing a programming language, because:

1. Each data type can inline express its own value constraints (not just type constraints, but rather value constraints) which is key to source code parsing

2. With subclasssing you can combine data types into higher level objects that inherit all of the lower level value constraints. This is similar to how a recursive lexer drills down from a high level statement into the different parts of each statement.

3. The data type value constraints support sequences, alternatives, and unions, most of what a lexer needs to parse a language based on a grammar spec

With these ingredients you can actually parse code, and thereby write a code interpreter (or parser at least). You begin with a grammar rule (a high level data type) that defines a statement, and a statement is defined as a list of alternatives forms of a statement, and each statement form is defined as a sequence of keywords and supported value types, and so on down the language primitives.

... but I'm not sure if supports recursion, and thereby recursive descent parsing. Can you say something like "Sum = [Sum, Operand, Sum]" If not then you're limited to immediate values only and no nested expressions.

elwell• 12 years ago

Finally a new language that does something remotely novel. I like the blur between data and code here; (very) vaguely lispy.

Q – A Data Language

Related Stories