Glom – Restructured Data for Python

https://sedimental.org/glom_restructured_data.html

by mhashemi • 8 years ago

313 97 8 years ago

amock• 8 years ago

I haven't played with this yet, but it looks really handy. I deal with much more JSON on the command line than I'd like, so I think having both a single library and command line tool to reshape that data will make that much easier. I've used jq a few times, but when I want to move a little beyond what it does I usually end up writing a Python script. Hopefully this will make that transition smoother.

mhashemi18• 8 years ago

Haha, I'm all for console usage, but let me tell you, there's nothing quite like that feeling of moving a working spec into a dedicated application with exception handling, logging, etc. :)

tincholio• 8 years ago

It looks quite similar in spirit to Clojure's Specter library (https://github.com/nathanmarz/specter), and even seems to have a nod to it (The Spectre of Structure).

mhashemi18• 8 years ago

Oh nice! Total coincidence, I assure you. Still, I should read on this and add it to the analogy doc: http://glom.readthedocs.io/en/latest/by_analogy.html

Declarative data transformation generates a lot of comparisons (almost all of them great, though!).

hessammehr2• 8 years ago

Ah you beat me to it! :)

faitswulff• 8 years ago

This is really cool. Did you ever consider an API to do the reverse - to insert a value at a particular point in the data?

My interest stems from this issue[0] on the Ruby issue tracker to make a symmetrical method to Hash#dig (which does something similar to, but more limited than glom) called Hash#bury. The problem in the issue was that inserting a value at a given index in an array proved difficult and unnatural in Ruby, so I was wondering if there were other solutions out there.

Another question occurs to me - does glom only support string keys?

[0] https://bugs.ruby-lang.org/issues/11747

mhashemi18• 8 years ago

glom not only supports more than string keys, it also supports assigning to non-dictionary objects. That's a part of the API we're working on right now, actually.

As for the data insertion, mutation may be in the future, but for now glom only transforms and returns new objects. Definitely something to think about though, bookmarked! :)

hiccuphippo• 8 years ago

Check the updateIn, mergeIn, mergeDeepIn methods in immutable.js. Maybe even asMutable, asImmutable for mutation.

doublereedkurt3• 8 years ago

'string-key', T['string-key'], T[not-string-key] :-)

smittywerben• 8 years ago

Perhaps assoc_in or update_in from the Toolz library fits this?

https://toolz.readthedocs.io/en/latest/api.html#toolz.dictto... https://toolz.readthedocs.io/en/latest/api.html#toolz.dictto...

tathougies• 8 years ago

This seems like lenses for python... neat! I often use python to mess around with things, and almost always miss Haskell's lenses when doing so. This seems like an interesting solution.

hessammehr2• 8 years ago

This might have been unintentional, but I suspect "Spectre of Structure" and "Python's missing piece" refer to Nathan Marz's specter library for clojure [1], similarly touted as clojure's missing piece. I tend to agree in the case of specter, given the mind-boggling types of transformations that are easily (and simply) expressed in it (and often run faster than idiomatic clojure as well). Highly recommended if you ever need to work with deeply nested data structures.

[1] https://github.com/nathanmarz/specter

mhashemi18• 8 years ago

Total coincidence! Reading the README, Nathan and I are definitely on the same wavelength though. When I get a chance I'll add it to the analogies doc: http://glom.readthedocs.io/en/latest/by_analogy.html :)

mhashemi18• 8 years ago

What's really interesting is that we're approaching the same ideal state from different directions. Specter goes from Clojure's immutability to something more practical, from Python's super dynamic system to something more declarative and immutable.

lapnitnelav• 8 years ago

Looks really neat.

Striking a balance between ease of use / simplicity and powerful features is a tough exercise but you did well.

I can foresee the CLI being quite useful to do away with the run-of-the-mill sed / awk / grep [...] mess. Specifically for the less CLI inclined people out there.

icebraining2• 8 years ago

Can it be used bidirectionally, without having to repeat the work?

I have a need to transform between pairs of structures, in both directions, and ever since I found JsonGrammar (https://github.com/MedeaMelana/JsonGrammar2) I've been pining for a Python version.

mhashemi18• 8 years ago

It depends on the complexity of the spec, but we've already done some programmatic building of glomspecs, so for many cases I think the answer is yes! Once we feel out the patterns I think glom will gain some utilities for this purpose.

icebraining2• 8 years ago

Thanks, that's awesome!

neuronexmachina• 8 years ago

It seems like a subset of glom specs would be uniquely invertable. For example, the spec `{'c': 'a.b'}` could trivially invert to `{'a.b': 'c'}`. I'm not sure how you'd invert more complex specs which make function calls, e.g. sum or len.

boringg• 8 years ago

Looks slick!

jasonpeacock2• 8 years ago

I'm probably being dense, but I don't see a good description of the input data types supported - the CLI says "json or python".

It would be great to have clarification if this is JSON only, or supports other data structures, or parsers could be plugged in?

mhashemi18• 8 years ago

From within Python, all objects are supported by default. If you can parse it, you can glom it. You can even register additional behaviors for specific types to keep your specs tight: http://glom.readthedocs.io/en/latest/api.html#setup-and-regi... (example: http://glom.readthedocs.io/en/latest/snippets.html#automatic...)

The CLI is in a pretty preliminary state, usable but not as robust as it will be in a few weeks. It only supports built-in parsers (JSON and Python literals) What formats are you thinking? YAML?

jasonpeacock2• 8 years ago

Aha. Thanks! I understand better now, I didn't realize it was so general-purpose :)

nerdponx5• 8 years ago

It's a nice idea, but i never like writing what amounts to a DSL in strings in my code (yes, that applies to in-code SQL as well, although that's often unavoidable).

I prefer the `get_in()` method from Toolz: http://toolz.readthedocs.io/en/latest/api.html#toolz.dicttoo...

doublereedkurt1• 8 years ago

the only string parsing is 'a.b.c', which is mostly equivalent to T.a.b.c so you can completely ignore that capability

the very slight difference is that using T you must be explicit about attribute access vs key access whereas 'a.b.c' will try both

radmin• 8 years ago

My thoughts also turned to toolz.

Here's a example comparison:

glom

  glom(target, ('system.planets', ['name']))
  # ['earth', 'jupiter']

toolz

  list(pluck('name', get_in(('system', 'planets'), target)))
  # ['earth', 'jupiter']

mhashemi18• 8 years ago

The real advantage over toolz/get_in is in the breadth of types glom can support (http://glom.readthedocs.io/en/latest/api.html#setup-and-regi...), the extensive fallback behavior (http://glom.readthedocs.io/en/latest/api.html#advanced-speci...), and "T" specifier (https://sedimental.org/glom_restructured_data.html#true-pyth...), which allows performing object-oriented traversals and calls.

Check the post for more examples of where path-based access falls short of the mark!

(Sidenote: I find it very clunky that get_in() takes a "default" kwarg and _also_ a n"o_default" kwarg. Use a Sentinel object! http://boltons.readthedocs.io/en/latest/typeutils.html#bolto... )

abuckenheimer• 8 years ago

I feel the same way, although my instinct is generally to build a custom generator. Only costs a couple lines but is plain old python and quite explicit

    target = {'system': {'planets': [{'name': 'earth', 'moons': 1},
                                     {'name': 'jupiter', 'moons': 69}]}}

    glom(target, {'moon_count': ('system.planets', ['moons'], sum)})
    # vs
    def iter_moons(t):
        for planet in target['system']['planets']:
            yield planet['moons']

    sum(iter_moons(target))

would have to combine with `defaultdict`s if your nested data is only sometimes there though

mhashemi18• 8 years ago

That's a great example! Mind if I use it? ;)

jeremiahwv2• 8 years ago

For simple cases like this it doesn't even cost a couple lines as sum() can take a generator expression:

  sum(planet['moons'] for planet in target['system']['planets'])

typon• 8 years ago

Why is writing DSLs strictly worse than writing complicated transformations built on top of the limited constructs provided by the language itself? (You said never)

joshuamorton• 8 years ago

The issue isn't dsls, it's string dsls. Doing it within the language can provide stronger garauntees.

As a simple example, you get some syntax checking.

jkaptur2• 8 years ago

They’re pretty comparable when the strings are constants. The fun begins when the string is dynamically created.

jeremiahwv2• 8 years ago

I agree, I don't like the magic string approach (even if it is mostly just dot-notation attribute lookup). However, there is some good stuff here, and nested data lookup when value existence is unknown is a pain point for me.

In addition to the string based lookup, it looks like there is an attempt at a pythonic approach:

  from glom import T
  spec = T['system']['planets'][-1].values()
  glom(target, spec)
  # ['jupiter', 69]

For me though, while I can understand what is going on, it doesn't feel pythonic.

Here's what I would love to see:

  from glom import nested
  nested(target)['system']['planets'][-1].values()

And I would love (perhaps debatably) for that to be effectively equivalent to:

  nested(target).system.planets[-1].values()

Possible?

--- edit: Ignore the above idea. I thought about this a bit more and the issue that your T object solves is that in my version:

  nested(target)['system']

the result is ambiguous. Is this the end of the path query and should return original non-defaulting dict, or the middle of the path query and should return a defaulting dict? Unknown.

The T object is a good solution for this.

Too• 8 years ago

I prefer your nested() version. I don't see why your final example would be ambiguous either, if it always returns a defaulting dict which you always have to extract with .values() there is no ambiguity. This is similar to how Javas Optional<T> work. The problem could be that the type checker isn't good enough and writing code like if nested(target)['system'] == "lazer" could pass the type checker.

emidln• 8 years ago

Objects are overkill. Use an iterable of getitem lookups. They compose easily and don’t have much overhead (cpu or cognitive). From the library I use at work, inspired by clojure.core/get-in, it would look like this:

    def get_in(obj, lookup, default=None):
        """ Walk obj via __getitem__ for each lookup,
        returning the final value of the lookup or     default.
        """
        tmp = obj
        for l in lookup:
            try:
                tmp = tmp[l]
            except (KeyError, IndexError, TypeError):
                return default
        return tmp


    data = {“foo”: {“bar”: [“spam”, “eggs”]}}
    # find eggs
    get_in(data, [“foo”, “bar”, 1])

By using __getitem__ you naturally work with anything in the python ecosystem.

bilboa• 8 years ago

The other problem that T solves and nested doesn't is being able to reuse a spec. Once you have a spec, whether created with T or directly, you can call glom multiple times with the same spec, pass the spec to another function, tc.

rodrigoalviani• 8 years ago

Cool... it looks like... errr... javascript :D

sixdimensional2• 8 years ago

I had a quick look, but I didn't see filtering expressions, only shaping expressions. It seems like glom is more of a result shaper/mapper. Can you filter with glom (maybe with lambdas or something)? I could see the two going together quite well if you were "glomming" a big Python object.

mhashemi18• 8 years ago

Filtering is supported through the OMIT value: http://glom.readthedocs.io/en/latest/api.html#glom.OMIT (another example: http://glom.readthedocs.io/en/latest/snippets.html#filtered-... )

Lambdas and functions are always a safe fallback, but glom does its best to keep your specs readable and roundtrippable (gotta love a nice repr()).

sixdimensional2• 8 years ago

That's cool! Thank you for pointing it out!

zestyping3• 8 years ago

The writing style is just insufferable. Even the API documentation is littered with hyperbole and self-congratulation. We get it, you're proud of your work and extremely proud of yourself.

"as simple and powerful as glom"
"big things come in small packages"
"small API with big functionality"
"power is only surpassed by its intuitiveness"
"simplicity is only surpassed by its utility"
"shortest-named feature may be its most powerful"

For heaven's sake, give it a rest!

It's a big red flag about your priorities that when I go looking for a precise specification, I can't find answers to simple questions and instead end up wading through incessant marketing phrases. I tried, and I finally gave up halfway through the API doc. It might even be the case that glom is a good idea—but you're making it really hard to trust you as a source of objective information about it.

Show, don't tell. My advice to you: you'll generate more interest if you delete every congratulatory word on those pages and focus entirely on helping your readers understand what glom does instead of trying to sell it to them.

mhashemi18• 8 years ago

Hey Ka-Ping! Maybe I did get carried away :)

How I wish one could publish a dry document and expect people to read all the way to the bottom. I've published enough libraries to know that's not the case. glom's free software so it's all there, as "shown" as can be.

But referring you to the code wouldn't be very considerate either. Instead, here's this literate code version that I prepared in advance. Hopefully this will be of more help to you: http://glom.readthedocs.io/en/latest/faq.html#how-does-glom-...

liteye• 8 years ago

In http://glom.readthedocs.io/en/latest/tutorial.html#access-gr...,

After years of research and countless iterations, the glom team landed on this simple construct:

'years of research', 'countless iterations', 'glom team', really?

skosch• 8 years ago

Thank you for glom, and don't let those comments get you down.

Personally speaking – my heuristic is that maintainers who put effort into marketing copy (even if it's awkwardly exaggerated) are the kind of people who really want their users to enjoy the project, and that often predicts a low-friction experience. Keep doing what you're doing!

cscotti• 8 years ago

Dude, this looks really nice! I use nested stuff like that all the time - nested layers of flat data ;) - and glom (+T) do seem super appropriate and nice!

some_account• 8 years ago

Looks like a nice package. I will try it in my next project. The upcoming features sounded interesting. Keep up the good work!

brylie• 8 years ago

My only concern is that the documentation renders poorly on mobile. So I couldn't read the right half of the page.

zestyping3• 8 years ago

I was frustrated. I apologize. I do still believe the frequency and intensity of hyperbolic language is a real obstacle to understanding and appreciating your project, and I hope you take that feedback to heart. But to call you "extremely proud of yourself" was unnecessarily personal, and I'm sorry I said that.

mlevental3• 8 years ago

you know what's actually insufferable? taking pot shots at someone giving you something for free. either say thank you or move on. it's like yelling at your mom for making you breakfast in the morning: downright unseemly.

genidoi2• 8 years ago

if the condescending tone of the top comment is ignored it becomes valuable feedback. good libraries/api's (free or otherwise) do not need to use marketing buzzwords to sell themselves when a clear demonstration of the functionality is usually more than enough.

see the python requests library documentation for a good example

http://docs.python-requests.org/en/master/

kazinator• 8 years ago

Also, often it's better if power surpasses intuitiveness a bit. :)

jbritton• 8 years ago

There is also this Python lens library. https://github.com/ingolemo/python-lenses

I can't say how they compare, but they have some overlapping features.

codezero• 8 years ago

This is pretty neat and is something I've been thinking about for a current project.

Does anyone know if something similar exists in Java/Scala land?

zaptheimpaler• 8 years ago

I think this is similar to lenses in FP languages - check out Monocle for Scala.

harel• 8 years ago

This reminds me a little of the excellent dpath lib: https://github.com/akesterson/dpath-python

spedru• 8 years ago

I'm not really versed in the idioms/social mores of Python, so please take the following with a grain of salt:

This seems like it usefully solves a problem, but the invocation pattern is suspect to me -- Instead of "glom" taking the target for picking-apart plus a magic little bit of DSL, what if "glom" took a single parameter, the aforementioned DSL, and returned a function that would perform the corresponding search when called on a target? Even if Python or this package optimises away repeatedly searching (by the same spec|in the same manner), the convention the package prescribes is odd to me, right after the first few paragraphs of intro.

mhashemi18• 8 years ago

The big, classical school of Python definitely prefers top-level functions. Still, I definitely understand that aesthetic, and am on board with not using functools.partial to achieve it. So: https://github.com/mahmoud/glom/issues/14 :)

seanc• 8 years ago

Python regex library does this, optionally

daturkel• 8 years ago

Similarly, statistical distributions in SciPy can be used in "frozen" form (pre-parameterized) or in a more general form where you supply the parameters at the same time you are requesting some attribute it the distribution. Seems to me to be a situation where one is useful if you expect reuse, and the other is useful if you don't.

_x5tx• 8 years ago

I'm curious about how the author replaced DRF with glom

mhashemi18• 8 years ago

DRF still takes care of negotiating formats, etc., but it replaces the serializers. I'll see about getting an example in the repo, stay tuned.

staticautomatic• 8 years ago

Aside: are there any libraries for JSON or dict-like formats with xpath-style querying that are as quick under the hood as lxml is for xml?

bluemanshoe2• 8 years ago

Compare/contrast with pstar: https://github.com/iansf/pstar

agf• 8 years ago

It seems to me like the advantage to focus on here is the improved error / `None` handling, which will speed debugging and make handling expected edge cases easier. I've seen a lot of inexperienced developers tripped up entirely by this kind of data access, and seen plenty of experienced developers waste time debugging it because of the exact error cases the announcement references.

The `T` object, which the article describes as its most powerful, can be a useful pattern in some situations, but it's worth pointing out it isn't new or unique to this project.

The author says in another thread here that he first started working on the "stuff leading up to glom" in 2013. One older example, which is virtually identical though less complete, is this Stack Overflow answer I posted in 2012: https://stackoverflow.com/a/9920723/500584

I'd seen the general pattern even before that post, if not the Pythonic syntax. I don't think that it's much of an improvement over defining a `lambda`, so again I would say the thing to focus on is the improved debugability and the simpler, dot-notation-as-generic-attribute-or-item-accessor syntax. I think `T` is largely a distraction, or should be reserved for advanced users.

heavenlyblue3• 8 years ago

I would like to see the author debugging an application with 10 levels of object wrapping that had one of the middle object’s name misspelled.

Libraries like these shine only if they have brilliant tracing and debugging capabilities; otherwise are too easy to reduce to literally a single function.

doublereedkurt3• 8 years ago

http://glom.readthedocs.io/en/latest/api.html#debugging

affordances to add tracing prints, or drop into a pdb at any level

The Inspect specifier type provides a way to get visibility into glom’s evaluation of a specification, enabling debugging of those tricky problems that may arise with unexpected data.

Inspect can be inserted into an existing spec in one of two ways. First, as a wrapper around the spec in question, or second, as an argument-less placeholder wherever a spec could be.

Inspect supports several modes, controlled by keyword arguments. Its default, no-argument mode, simply echos the state of the glom at the point where it appears:

wildleaf• 8 years ago

Shameless plug: https://www.npmjs.com/package/safely-nested

Nothing special, glom just reminded me of it.

derefr• 8 years ago

My favourite approach to this so far, that I would like other libraries to copy, is Elixir’s Access protocol, which gives you e.g.:

    foo = %{key: [[1, 2], [3, 4], [5, 6]]}

    path = [:key, Access.all, Access.at(0)]

    get_in foo, path
    # => [1, 3, 5]

    update_in foo, path, &(&1 * 10)
    # => %{key: [[10, 2], [30, 4], [50, 6]]}

    foo
    |> put_in([:key, Access.all], “foo”)
    |> put_in([:new_key], “bar”)
    # => %{key: [“foo”, “foo”, “foo”], new_key: “bar”}

That third form is essentially the equivalent of building up a complex object through a series of mutations—but entirely functional.

wbolster• 8 years ago

in a similar spirit, i wrote "sanest", sane nested objects, tailored specifically for json fornats: https://sanest.readthedocs.io/

it does not have the exact same feature set though. my focus was mostly on both reading and modifying nested structures in a type safe way.

heavenlyblue3• 8 years ago

I think this project defines the first step at transitioning pip to a js-like repository of single-function modules. Hurray for kool-aid.

pdobsan• 8 years ago

There is already a well established Gnome project with the same name: http://www.glom.org It is a GTK+ front-end to PostgreSQL, similar to Microsoft Access.

Glom – Restructured Data for Python

Related Stories