geokon

Where is it actually explained how it works..?

Usually these kinds of systems either don't scale dynamically or have caching issues. The first example, a spreadsheet, is "easy" because there are a fixed amount of cells to track. A GUI can be a lot harder (imagine sub windows and sub-sub windows dynamically popping up and tracking some redundant and some unique "computations". Entities can appear and then be removed at random). Though the wording carefully says "constructing views" so maybe it doesn't handle dynamism

A cemetery that charges an entrance fee..

Thanks :)

In Paris they use the corpse of their heroes like Voltaire and Napoleon to flease tourists of their Euros

Using a tomb to expand our knowledge of the ancient world seems like small potatoes in comparison. Overton window is already far off

are there any "open source" efforts to do distillation? Like some place one can submit one's anonymized chat logs? So they can be pooled and used as an open training set (similar to OpenCrawl)

Sorry for the delay, I've been thinking about how to respond :)

For me it is.

if you can have a running example I'd love to see it

I want some good guidelines

Yeah, I totally understand. I don't have a very clear mental model myself as of yet.

try to pack related entities into their own nested collections

I'd try to just be wary of trying to map OO concepts to Pathom. I get the impulse, but my gut feeling is that this won't be fruitful or will make things muddled.

you can make bridges between that resolver

tbh I haven't had to really use bridges. The few times I did, I then rearchitected till they disappeared. I do sometimes have different ns's using keys from each other though .. which feels wrong. No clear picture here yet.

My mental model at the moment is a bit fuzzy but goes like this:

Pathom is fundamentally a web of resolvers. The resolvers are best thought of as functions that can only be run once. This is the fundamental constraint you're working under - but you get a bunch of benefits - and I wouldn't try to fight this fundamental paradigm

1.

How do we deal with resolvers that conceptually have the same I/O keys? Say we have two resolvers:

    :filename -> [readFileJPG] -> :image
    :filename -> [readFilePNG] -> :image

You have two choices/branches. Pathom can't make that choice for you:

In your systems either the provider has to decide:

    :filenameJPG -> [readFileJPG] -> :image
    :filenamePNG -> [readFilePNG] -> :image

    :filename -> [readFileJPG] -> :imageJPG
    :filename -> [readFilePNG] -> :imagePNG

But bear in mind that Pathom is a "compile time" tool in a sense. If it's something determined from a db query.. then it's not part of the program control flow and needs to be decide at "runtime". It's not a Pathom problem! It has to happen inside the resolver code and it can't be part of the web of resolvers.

In that previous :fat-pack example, the equivalents would been :id-FAT (provider) or :customers/phone_number-FAT (consumer)

2.

Okay, but say I have readFileJPG and readFilePNG provided by some library/module/ns. I can't modify it's I/O keys. Then hopefully the keys are namespaced:

    ::jpg/filename -> [jpg/readFile] -> ::jpg/imageJPG
    ::png/filename -> [png/readFile] -> ::png/imagePNG

3.

Okay, but say I have a pipeline that is filetype-agnostic and for some reason :filename are coming in from "outside" (maybe user is typing it into a text box) and the output really just wants :image keys for some subsequent processing. What are your options?

- let the provider/upstream decide: The :filename is coming in from somewhere, but it know the file type a priori. So it can supply a flag that the resolvers require - ex: :is-jpg , :is-png

    ::filename + :is-jpg -> [jpg/readFile] -> :image
    ::filename + :is-png -> [png/readFile] -> :image

    ::filename -> [jpg/readFile] -> {:jpg [:image]}
    ::filename -> [png/readFile] -> {:png [:image]}

- The first feels fragile (you live in fear of it taking a wrong branch), esp if you have "default no-flag" case like in your db request example.

- The second creates a packed thing that needs to be unpacked. (this is what I showed with :fat-pack). I actually think this is the worse solution b/c it breaks composability. The packed space is completely isolated from the wider soup of keys, which can severely limit what you can do (as I'll illustrate).

EDIT: Part 6 I show the better solution for the consumer

4.

The second solution also conceptually clashes with another tool in the toolbelt. Nested inputs and outputs have a fundamental, completely unrelated, different usecase. They allow you to create a mini-context in which you can re-execute your Pathom resolvers. So if you have two images in your application, you can read them both in. These two transformation can co-exist:

    {:profile-pic [:filename]} => {:profile-pic [:image]}
    {:background-pic [:filename]} => {:background-pic [:image]}

    :image + :username -> [addUsernameOverlay] -> :image-with-username

     {:profile-pic [:filename :username]} => {:profile-pic [:image-with-username]}

So for the consumer side solution:

    ::filename -> [jpg/readFile] -> {:jpg [:image]}

    ::filename + ::username => :image-with-username [XXXX - doesn't work!]

    ::filename + ::username => {:jpg [:image-with-username]}

    ::filename + :username -> [jpg/readFile] -> {:jpg [:image + :username]}

6.

After chewing it over, I think the best solution is to pack everything when possible:

    {:jpg [:filename]} -> [jpg/readfile] -> {:jpg [:image]}

    {:jpg [:filename + :username]} -> {:jpg [:image-with-username]}

When to create these mini-contexts is not something I have a clear mental model for. It's related to the fundamental downstream output

Okay - actual code is good :))

But I can't replicate the behavior. Here is the code:

https://github.com/kxygk/ednless/blob/master/pathom-prectest...

I didn't know what jdbc was so I just plugged in dummy values - but I think I'm matching your logic one2one

Let me know if you can tweak it to have the weird results trigger

At a high level I'd say a couple of things stand out.

- Your resolvers take in an id and return an id.. That sets off a bunch of alarm bells for me. I wouldn't ever do that. There is no good reason to have that, even if the values are identical (if the values change then that's even more dangerous). I can't point to exactly what will go wrong, but you're exacerbating the problem of having multiple resolvers providing an input. Worse yet, in this example you are guaranteed that they are all running. So where is it going to take the ID from..? I don't even know

- The :fat-key isn't doing anything here. If you wanted to do the nested request, it'd be a solid way to guarantee the fat branch is always taken:

    (pco/defresolver get-user-with-customer
      [{:users/keys [id]}]
      {::pco/output [{:fat-pack [:users/id
                                 :users/email
                                 :users/customer_id
                                 :customers/id
                                 :customers/billing_number
                                 :customers/phone_number]}]}
      (println "get-user-with-customer FAT triggered")
      {:fat-pack {:users/id                 66
                  :users/email              "66@66.com"
                  :users/customer_id        id
                  :customers/id             id
                  :customers/billing_number 666
                  :customers/phone_number   6666}})

    (def env
      (pci/register [get-user
                     get-customer
                     get-user-with-customer
                     user-customer-bridge]))

    (p.eql/process env
                   {:users/id 1000}
                   [{:fat-pack [:customers/phone_number
                                :users/email]}])
    ;;{:fat-pack {:customers/phone_number 6666, :users/email "66@66.com"}}

    (pco/defresolver fat-eater
      [{:keys [fat-pack]}]
      {::pco/input[{:fat-pack [:users/id
                               :users/email
                               :users/customer_id
                               :customers/id
                               :customers/billing_number
                               :customers/phone_number]}]
       ::pco/output [:response]}
      (println "fat-eater triggered")
      {:response (str "yum, just ate: "
                      (:users/id fat-pack))})

    (def env
      (pci/register [get-user
                     get-customer
                     get-user-with-customer
                     user-customer-bridge
                     fat-eater]))

    (p.eql/process env
                   {:users/id 1000}
                   [:response])
    ;; {:response "yum, just ate: 66"}

EDIT: At the bottom of the .clj I added a bunch of other tests for the pipeline branching (the A->B->C thing)

- Adding an XXX resolver to override step B worked when "forced" with an extra input key.

- Adding an YYY resolver to override step B by outputting an extra key ends up always overriding B (whether you request the new dummy output key or not!)

I'm guessing here.. but it's probably b/c YYY has the same input requirements as B, but YYY outputs two keys instead of one - so it's preferred.

In a way both seem useful. The first lets you use a flag to select when you take a branch. The second make you always override the branch

It doesn't seem petty at all. These are the fundamental primitives of how you want to decouple and organize code. You have to look at them through small examples.

In the first case, the situation looks largely the same. I mean you can either uses the same nested inputs strategy but have a special key that triggers the fat-query resolver. Something like :employed-user-id.

The other alternative is using nested outputs. You have the resolver returned a keyed bundle. Something like {:fat-request [:users/email :company/phone_number]}. The downstream resolver then consumers a :fat-request and unpacks it using nested inputs.

As for the second example. I'm a little confused on some of the specifics. Writing out the resolver mappings more explicitly.. I'm inferring this is what's going on:

- GetUser - :users/id -> :users/email :company/id

- GetCompany - :company/id -> :company/phone_number

- GetUserWithCompany - :users/id -> :users/email :company/phone_number (<- this is a shortcircuit bypassing :company/id)

From this I can see why the first request triggers number 3. You could of course make the third resolver return a bundled output which may simplify things.

The second request.. I get a bit confused here.

1. I'm a bit confused about the bridge. Maybe there's a typo? Or are you saying `customer` is some completely separate namespace that's interfering with the behavior? In the text you seem to imply you're aliasing company/id and user/id.. but that would be a bit crazy :)

2. You then say "and use the cached result to get the rest of the data". So you have cached resolvers and there is memory from previous requests?

Is it internally, after the first request, remembering the :company/id associated with this :users/id? So it triggers the first two resolvers instead of the third one (but why was the :company/phone_number and users/email not in the cache?)

Imagine working with 12 people and having hundreds of tables.

Yeah, I think at this point in time there is no sense of best-practices or common programming patterns. From reading the docs and issues, I don't even get the sense Pathom's author has a good sense of how best to hook things up. So we're in the `goto` era of using Pathom and you can end up with a messy web of resolvers. Playing around with the system.. I'm left feeling like there is some emergent logic and ways to organize code. But maybe there are corner cases where it all breaks down and you start to miss imperative programming. My gut reaction is that if I see two separate paths on the same inputs/outputs .. then I'm immediately thinking - "can I redesign my system to avoid this?"

As a simple thought experiment. Say you want to inject stuff in to a pipeline (So some A -> B -> C -> D becomes A -> B -> ZZ -> D). Pathom's author suggests using the Priority attribute. https://pathom3.wsscode.com/docs/resolvers/#prioritization

But you have other alternatives...

- You can also add a dummy key :take-zz-branch. You have resolverC that takes :B and you have resolverZZ that takes :B and :take-zz-branch. Precedence rules .. should .. make it take the branch (unless it for some reason requires fewer inputs?).

- You can make resolverZZ output some dummy key :zz-was-run. If you request :zz-was-run then I think it should also take the branch? (or maybe it runs both branches).

Maybe there are other methods I've not considered. But at this point I'm not clear which method is best!

great example.

Some of this is out of my bailiwick, but on a high level I agree with you. I think your intuition is right. If you have behavior that's dependent on priority, this is a code-smell. It feels like you're just sort of #yolo'ing and hoping the right resolver is called. So far.. In these situations I usually pause and reconsider my architecture. There are probably several solutions here.

(Do bear in mind that I'm still learning the Pathom kungfu here, so I can't guarantee these are the best solutions..)

1.

So in your example the first step in isolating the behavior would be to re-think of it as three keys

- ::employee-ID

- ::company-ID

- ::employee-company-id-pair

and make more resolvers

- ::employee-ID -> ::employee

- ::company-ID -> ::company

- ::employee-ID + ::company-ID -> ::employee-company-ID-pair

- ::employee-company-ID-pair -> ::employee-company-pair

You can then just request an ::employee-company-pair and it should be disambigious. The problem is that you've now have a dense pair that doesn't hook back up with the rest of your resolvers. But this can be addressed with ...

2.

isolating behavior using "nested inputs/outputs". They allow you to go from a soup of keys to a narrow subset

Again:

- ::employee-ID

- ::company-ID

- ::employee-company-id-pair

first you can just have the original three 1-to-1 resolvers

- ::employee-ID -> ::employee

- ::company-ID -> ::company

- ::employee-company-ID-pair -> ::employee + ::company

At this point, as you illustrated, you have a bit of a priority issue. With a ::employee-ID and ::company-ID keys it's unclear which path is taken.

The trick is to now use nested input to disambiguate things.

You make a resolver that returns the results wrapped in a key (nested output):

- ::employee-ID + ::company-ID -> {::packed-request [::employee-company-ID-pair]}

The "consumer" resolver that only wants that efficient db call has on input a ::packed-request and just "unpacks" the request using nested inputs. Furthermore on input it will directly requests {::packed-request [::employee ::company]} and the engine handles the ::employee-company-ID-pair -> ::employee + ::company conversion. This nested input scope (ie. the inside of ::packed-request) doesn't have ::employee-ID and ::company-ID keys, so the request is always unambiguous.

The Pathom docs could be a bit more clear on this. They just show the basics and unfortunately don't walk through these tricks. But you can be explicit about both input and output map shapes and the engine uses these to do conversions. This allows you to narrow the set of inputs. So here one resolver outputs a {:packed-request [::employee-company-ID-pair]} and another takes a {::packed-request [::employee ::company]} - and the conversion is implicit. The engine finds only one resolver that returns a ::packed-request and it sees that it internally it will have a ::employee-company-ID-pair key. It then looks for a path from ::employee-company-ID-pair to the requested ::employee + ::company pair and it finds the corresponding resolver(s). Sometimes you need to forward other keys into this inner context, but you just provide them in parallel to the ::employee-company-ID-pair - it's all explicit.

I'll admit.. this looks weird. As I said elsewhere.. it's a real paradigm shift in how to think about and organize code. But I find after some adjustment it's actually worked really nicely for me so far.

I'll be honest, I work in a very different area (scientific computing) so my code is a lot more exploratory and I don't ever deal with DB access for instance. If you have a very stable interface and clear objectives then coupling isn't really a concern b/c there won't be anything to refactor and extend.

avoiding intermediate keys at all can be helpful for performance, and graph querying encourages people to break their queries into small, atomic units that can fire in any order

EDIT: Reading the other comments, I realize here query is a DB query and not a EQL query.. so nevermind :)

I'm a bit fuzzy on what you're saying, but I think you may be misunderstanding an aspect (I could be wrong here). You typically have one complex top-level query and the engine builds the sequence/graph of resolvers that need to be run to derive the requested query. In that graph key values can be reused and branches can be run in parallel. You don't run a series of small queries and manually build up anything.

In my limited experience the order in which the resolvers are run is pretty clear (unless they're independent branches of the graph being run concurrently) and if you have a non-branching pipeline there isn't really any incentive to break it up. From a performance perspective I'm guessing you mean in terms of DB access? Because calling a series of functions or a series of resolvers is going to be quite similar performance wise - though you have some engine and destructuring overhead (can be significant in tight loop situations).

Is that exceptionally harder to do with regular functions though?

It's hard to make a general statement here b/c it really depends on how you've set up your functions. But yes, generally if you are just playing with functions it can be harder to plug in a different "backend" or step in the middle unless you've somehow planned for it. Is it very hard? Generally not super difficult - but you generally need to refactor code to make it happen and explicitly handle the branching logic - so the code usually gets uglier

If you want to do a mock or try injecting some step, with resolvers you can do that without touching your code

that I understand. providing that compatibility guarantee is extra work for you

It's always nice talking to you about these things. Thanks again :)

Lots of great thoughts

As for function memoization, I previously tried this workflow and after scratching my head about it, I think it's just not possible to make it scale properly (in the sense of making a library of resolvers/functions where you don't know how they'll be used exactly). The memoized function has no way to know how often it's called. It can be called 2 times, or 2000 times. So it's unclear how large its cache should be and there isn't a clear mechanism for when to flush the cache. I couldn't find a good mechanism to safely use it. In the Pathom model .. as far as I understand you just don't need to worry about that since the outputs are "cached" in the context of a query (or an inner input) and discarded when you're "out of context".

Since often you have many similar requests it can make sense to add a layer of memoization a the top level to remember the last request (cache of size 1) but otherwise it should scale okay. Though I'm sure it's not difficult to create pathological cases where it probably doesn't work and you end up recomputing stuff.

I think caching is an unresolved problem

Oh sorry, I kind of meant it the other way around. You start with biff.graph and you'd swap in Pathom if the featureset or performance wasn't adequate. It makes sense that since you support a subset that it doesn't work the other way around! It might make sense to reinvent the wheel if there is a clear gain - but if there isn't any big innovation going on in the library interface, it's generally nice to keep the same interface if it's easy enough to do - but that's just my opinion haha

And yeah, now that I have a larger application with Pathom.. I should retry Viz too :))

And that's very cool you're taking error seriously. Sorry, I missed it when I looked at the rep the first time! Thanks for your help with Pathom a few months back (kxygk on Github)

the tweaks I made to e.g. `defresolver` were really just a side thing

If it weren't for those, would Pathom be a drop-in replacement? Or is there different logic?

I'm a bit of a beginner with this all myself, so yeah, I get how it's a bit of a black box :)) Think it's very cool you re-implemented it.

To give some more background on the motivations, an issue I've had sometimes with Pathom is figuring out what's going wrong when my queries don't give me the results I'd expect

I'm curious in what scenario PathomViz is not giving enough info. I had a lot of trouble getting it working tbh (never got the nested query working) but from the docs it seems like it should give you all the information you'd need to reason back to why you get a particular output. Reimplement all this debugging stuff seems potentially a lot of work - but maybe I'm wrong. More tools around the diagnostic output https://pathom3.wsscode.com/docs/debugging/ is something I hope to explore eventually.

You can use regular functions, but there are several things you lose:

- intermediate keys are not recalculated if they're used across different resolvers. This means you basically never need to manage caches of precomputed results. So if you're calling `my-func` on `input-a` everywhere, you don't need to do all the ceremony of computing it once, storing it somewhere, and then passing it around to everyone that needs it. It's all just handled automatically. Code simplifies greatly

- It's much easier to "inject" lower-level steps b/c resolvers are essentially declaring an interface. If you suddenly don't like your interface and want a new interface, then you make a new interface and a bridging resolver. Refactoring is much easier. If you want to introduce an entirely new input format that usually just involves adding a single new resolver that outputs the inputs to your system (at whatever part of the pipeline you want). While with a pipeline of function calls it's generally more messy. It hard to make a generalization here b/c it depends on how your functions are organized.

- With the async engine you can automatically resolve branches concurrently without having to manage or think about it. You get a lot less stalls in the code.

I haven't really hit an "exploding their fetches" scenario personally. Things like optional inputs and resolvers that rely on precedence rules are generally a bit of a code smell and are usually points where I start to think about how to reorganize my code

It's very cool you managed to make a mini Pathom - esp in so few lines of code :))

But the end result looks almost identical? Resolver declarations are a bit reorganized and look a bit cleaner - though you could do that with a wrapper around Pathom. Why not fork Pathom and just make some QOL adjustments?

If you are okay only having a subset, then I think you could streamline Pathom quite a bit more.. Off the top of my head:

- adding and "registering" resolvers in to an environment is always annoying. To me this feels like it should be abstracted away and you should never has to manage this stuff. Each time you add a resolver you have to copy the new resolver name, scroll down to the bottom of the file and paste it in to the resolver list. Stale resolvers floating around in the ns and forgetting to register resolvers regularly leads to weird scenarios where you're wondering why something isn't resolving. I think the user shouldn't really have to think about any of this.. the environment should be constructed automatically by the engine. Scan the namespace and register ever resolver.

I get that it'd be not as flexible this way.. but I've never had to make multiple environments in one namespace.

- You should be able to safely reregister resolvers. This happens when you try to decouple sub-systems that depend on common resolvers. Ex: I have some utility resolvers that do some format conversions. If I add them to the environments of two namespaces, then those two namespaces can't be registered in a parent namespace. It's not a dealbreaker, you just register all your ns environments all the top-level and do all your queries there. But you can't add inline test queries in these lower level namespaces and it sort of breaks the decoupling (esp if it's across library boundaries).

- The Pathom errors are actually pretty good once you know how to read them (but there is a ton of visual noise). My guess is it's going to be a challenge to get to the same level in a rewrite. It feels like there is room for improvement here, but I don't have concrete ideas. Maybe a ASCII diagram of the chain of missing keys? There is Pathom-Viz, but from what I understand it doesn't handle nested queries (which in a not-toy program will be basically all your queries)

I've been rewriting a codebase using Pathom resolvers and it has been extremely fun and has made me really reexamine how I organize code. Without being hyperbolic, it's really a new coding paradigm. You get some extreme decoupling and it allows the engine to automatically maximizes concurrency.

My feel would be that 2.4 has the advantage of being much longer range and there are plenty of sources for 2.4 - if you wanted to look at back-scatter and not just sources of emission. The primary demo of this piece of equipment is looking at radiation sources, but as you guys show there are plenty of other possible applications. I feel like since 2.4Ghz stuff has been around for ages.. the stuff should be much cheaper? Just a guess though :) Would be curious to know what the reality looks like

And cool to see you guys are from Santa Barbara. Lots of relevant talent there :)

One thing to note that's maybe less obvious is that you can destructure some keys with the check and others without. This makes the function interface a bit self-documenting. At a glance you see that the username is required and other parts are maybe not.

    (defn my-function
      [{:keys! [username]
        :keys  [firstname
               lastname]}]
      (do-stuff username
                firstname
                lastname))

Feels like throwing an error on nil would have been better/simpler? But I'm sure there's an angle I've not considered

looming requirement that all Android developers register themselves centrally

Does this somehow also apply to developers in China? Are Chinese OSs (Vivo/Honor/Oppo/etc.) entirely forked off of Google's Android?

Is the solution to just a Chinese phone without the Play Store?

An FDroid desktop client that adb installs APKs would actually be lovely. I pretty much exclusively use FDroid, but I gotta say I unfortunately find all their frontends to be rather buggy and with very little user feedback when things break (repo updates are hard to observe, downloads hang, updates mysteriously fail)

There are plenty of Icon packs suitable to all aesthetic preferences. Just nobody is going to write a blog post ragging a some Icon Pack b/c if you don't like it then it's trivial to change to a "better" one (that said I still think the arguments in the blog post are interesting and worth considering)

To the blog's point - many KDE Icon Packs have non-uniform shapes (ex: I'm currently using Newaita)

"Correct" modern C++ eliminates whole classes of problems. You can of course still write C code, but no one would merge that in to their codebase

Theyre both complicated languages in their own way :)

I think there are very few places where C makes sense and C++ doesn't. Its mostly legacy things like the Linux Kernel, or more aesthetic projects like demoscene or suckless. Where its easier to agree to write C instead of trying to agree on what subset of C++ to use (even then its usually C with a mishmash of C++ ergonomics)

That said, leafing through the first chapters of "Expert C Programming" should dissuade anyone of the idea that C is a simple language. It'll leave you amazed anyone's been able to write working programs in it

Ah okay. That's a good insight! and something to keep in mind. There is a reality distortion lens to it all in the end. As you say, people want to explore the frontier of technology

Are pretty much all surface mounted components fine with this?

I wanted to waterproof a micro devboard by submerging it in mineral oil. I was worried the board may delaminate or components would turn to goop

Seems like a fair play by Alibaba. However, is there any "open source" attempt at crowdsourcing distillation?

Like some place people can submit their chatbot convos so they can be aggregated?

Like an equivalent to OpenCrawl but for mining the models. It feels like thatd be a richer dataset than Alibaba generating queries and feeding them into Anthropic/OpenAI models

PS: Does anyone know how when companies distill each others' models the synthetic queries are generated? Im just assuming theyd be worse than organic ones

Where do you find more nuanced discussions?

I actually don't understand the tearing they're talking about. If the fields are final then you can't modify the Value Type anyway? And a simple write-lock bit for fat Value Types would solve everything while maintaining most of the performance benefits (both on read and write)

Is there some reason there isn't simply a write-lock/semaphore on Value Types that are over 64bits? The overhead should beat pointer-chasing. I mean maybe someone wants to concurrently write to values from different threads with no coordination, but that's not super common. As you illustrate, having "fat" Value Types would open up a lot of potential.

In the current setup will a Pair Value Type be a compiler error, or will it silently just have bad perf?