Programming at the REPL: Data Visualization

https://clojure.org/guides/repl/data_visualization_at_the_repl#_graphical_and_web_based_visualizations

by tosh • 4 years ago

104 20 4 years ago

avindroth• 4 years ago

As a user of Common Lisp but not Clojure, where does Clojure dev environment stand with respect to something like SLIME? What unique features do Clojure dev environments have?

pjmlp4• 4 years ago

A closer experience to the old Common Lisp commercial IDEs, when using cursive.

https://cursive-ide.com/

sokoloff• 4 years ago

I’ve only dabbled with CL/SLIME, but Cider seems to me to be a very close analog (and is similarly very useful).

jpmonettas2• 4 years ago

Similar to SLIME, there is Cider (https://docs.cider.mx/cider/index.html) for Emacs.

There are a lot of debugging and data visualization tools for the repl with features I haven't found in Common Lisp like :

- https://github.com/jpmonettas/flow-storm-debugger (which I'm actively working on)

- https://github.com/djblue/portal

vindarel• 4 years ago

These look like nice tools. In CL land I think we can find some of their functionality in other tools than Slime, such as SLY, a fork of Slime with more features:

- Instrument any Clojure form

This would be Stickers: annotate any Lisp form with a sticker, run your code, interactively walk through the recordings.

https://joaotavora.github.io/sly/#Stickers

- Provide a GUI to explore your values and execution flow

SLY has an improved trace dialog: https://joaotavora.github.io/sly/#Trace-Dialog Also LispWorks' visual stepper is easier to use than SLY or Slime's. There is also an in-progress portable visual stepper for CL: https://zenodo.org/record/3742759

time travel stepper

ah. Nice job.

jb1991• 4 years ago

SLIME was used in the early days of Clojure but is long sinced replaced by Cider (also emacs) or other impressive tooling, i.e. add-ons for IntelliJ (like Cursive). I would say there is nothing that SLIME offers that was not replicated or improved upon in Clojure's tooling.

taeric• 4 years ago

I'm curious what you would count as improved upon?

roenxi2• 4 years ago

clojure.inspector is an interesting namespace, but in practice I haven't found it very useful.

- It is rare that this problem is the limiting factor. In Emacs, which has horrible problems rendering long lines, usually by the time I realise there is a visualisation problem my editor has died.

- There is a thin band between too-big-to-print and too-big-to-visualise where this tool is useful. Almost all practical data seems to fall outside this range.

That said, Clojure already has some good tools for solving this problem. Immutable data structures and good generic primitives that are easy to inspect make it fun to deal with inspecting objects.

capableweb• 4 years ago

- It is rare that the problem is the limiting factor. In Emacs, which has horrible problems rendering long lines, usually by the time I realise there is a visualisation problem my editor has died.

Sounds like the problem is your editor, not the Clojure namespace :)

Maybe try a real editor, such as vim/neovim ;) It gets a bit bogged down, even on my desktop computer, but at least it never crashes.

roenxi2• 4 years ago

There is nothing wrong with the namespace. I just never found situations where it was useful.

And I see I accidentally wrote "the problem" instead of "this problem". I fixed that.

yogthos2• 4 years ago

One thing that's really underappreciated in my opinion is the ability to use the REPL as a general tool for interacting with the system. A lot of devops tasks can be done very conveniently from the REPL.

For example, you can start a Babashka nREPL by running bb --nrepl-server and connect an Clojure editor to it. Then you can do a lot of fun stuff from there. For example, if you have osquery installed, then you can start querying your system for all kinds of stuff, get data back in EDN and manipulate it using standard Clojure functions:

    (require '[clojure.java.shell :refer [sh]]
             '[cheshire.core :as json])

    (defn osquery [query]
      (let [{:keys [exit out err]} (sh "osqueryi" "--json" query)]
        (if (zero? exit)
          (json/decode out true)
          (throw (Exception. err)))))

    => (osquery "select * from routes where destination = '::1'")

    => ({:hopcount "0",
         :interface "lo0",
         :mtu "16384",
         :type "local",
         :source "",
         :gateway "::1",
         :netmask "128",
         :flags "2098181",
         :destination "::1",
         :metric "0"})

I find I often just leave a bb nREPL running with Calva connected to it and some scripts to do common system tasks in it.

tosh• 4 years ago

https://www.osquery.io/

https://github.com/osquery/osquery

pjmlp4• 4 years ago

The difference is that each of those are their own little silo, instead of being part of a single language ecosystem.

pjmlp4• 4 years ago

I always mention that a shell is a poor man's REPL, based on REPL based OSes like the ones from Xerox and ETHZ used to offer their OS interaction workflows alongside the development experience.

Shells like Powershell and Fish are the closest to it, due to their integration with programming language and OS APIs.

yogthos2• 4 years ago

Completely agree, having an interactive programming interface to the system is superior to using a shell in practically every way.

0x445442• 4 years ago

Why do you think Powershell and Fish are closer than EMACS Eshell to the REPL based OSes?

agumonkey• 4 years ago

I think the time is right to revamp a basic shell. Something that generalizes pipes to async DAGs. The idea floats in the air between reactive objects graph in a browser or systemd services, docker composer ..

It would reseat the base os user interaction to something cleaner and leaner IMO.

gemstones• 4 years ago

This is great! For any SWEs that would be more comfortable in a Python REPL, you can do the same thing like this:

    import subprocess
    import json

    def osquery(query):
      return json.loads(subprocess.check_output(["osqueryi", "--json", query]))

    #>>> osquery("select * from routes where destination = '::1'")

    #>>> [{'destination': '::1', 'flags': '2098181', 'gateway': '::1', 'hopcount': '0', 'interface': 'lo0', 'metric': '0', 'mtu': '16384', 'netmask': '128', 'source': '', 'type': 'local'}]

I hadn't heard of osquery - thanks for pointing me to that!

Programming at the REPL: Data Visualization

Related Stories