Show HN: Early-stage Yahoo Pipes spiritual successor

https://www.pipes.digital/

by onli • 9 years ago

280 126 9 years ago

rjeli2• 9 years ago

Dead (could not connect to server)

onli17• 9 years ago

Please try again. It should be up (works for me and http://downforeveryoneorjustme.com/www.pipes.digital supports that). Though I forgot scaling up the droplet it is running on before posting here...

rjeli2• 9 years ago

huh, it fails immediately for me about 5/6 times on iOS Safari. weird

onli17• 9 years ago

I'd be interested to know which blocks and functionality is missing to support the use cases you used Yahoo Pipes for! And of course general feedback is always welcome.

insomniacity• 9 years ago

Nice. It's not very clear that you need to link the pipeline to the right-hand side. (I can't remember if you had to do that on Yahoo, it's been so long!). It's particularly a problem on a wide screen because the eye starts on the left, and I built my pipeline on the left and it took me a while to figure that out.

But well done on scratching an itch that lots of people have had!

onli17• 9 years ago

I can't remember if you had to do that on Yahoo, it's been so long!

I have a screenshot on my disk: https://imgur.com/a/TExPX - it had a bigger pipe output block at the bottom. I can see how the red circle alone is not enough, thanks!

RRRA• 9 years ago

Librarians are gonna go wild over this...

arjie2• 9 years ago

Nice. Used to love Yahoo! Pipes What's your data model look like?

onli17• 9 years ago

Every pipe is a recursive list of blocks, like a tree. Basically, a pipe has a starting block that produces the pipe output. To do that it calls the process function of its input block, which again run their input blocks, ..., the feed blocks serve as tree leaves.

pbiggar• 9 years ago

Cool product, and great to see this get traction! My new company is very much inspired by Yahoo pipes and making something similar. Would love to chat about your approach!

onli17• 9 years ago

Absolutely, I'll send you a mail

michaelbuckbee• 9 years ago

This is cool, but I feel like Zapier, IFTTT and their open source equivalents have moved far past Yahoo Pipes. How do you see this stacking up?

onli17• 9 years ago

I'm interested in the answer to this questions myself :) For me, this is about the ease of use of the visual programming interface, and the focus on feeds to enable the pipe concept. I do not see this in the same space as Zapier and IFTTT, but I might be wrong!

wocram• 9 years ago

As far as I Kno, IFTTT only supports simple if a: then b triggers, you cannot have multiple inputs or outputs. Do zapier or others improve on this?

bryanh3• 9 years ago

Yeah, Zapier has multi-step workflows https://zapier.com/learn/getting-started-guide/multi-step-za....

kilroy123• 9 years ago

Agree Zapier is pretty awesome and pretty powerful.

(I have no affiliation, just a happy customer)

LoSboccacc4• 9 years ago

They do a fraction of the data manipulation options available from pipes and they require connectors being custom developed, plus they limit you on result delivery, which cannot be pulled.

bryanh3• 9 years ago

This is mostly out of date -- Zapier has lots of ways to do this without custom connectors:

* Formatter -- https://zapier.com/help/formatter/

* Filters -- https://zapier.com/help/filter/

* Webhooks -- https://zapier.com/help/webhooks/

* Code (JS/Py) -- https://zapier.com/help/code/ or https://zapier.com/help/code-python/

Plus if you want -- those custom developed apps can be added by anyone, for free: https://zapier.com/developer/.

GordonS• 9 years ago

Do you know what the most fully featured OSS equivelants are?

I've been trying to find such a system that can read from RSS feeds and post to a Facebook Company Page, but to no avail.

mjn• 9 years ago

Huginn is one option: https://github.com/huginn/huginn

bergie• 9 years ago

Interesting, have to take a proper look tomorrow. We've done similar things with NoFlo (https://noflojs.org)

zeptomu5• 9 years ago

I've some experience in visual programming languages (both professional and in my studies) and I advise against the graph-form where you connect nodes and edges.

This does not mean that text is the only way to implement programs (and even if you connect nodes, you're still programming), but maybe a good local optimum is in-between, e.g. interactive shells like Jupyter's Notebook and the Mathematica interface. I know there is LabView, the Blender-Editor and AFAIK some Unreal-Engine tool that uses this model, but bigger programs seem really incomprehensible to me.

It's always easy to show-case 5 programs in these node-&-edge editors, but I do not think it is the best approach for visual languages, as one should not under-estimate the layout problems and how to represent information in boxes.

So, I am all for new ideas in visual programming, but I am not sure if the "free canvas" approach works.

phreeza2• 9 years ago

I wonder if there is a way to get the best of both worlds? What do you make of Blockly for example? https://developers.google.com/blockly/

zeptomu5• 9 years ago

I wonder if there is a way to get the best of both worlds? [...]

I believe so.

Furthermore high-level languages with syntax-highlighting (think Haskell, Python, Elm, etc.) in state-of-the-art text editors are in a better position than the "free-canvas" node-edge editors to provide the base for such a widely used language.

Blockly goes into the right direction, but has a childish touch and I do not think the keywords should have a background with padding.

fiatjaf4• 9 years ago

Blockly (and all the similar visual programming languages) are so awful! They're basically writing, but much harder.

"set", "repeat", "do"? What?

seanmcdirmid2• 9 years ago

"Patch and wire" can work in certain contexts, especially for data flow models. You are right they don't scale well (Burnett's scaling up problem), but if the programs are relatively small, they are completely manageable. And it does force you to keep your components relatively small and simple (using compounds to hide wiring for more complex designs).

jldugger2• 9 years ago

IMO, the reason the wiring model doesn't scale up is that most don't provide a way to create submodules (compounds?). I expect that decomposing a problem into a chain of smaller subproblems is a fine way to manage a dataflow.

Yahoo Pipes didn't really offer that, or any anyonymous stdin/stdout. So huge graphs were necessary for complex problems.

metachris• 9 years ago

Interesting thoughts, I would be interested to read more aboute this. Can you recommend any good references to start?

zeptomu5• 9 years ago

Literature is very scattered as the domain "Visual Programming" is pretty broad. There are general-purpose languages on one hand that really try hard to provide all the tools to build complex systems and domain-specific languages for ... well specific domains.

The general-purpose ones did pretty bad (at least I don't know of any complex system tools - say kernels - implemented in them), but there are some niches were some people seem to like them.

  - LabVIEW for control system prototyping
  - Shader-Languages for image processing (games via Unreal Engine and rendering pipelines like Blender)
  - Model-Editors for simulations and in engineering

E.g. in the GIS domain people often process digital aerial imagery via pipelines that can be represented in a similar fashion than this Show HN proposes (for example the model editors in QGis and ArcGIS). Here is an image of a model builder to process some geo-information: http://3.bp.blogspot.com/-9iOyUC8RHXs/UrM2ML5Qe9I/AAAAAAAABx... I am not sure if building this chain via this editor is actually simpler than learning some command line tools or basic Python to accomplish the same thing.

It makes sense that domain-specific ones do better, as they don't have to provide tools for stuff, which is uncommon. It is also easier to implement fold-like structures (say function f: A -> B) that process data with clear input and output than programs that do more IO (say f: A -> IO B). If you do complex IO, you have to introduce some concept of order and time in your system and it is not clear how to do this via the graph approach (example: while-loop), or just can be misinterpreted (edges/connections may symbolize ordering on one hand and data-flow on the other hand).

Assuming one should be able to program general-purpose systems (like all the tools we love: operating systems, databases, interpreters, web-servises, etc.), the solution might be not trivial, but I suppose it is more like the interactive notebooks than the graph approach.

Furthermore I think it makes sense to always have a semantically identically text-based language, as this just makes tooling simpler.

LoSboccacc4• 9 years ago

I don't think you remember how the yahoo pipes worked. It was a brilliant mix of graphs for flow control, premade boxes for all common data manipulation and scripting for that extra feature if needed

I miss them a lot and appreciate the effort here, but don't forget the visual model for data manipulation wasn't invented here but was testsd refined and proved functional by previous existing services

zeptomu5• 9 years ago

I don't think you remember how the yahoo pipes worked.

True, I learned about it when it was already dead, but thought the goal was similar to the stuff Zapier offers and I would really like to try it. Not to automate stuff, but to checkout the visual editor.

ryanoshea• 9 years ago

It was a brilliant mix of graphs for flow control, premade boxes for all common data manipulation and scripting for that extra feature if needed

I work for a startup that builds a dataflow app platform that fits that description nicely. We support gathering data from databases, files, APIs, etc., full graph-based flow control, strong .NET types on everything, tons of premade boxes for common operations, and support for C#/Python/R/F#/VB for custom data manipulation. Here's a quick example dataflow: http://blog.composableanalytics.com/2016/09/25/querying-data...

jacquesm• 9 years ago

Compared to putting together the net list of even a mildly interesting digital circuit interactively this is actually really easy. Not as easy as doing it with a text editor and some programming language or scripting solution would be but I think that Y! pipes was never aimed at the developers, much more at ordinary users and those are much more comfortable with such visual solutions than they are with text.

lobster_johnson• 9 years ago

There are some successful commercial products that use a visual graph model. MaxMSP [1] and Shake [2] (now discontinued) come to mind. You're right that it doesn't scale well as a UI, however.

[1] https://cycling74.com/products/max/

[2] https://images-na.ssl-images-amazon.com/images/G/01/software...

mastax• 9 years ago

Also: LabVIEW, Unreal Engine 4 Blueprints, and pretty much every Materials Editor I've seen.

JPKab4• 9 years ago

See here for a modern product that leverages the visual graph model and is rapidly growing in adoption:

https://www.alteryx.com/products/alteryx-designer

Theodores2• 9 years ago

Have a look at the ideas SGI 'explored' with 'IRIS Explorer'. This was ahead of its time and ahead of even SGI hardware. However you could layer up lots of interesting datasets in ways that would otherwise require programming at some other level.

http://www-sldnt.slac.stanford.edu/hepvis/Papers/Web/9/x1602...

kls2• 9 years ago

IIRC this became the basis of Cosmo Worlds which a company I was with, at the time, used extensively to build computer based training. It was an interesting mix of visual programming and scriptiong. It was based on wiring inputs and outputs into scripts. I really enjoyed working in Cosmo Worlds.

shiven• 9 years ago

Pipeline Pilot (Biovia/Dassault) - Deeply entrenched, industry-grade visual (+textual(PLSQL)) programming system. Depending on who you ask, it is either the bees-knees or a f*ing cancer.

supergreg• 9 years ago

Yahoo Pipes was brilliant. I never saw it as programming but as an advanced feed configuration.

robinhowlett3• 9 years ago

I work for SnapLogic (an enterprise visual integration product) - the "free canvas" works great for us with live preview of results/data; here's an example linking a third-party code review tool's webhooks with JIRA: https://i.imgur.com/gNr1gGD.png

All of our integrations can be converted into a API with a click of a button too.

dyeje• 9 years ago

What kind of API? Generated code or a web endpoint or something else?

JPKab4• 9 years ago

I work at a sort-of-competitor to you guys (honestly not sure we cater to the same user bases or not) but I actually like your UX. It's quite sleek. At Alteryx, we have a more traditional Node/Edge model, which probably occupies less screen real estate than your UX does for the same pipeline. However, it would be interesting to study if it's easier for humans to cognitively process a given style. The line of products you and I work in have shown me just how much research we still need to do on human processing of visual charts. UX professionals are just way behind on this across the industry IMHO. We've got some great ones here, but it's not easy finding them.

zeptomu5• 9 years ago

The image shows some of the problems quite well. E.g. the following things are not clear to me:

[0] I suppose Rhombi symbolize choice. But where are the branches? What if "Already Done" is true? Or does the else-branch enforce implicit fail-state somehow?

[1] What's the difference between parallelograms and squares?

[2] What's the scope of the "For Each Issue"?

[3] Why is the second output of "Copy Webhook" named ("output1"), but not the others? Are they implicit default output (maybe "output0")?

[4] What happens with longer pipelines? Do you wrap or have a scrollbar?

Furthermore the size of the modules have same size, so one gets into rendering issues (e.g. the names of some nodes don't fit the "boxes"). Sorry, I didn't want to sound harsh, I just think that visual programming is not automatically simpler, just because it doesn't use text (at least directly - one has also to name "modules"). You also don't use the "free-canvas" in its extreme form I described (where you can rearrange everything), as you've e.g. implicit edges between nodes, which is a good idea probably, as users get a feeling about the complexity of pipelines by measuring their length (which would not work if there are different lengths for edges).

sidegrid• 9 years ago

"I've" is not normally used where it is spoken as "I have". This is similar to people using "an" in front of a word starting with "h" where the "h" is pronounced. Not sure why people do this.

aaron-lebo• 9 years ago

Because language evolves?

flukus• 9 years ago

"I've" is not normally used where it is spoken as "I have".

Are you saying people don't use "I've" in spoken language? Because it see it regularly, "I've never" is even a drinking game.

regularfry• 9 years ago

"an" in front of "h-" is fine, although it's dying out. You'll usually see it where the first syllable of the "h-" word is unstressed.

mgkimsal• 9 years ago

would like to suggest a couple of "starter pipes" - samples to more easily illustrate some possibilities.

anonfunction• 9 years ago

I used to use yahoo pipes for a few twitter bots I had. There was an rss to twitter webapp that I hooked up and it was fully automated for years before it stopped working.

I didn't see the thin red circle on the right, which I now understand to be the output. I almost gave up before realizing my mistake.

hehheh• 9 years ago

Neat. Any chance of adding JSON processing where you can pick a key (ala `jq`[0]) and then combine it with RSS or other JSON or what have you?

[0] https://stedolan.github.io/jq/

onli17• 9 years ago

I plan on enabling filtering via xpath, and in parts jq is like xpath for json. But I'm not sure whether you mean something more with "then combine it with RSS or other JSON or what have you"? Merging generic input into a feed at a chosen element?

Side remark: Pipes uses https://github.com/feedparser/feedparser to normalize atom and rss feeds, and that gem just got support for the jsonfeed format. Untested, but worth a shot if you are looking for json input.

arjie2• 9 years ago

Should you feel like, there's also JMESPath for JSON http://jmespath.org

fiatjaf4• 9 years ago

I've made something[1] that used "jq for everything" in the past.

RequestHub was initially a way to connect webhooks from one service to API calls to another services, using jq scripting for all the customization.

Later I realized it could be used for processing anything (even text!), not only webhooks, but it was too late. Maybe someday I'll do a Yahoo Pipes revival that will just use jq for scripting inside the boxes.

[1] http://archive.is/nGyH3, https://github.com/fiatjaf/requesthub.xyz

gingerlime2• 9 years ago

Why did you discontinue requesthub?

reubano3• 9 years ago

If I understand you correctly, you want to be able to join the results of multiple sources? You can do something like that with a library I wrote, riko [1]. A simple example would look something like this:

    from riko.modules import join, fetch, fetchdata

    rss_url = 'http://site.com/rss'
    json_url = 'http://site.com/json'
    json_path = 'path.to.data'

    fetch_conf = {'url': json_url, 'path': json_path}

    rss = fetch.pipe(conf={'url': rss_url})
    json = fetchdata.pipe(conf=fetch_conf)

    joined = join.pipe(rss, other=json)
    next(joined)

You can see the docs for the `join` pipe here [2].

[1] https://github.com/nerevu/riko

[2] https://github.com/nerevu/riko/blob/master/riko/modules/join...

Toast_6• 9 years ago

Do you plan on supporting scraping content via css selectors/xpath/regex?

RandomBookmarks• 9 years ago

Or maybe integrating visual selection and OCR like Kantu: https://a9t9.com/kantu/scraping#ocr

Toast_6• 9 years ago

Looks cool, reminds me of portia.

https://scrapinghub.com/portia/

onli17• 9 years ago

I'm not sure. There are a couple of existing services that do that already, and which give you a rss feed you could then import here. But if it gets requested I would look whether it is doable in the constraints of this system - at least I remember yahoo pipes supported it, that's a plus.

Toast_6• 9 years ago

I remember yahoo pipes supported it, that's a plus.

Yup, pretty much why I used pipes to begin with. I think being able to scrape/manipulate/output data, while being able to keep it private, would be a fantastic service. Looks good so far!

reubano3• 9 years ago

You can check out my library riko [1, 2]. While it doesn't have a slick GUI, it does support most of the original yahoo pipes (including xpath [3] and regex [4]).

[1] https://github.com/nerevu/riko

[2] https://www.youtube.com/watch?v=bpn2G3TAAYY

[3] https://github.com/nerevu/riko/blob/master/riko/modules/xpat...

[4] https://github.com/nerevu/riko/blob/master/riko/modules/rege...

Toast_6• 9 years ago

Hey, just saw your comment. Looks like a pretty slick library; would something like this run in a jupyter/azure notebook?

mortadelegle• 9 years ago

Very interesting, I do think that visual programming tends to work better when constrained to a small niche of computing, that's the hypothesis behind https://github.com/AlvarBer/persimmon

onli17• 9 years ago

Wish I had that when I was still doing machine learning. Looks nice!

unityByFreedom• 9 years ago

Nice. Anyone else here ever use Ab Initio? It's an ETL tool for transferring data between databases and manipulating in the process. Seems very similar.

I've always wanted to recreate something with those ideas because it was a very efficient cross between visualizing code and writing it. This looks pretty similar although for different data sources.

yownie• 9 years ago

couple of code examples would be nice

infectoid• 9 years ago

Where I work we have a custom made system that was inspired by Yahoo Pipes. We are in the slow process of rewriting it.

Also this reminds me of and IoT solution I was shown recently.

https://nodered.org/

nthcolumn• 9 years ago

Pipes was awesome. Would it have killed them to just let it go free instead? Yahoo engineering did a lot cool of stuff.

scottmcleod• 9 years ago

I really loved Yahoo Pipes - Was ahead of its time if you ask me..Built some very cool automations on top of this. Basically was IFTTT and Zapier well before them.

beardicus• 9 years ago

I'm surprised nobody has mentioned Node-RED yet: http://nodered.org/

Node.js-based with a Pipes-like visual wiring interface. It's quite popular with the Raspberry Pi and Arduino crowd. Lot's of input and output plugins, and you can drop into Node scripting when necessary. I quite like it.

perryprog• 9 years ago

Node Red is very cool, especially for IOT type apps.

dundercoder• 9 years ago

I thought RSS had died a rather unfortunate and inconvenient death... am I wrong and it's still been kicking all this time?

Viper007Bond• 9 years ago

Still very much alive, just not quite as popular due to the death of Google Reader.

j45• 9 years ago

Pipes was a handy tool, hope this project keeps up!

Mister_Snuggles• 9 years ago

I'd love something like this that I can run locally. From looking at the comments, it looks like Node-RED and NoFlo are two possibilities.

I was intrigued by Yahoo Pipes a while back, but didn't want to invest much in it in case it was shut down. Sadly, that worry was well founded.

Xeoncross• 9 years ago

Using yahoo! pipes I created a lifestream application[0] some years ago that won an honorable mention for the 10k javascript challenge. It was nice to have a dynamic site that required no server work on my part.

Updating it for this would probably be a neat way to test out the RSS feed usage.

[0] https://github.com/Xeoncross/MicroStream

fareesh• 9 years ago

Unrelated to pipes specifically but along the same lines, I feel like app developers ought to conform to a pre-defined standard like schema.org for their respective content so that everything can be inter-operable in theory. That way if I'm using Microsoft Todo or Google Keep or whatever, the potential for Google Assistant or Siri or Cortana to add to whichever Todo App I'm using is already there.

What are the drawbacks behind something like this?

bsandert• 9 years ago

Vendor lock-in.

comboy• 9 years ago

No incentive. If your service is used through something else then you are easily replaceable and users don't spend time in your app.

edit: Just to be clear, I'd love that too. Hopefully it will start happening when everything is driven more by micropayments rather than ads. I'd imagine you'll have "gui companies" and "service companies".

onli17• 9 years ago

It's also really hard to do. I mean, imagine how one would achieve interoperability for a project like this. Placing blocks via an API from another service? The language/spec supporting that and everything else would be too complex.

But thankfully we have RSS for the data representation, which is a related idea, just the output side. The core of that idea is what enables this site in the first place.

houshuang• 9 years ago

This discussion is very interesting to us, as we are building a workflow system for collaborative learning (students might do an activity individually, the output of that activity is aggregated and transformed, students are grouped based on an algorithm that processes data from the previous activity, output from one activity is redistributed to another activity, etc). (quick demo: https://www.youtube.com/watch?v=HQ9AyzLOn3Q)

So this discussion about visual languages, data transformation etc, is very relevant to us. One thing we're working on is how to make data transformation more intuitive... Right now we are using JSONPath to enable selection of one field to aggregate on (ie. you have a form where people input ideas, and another activity that takes a list of ideas, so you can input a JSONPath for the field to get aggregated). However, looking at JMESPath (http://jmespath.org/examples.html), it looks much more powerful. Has anyone seen any examples of graphical interfaces for going from one data representation to another, with preview, selecting fields, aggregation etc?

technologia2• 9 years ago

Its interesting and I do applaud your work, but I'll probably just stick to Apache Nifi with its flexibility.

JPKab4• 9 years ago

How do you like NiFi? I work at Alteryx (we have a closed source visual data integration tool targeted at business analysts and data scientists), but was actually investigating NiFi quite heavily before I joined my current company. It didn't appear to me to have much adoption, but that was over a year ago. It certainly looked extremely compelling as a tool targeted at advanced users.

technologia2• 9 years ago

So I would agree that its not intended to be a tool that you could use like yahoo pipes. To the extent that it requires programming, I guess I would agree it requires an advanced user.

Putting that aside, I would say the Nifi is easy to work with, performant with large datasets, and I personally love the notifications I get when tasks get screwed up, etc. I would not suggest using this to expose data flow to your user as its UI is not really geared towards folks who are programming averse (to a certain extent).

laretluval• 9 years ago

Doesn't seem to work on mobile Safari.

jonincanada• 9 years ago

I archived all the public yahoo pipes as Python modules -- before it was shut down #oldhardrives

onli17• 9 years ago

Wow. Did you do anything with the data? It would be very interesting to see for what it was really used!

jwives• 9 years ago

I'm not sure what the licensing on that would be; would it be possible for you to share that archive?

timvdalen• 9 years ago

Looks interesting but the editor didn't give me any feedback/output and the login email never arrived.

onli17• 9 years ago

So far you need to log in to save the pipe and then see the output.

Shoot me a mail to admin@pipes.digital (or to the one in my HN profile) if you want to try to sort out the non-arriving email. Or if you have a gmail address, log in with that, portier (the login system) supports the OIDC flow for gmail.

hehejo• 9 years ago

What about Talend ETL?

You connect and configure various compontents (Input, Output and Manipulation) to achieve you desired dataflow. It's compiled to Java and really Vera versatile.

Show HN: Early-stage Yahoo Pipes spiritual successor

Related Stories