Yahoo Pipes End-Of-life Announcement

http://pipes.yqlblog.net/post/120705592639/pipes-end-of-life-announcement

by michaelhoffman • 11 years ago

361 177 11 years ago

pipes.yqlblog.net

michaelhoffman• 11 years ago

I figured this was going to happen years ago. Any suggestions for a replacement?

atourgates• 11 years ago

Platforms like IFTTT Zapier have worked well for me, though they're certainly not direct 1:1 replacements.

Concours6• 11 years ago

Sure, check out http://www.feedsapi.com , it's probably the best replacement at this point, iit's however not a 1 to 1 replacement.

cloud36• 11 years ago

I wonder how many people still use pipes?

WalterGR2• 11 years ago

One data point:

I used it for the first time 2 weeks ago.

Meetup.com has calendar feeds. You can get a calendar feed for all the events you've RSVPed "yes" to. You can get individual calendar feeds for each group's events. But there's no way to get a calendar feed of all the events happening in all of your groups.

I'm a member of something like 40 groups. The thought of manually adding each feed as a separate calendar in Thunderbird - and the thought of Thunderbird then trying to cope with the result - was... unpleasant.

So I set up a pipe to combine all the feeds into one. Very cursory googling suggested that Yahoo Pipes was the easiest way to accomplish this.

Fairly slick UI. Unfortunately it turns out that meetup.com apparently serves 503s to requests coming from Pipes.

anigbrowl4• 11 years ago

Unfortunately it turns out that meetup.com apparently serves 503s to requests coming from Pipes.

That's really interesting. I think it exemplifies a problem that has been bothering me for a while: for many sites, scripting and design are antipatterns designed to corral rather than enable the user and leverage branding.

deskglass• 11 years ago

I used a pipe that created RSS feeds from Google+ pages in conjunction with some other stuff to get all the articles I'd want to read in one document (with a table of contents) delivered to my kindle. I'm now working on something that doesn't use Y! Pipes and simplifies the process.

We all knew it was coming, but the end of Pipes still sucks.

chrisamiller2• 11 years ago

Oh man, I use 3 or 4 pipes to filter various RSS feeds. Ugh. Good suggestions for replacements (especially those that don't require my own server) would be much appreciated.

maxerickson2• 11 years ago

Are they also shutting down YQL? I think YQL queries can do everything pipes can, you just have to write little scripts to do it instead of the UI.

Still has the Yahoo problem, but it might be the easiest short term.

wsc981• 11 years ago

  Still has the Yahoo problem, but it might be the easiest short term.

The Yahoo problem being that it might be shutdown in the near future?

giancarlostoro• 11 years ago

Maybe they're focusing on YQL instead, if it does offer similar functionality.

spullara• 11 years ago

My understanding is that YQL is being used as the middleware between services internally. Will probably not be shut down, but they could shut off the publicly accessible one.

One thing not mentioned in this thread is that Pipes 2.0 was a rewrite from the original Pipes which was written in Perl. We ported it to the same engine that powers YQL.

As for why shut down Pipes if it shares infrastructure with YQL? Probably because usage is mostly RSS crawlers that no one looks at anymore.

anigbrowl4• 11 years ago

YQL queries can do everything pipes can, you just have to write little scripts to do it instead of the UI

Argh! The UI was the innovative part. The whole point of something like this is that you don't have to learn a bunch of syntax - when you add a module, it tells you what sort of data it needs to know (such as a URL or...) and when you connect modules together it tells you what sort of inputs and outputs are available. This allows you to focus on the actual problem domain instead of the morphology of the query language, which is not important to the user.

I keep encountering this attitude among programmers - 'I type code all day, so why don't you want to do the same thing?' If people wanted to type stuff all the time the command line would still rule, but the majority of people prefer graphic user interfaces because they don't want to write code, they want to make selections from available options and use the computer to take care of the plumbing. This is a Good Thing. It's easier, more fun, and yields greater productivity, for the same reason that handing someone a box of Lego is better than handing them a few blocks of wood and a set of carving tools. The existence of Lego hasn't led to a collapse of sculpture or mechanical engineering. On the contrary, the simplicity, consistency, and interoperability of the different components has made a massive creative, educational, and commercial success.

I don't mean this as a hit on you personally, but I find comments like the above frustrating. Imagine you went into a police station to report the disappearance of your bicycle and the person behind the desk shrugged and said 'you're still able to run around on your legs, what's the problem? Your bicycle may have allowed you to go faster down the street but it's not like you could ride it up and down stairs so you should be happy with walking on your feet.' The problem is the loss of a tool that offered substantial efficiencies on some common tasks even though it wasn't ideal for every situation. You'd probably be equally pissed off if your favorite high level language went away and you were told you could just go back to writing assembler. I personally love doing things in assembler because it makes me feel smart but that doesn't mean it's better than high level languages.

nekitamo• 11 years ago

I get the impression that YQL is suffering from the same kind of neglect Pipes did. It was down multiple times a week for me, and I had to move to other solutions.

Concours6• 11 years ago

Suggestion for your use case: http://www.feedsapi.com , supports keywords and RegEx filtering among other things.

chrisamiller2• 11 years ago

Thanks. Guess I should have specified "preferably free" too :)

psadri5• 11 years ago

pipes creator here: sad to see pipes shutdown. i am surprised it lasted this long -- it was abandoned years ago. on the positive side, it is nice to see it mentioned in the same sentence as y! maps which used to be a big deal back in the day.

ceejayoz• 11 years ago

It was really freaking awesome. Thank you!

webwanderings• 11 years ago

Please suggest alternate. You built a good thing. RSS overall is becoming history.

psadri5• 11 years ago

IFTTT and Zapier are spiritual successors to Pipes, although their focus is on APIs vs feeds.

Maybe start a new, open source version of Pipes?

Concours6• 11 years ago

Check out http://www.feedsapi.com , we do almost everything with rss feeds...

olviko2• 11 years ago

Checkout https://github.com/olviko/RssPercolator.

RssPercolator is a .NET library used for downloading, aggregating, and filtering RSS feeds. Developed out of frustration with Yahoo Pipes.

- RSS and Atom feed formats - Asynchronous multi-feed download - Multiple sources and multiple destinations - Feed filters (string match, wildcards, and Regex)

ilaksh• 11 years ago

This is an amazing product. Have you tried talking to VCs or anyone about someonr adopting it somehow?

I wonder if it would be possible to build something similar using IPV6/some kind of P2P/STUN/TURN/UDP hole punching/swarm/ethereum / any way to make a distributed system that does the same thing. Or maybe just volunteer running Node on their own VPSs.

infinite8s• 11 years ago

Something like Pipes running on sandstorm.io would be nice.

anigbrowl4• 11 years ago

Why did it fail, do you think? I thought it was an outstanding product in terms of both functionality and UI, but it never seemed to gain traction.

idlewords2• 11 years ago

Several reasons I can think of. For one, no one at Yahoo seemed to understand what Pipes was for (part of a wider problem where Yahoo didn't seem to know what the company was for).

For another, Yahoo legal prevented us from doing any kind of long-term storage of RSS feeds. That made the service far less useful than it could have been.

Finally, they didn't give it any resources at all. Management was too preoccupied with the failed Brickhouse project (internal startup incubator) and the host of failed ideas that it spawned.

dankohn1• 11 years ago

Thank you for the great work.

djjose• 11 years ago

Let me just tell you and anyone that worked on the team, thanks so much for it. You saved me hours of work and let me do it really damn well. This is a sad, sad day. I used Pipes extensively and ironically enough at Yahoo for some features/prototypes on News and Search. It was such an elegant and powerful app, I really wish Yahoo had given it more love.

bryanh2• 11 years ago

Thanks so much for you and your team's work on Pipes - I'd be lying if I said it didn't inspire many parts of Zapier!

mindcrime3• 11 years ago

Same here... Pipes has been an inspiration to use with a lot of the stuff we've been doing as well.

smacktoward3• 11 years ago

Pipes has long been a sort of monument to the faith of a bygone age -- the first flush of Web 2.0 optimism, when standard formats and open APIs were going to let us mix and match and mash up services of all kinds at will.

That faith is dead now, of course, scoured from the earth by walled gardens and VC money. But like Catholics in Elizabethan England, some of us quietly tend our secret shrines and pray for its return.

(Hopefully that will work out better for us than it did for them.)

jrochkind1• 11 years ago

Yep, I was trying to figure out how to say what you just said, but you did it better.

paganel• 11 years ago

Thank you for your comment, for a (long) while I thought that I was the only one remembering those times and especially the ideals people were bringing up then. Sometime in 2005 I even copy-pasted this quote on my blog, high in believing that the future will belong to open data and open information:

Bosworth advocated an open model for data.(...) Imagine if you can query any data that is available anywhere in the world. Bosworth said that what this requires is a single, simple, open wire format for items. The format needs to be simple for any P programmer to deliver and any JavaScript programmer to consume. He also pointed out that "complex things tend to break and simple things tend to work." Google has the simplest query language in the world. There is no structure and no syntax. (my note: at that time Bosworth was VP of Product Management at Google)

That is all dead now.

joepie91_2• 11 years ago

Perfectly worded.

jaza2• 11 years ago

Love your analogy, sure made me laugh (visualising a geeky dev like myself tending a weathered gnarly monument in a secluded English glade... in real life, that is, not in an RPG).

However, why such pessimism? I hardly think that standard formats and open APIs are dead. Sure, there are some nasty blights upon cyberspace in this regard (ahem, Facebook, Apple), but other players are still keeping the dream alive to some extent (e.g. Google, Yahoo).

callmevlad• 11 years ago

Hey Pasha, thanks for your pioneering work!

We (Webflow) are actually considering building a Pipes alternative in the near future, and I'd love to sit down with you and brainstorm some ideas. If you'd be willing to make some time to see a very rough demo, I'd love to get in touch - my email is in my profile.

edho3• 11 years ago

Thanks Pasha for letting me be a part of your brilliant idea.

smange• 11 years ago

Thanks for creating something so awesome. I have relied on it heavily for years to clean up dodgy feeds.

Seriously, I think the only Yahoo products I use are Pipes, Groups and Flickr. And Flickr doesn't really count.

arepb• 11 years ago

In an alternate universe not too dissimilar from our own, Pipes is an independent startup and this announcement is Yahoo buying it for 8-9 figures

Touche• 11 years ago

There was IFTTT but looking at their page now it seems like maybe they have pivoted, at least I can't tell that it works like it used to: https://ifttt.com/

enobrev• 11 years ago

If you sign in all the previous functionality is still there. Seems their landing page is only advertising their new "Do" apps (which are a one-button extension of the old IFTTT recipes). Installed it, but never used them much.

driverdan• 11 years ago

IFTTT still does the same things it always has.

TheGRS• 11 years ago

I liked the concept and probably didn't give it a fair shake, but it flooded me with notifications and I just didn't want to take the time to tweak it to my needs. Its one of those apps you need a good tutorial (from someone other than the creators) to walk you through useful functionality and setup.

rhizome• 11 years ago

Wow, this is a jumble of I-don't-know-what:

https://ifttt.com/products/do/button

Their site aesthetic is trying to be Nest-y, but the UX does not communicate simplicity at all.

ris• 11 years ago

Too true.

The only thing Pipes would need is a .io domain and it could be dressed up as the Next Cool Thing which LargeCorp is desperate to get its hands on to prove it's still cool.

danso• 11 years ago

Yahoo Pipes was the backbone for ProPublica's first news web application in 2009: http://www.propublica.org/article/changetracker-howto

ChangeTracker watches the White House’s web site so you don’t have to. Whenever a page on whitehouse.gov changes, we’ll let you know — via e-mail, Twitter, or RSS. But ChangeTracker is not a piece of software. It’s the output of a series of powerful and mostly free Web-based tools, lovingly connected over the Internet. Here’s how to do it yourself so you can track changes on any Web site on the Internets.

matznerd• 11 years ago

Who is working on a pipes replacement and where can I sign-up? Maybe this will be like when google killed Reader and RSS replacements became a hot area...

darkstar999• 11 years ago

Have you tried IFTTT? Not sure what your usecase is, but that might do the job.

durin42• 11 years ago

IFTTT doesn't let me basically reformat an RSS feed and then see it in my RSS reader. Most/all of my use of pipes was "go scrape this thing and turn it into RSS, filtering out stuff I don't want to see."

It was great for that - I'm unaware of anything that is even trying to replace it.

WalterGR2• 11 years ago

    Who is working on a pipes replacement

I'm curious if anyone has suggestions about solving a piece of that:

What would be the best toolset to recreate the Pipes UI?

It's pretty slick, and I could see the underlying UI being useful for modeling a variety of things...

dschep• 11 years ago

Huginn fits a lot of Pipes' usecases: https://github.com/cantino/huginn

jdavid• 11 years ago

I was looking into doing a Pipes 2.0 sort of project. If you live in the bay area i'd love to get coffee and chat about it. There are a number of the pieces to pipes that are opensource, but none of them are strung together.

I also thought that pipes queries were amazing, but slow, so it was hard to use them as part of an application.

voltagex_• 11 years ago

I'm too far away for coffee, but my email address is in my profile.

mindcrime3• 11 years ago

We're working on something that covers a lot of the same ground, although it's not exactly the same thing. It's also not very polished right now, but we definitely hope to get to that level eventually. It's all open source, Apache Licensed, and built using Grails. We don't have YQL, but we extract entities from the content and store it in a triplestore that can be queried using SPARQL, and eventually the SPARQL support will be woven more completely through things.

https://github.com/fogbeam/Neddick

Concours6• 11 years ago

Shameless relevant plug: http://www.feedsapi.com might be an option, it'sit's not exactly a replacementbfor pipes but it does a lot of what pipes did and is a work in progress

joshstrange• 11 years ago

This a disappointment if not an unexpected one. I used pipes again just a few weeks ago to do some simple modification of feed but thankfully ended up taking another approach (not a pipes replacement I just wrote some code for it). Pipes was a really cool tool and it will be missed. Does anyone know of/use any alternatives?

idlewords2• 11 years ago

I worked on Pipes briefly, shortly after it launched. It was really two distinct products rolled into one. The visual interface was groundbreaking and stunning-I think half my team was hired on the strength of that demo. No one could believe Javascript could do that.

The backend was an extremely useful tool for munging RSS feeds. With any kind of support, or even benign neglect, the product would have been successful. It took a lot of active mismanagement and folly to keep Pipes from living up to its promise.

Hats off to psadri and the other Pipes creators for a really stellar piece of work.

psadri5• 11 years ago

Thanks :)

Pipes's interface is primary the work of Jonathan Trevor, with lots of help from Daniel Raffel, Ed Ho and others.

smacktoward3• 11 years ago

It took a lot of active mismanagement and folly to keep Pipes from living up to its promise.

I feel like this unique blend will be Yahoo's lasting contribution to the field of management theory.

bsder• 11 years ago

DEC/Compaq/HP is really hard to top in the area of mismanagement.

georgemonck• 11 years ago

It took a lot of active mismanagement and folly to keep Pipes from living up to its promise.

I would love to read a blog post about how exactly it was mismanaged. It would be great for others to learn lessons from the mistakes.

smacktoward3• 11 years ago

The first rule of Mismanagement Club is you never talk about how you got into Mismanagement Club.

meatysnapper• 11 years ago

Get a job at Twitter and you can see this in action. It's pretty epic.

jakejake• 11 years ago

I wrote a simple Wordpress plugin that works with RSS feeds. In addition to aggregating fees, Pipes has been a go-to solution when people report feeds that are malformed, have some weird characters or anything that chokes the Wordpress RSS parser (simplepie). People would run them through pipes, which seemed to handle anything and output clean RSS.

I'm sad to see it go and it'll be missed by quite a few people who use my plugin.

yenda• 11 years ago

maybe www.busit.com can replace it for that matter

edho3• 11 years ago

It was really great to work with you. During the post-launch chaos, it was life saver that you didn't have your hair on fire like the rest of us :)

Jonathan Trevor's Javascript was out of this world amazing for the time.

aleem• 11 years ago

I'll echo those sentiments, the UI for that time certainly had a wow factor.

With any kind of support, or even benign neglect, the product would have been successful.

Look at IFTTT services now and it's easy to realize that pipes even in its original incarnation could have been so much more given its brain dead UI ease.

It's a shame that Yahoo is shutting this down rather than pivoting it as an IFTTT or similar. I'd wager that it's lack of ownership that makes it hard to pivot. I also don't recall it having a premium tier which has got to weigh down the resources--talent always gravitates toward revenue centers it seems.

jldugger• 11 years ago

talent always gravitates toward revenue centers it seems.

Ie the 'new management can't fire my entire division for being unprofitable' center. Seems like a smart move for the people who wrote Pipes but never found a business model for it.

Volt• 11 years ago

I've used this pipe as my Hacker News RSS feed for the last 5 years, which flips the link structure so the main link for each entry is to the comments.

http://pipes.yahoo.com/pipes/pipe.info?_id=0e19519adf0515251...

Thanks to whoever made it. I guess I'll have to find a way to do it myself now.

bbrian2• 11 years ago

The main one I made was to switch the actual link for the Reddit RSS feed to be the main link, set the true article title as the item title, use the submitted title as the item body and add a link to the link's domain in the bottom.

http://feeds.feedburner.com/RedditArticles

And another where I put the pictures inline (though the 3rd party API I was using isn't handling things too well anymore)

http://feeds.feedburner.com/RedditPicturesWithPictures

I still have 146 subscribers to keep happy, though Feedburner should make any transition transparent. I'm disappointed it's going. Thank you to those who made it!

callumjones• 11 years ago

I can understand why Pipes would be shut down by Yahoo, it was a impressive technical feat but it definitely appealed to very small developer focused crowd.

I feel like IFTTT is our best bet, while it doesn't do the "Unix pipe" that pipes offered it at least provides some usability to the process.

jccalhoun• 11 years ago

Sad but not expected. It was really powerful but, like most people, I stopped using it years ago. It kept breaking either due to the sites I was using it with changing things or the service being neglected (and my limited ability to understand how it worked!)

pjbrunet• 11 years ago

If I was looking for a replacement, maybe I'd look for an RSS aggregation plugin for WordPress? Personally, I have my own PHP code that will slice and dice feeds however I want.

I got about halfway through building a "news blend" app where you tap just the categories you want and it would blend all the latest articles from just your categories into one news feed and email that to you daily. But we never finished it.

brettproctor• 11 years ago

Pipes was the first service I ever owned from an ops perspective (circa 2008). We had two clusters, one on each coast. I thought we had capacity to take one offline to do a full upgrade, turns out we didn't.

By the time I turned the west coast back online, the east coast was failing health checks, and the LB failed everything over to the west coast, which then proceeded to be overloaded, and everything flipped back to the east coast... Ping ponged for about 2 hours until they finally settled.

Fun times.

Another fun story: For the longest time the front page had an example pipe that merged search results from various online sites (amazon/ebay/cl). It was made by a former employee and was easily one of the most popular pipes. One day we found out he had his affiliate id in all of those links. We chuckled and moved on.

psadri5• 11 years ago

Business model :)

Just to be clear, I had not idea about that until now.

product50• 11 years ago

Did your remove the affiliate ID?

scoot• 11 years ago

We chuckled and moved on.

fletchowns• 11 years ago

had his affiliate id in all of those links

Wow that must have added up to quite the haul

Someone1234• 11 years ago

I wonder if they had put in a Yahoo! owned affiliate link, if the site would not still be operating (and self-funding) today...

jonbishop• 11 years ago

Haha, I loved that search results pipe! The product was amazing at the time. It's really too bad that Yahoo didn't try to make something more out of it.

paulhauggis• 11 years ago

This is why I just roll my own data mining tools.

tomcam• 11 years ago

Pipes was awesome. But was there ever a credible strategy to monetize it? Does anyone on HN have any ideas?

bryanh2• 11 years ago

Aw man - that is sad. We at Zapier took great inspiration from Pipes, it was such a cool and powerful product! It was a shame to see it languish.

I'd sure love to pick the brain of anyone formerly involved in the project - it seems like there is a lot we could learn from the trail it blazed. My email in profile, obligatory beers/coffee/etc. offer.

zacharycohn• 11 years ago

You guys are fantastic. I use Zapier for several of my organizations and companies, and I cannot understate how incredible of a service you guys have created. <3 for days.

My only wish would be for a paid tier between 100 and 3000. If I went hog-wild and did everything I want to do, I'd probably use 200-300 a month. I would love to pay for that. But there's some loss-aversion psychology happening that makes me not want to pay for 3000 if I know I'll only use 300.

I'd give you my cc number right now if you did a 1000 zaps at $10/m.

That being said - you know your business better than I do. If it would hurt conversions of $20/m more than it would help, I get that. :)

<3 you guys though.

mikeknoop• 11 years ago

Thanks for this Zach, we keep re-evaluating and running pricing tests, and this plays into that a bit. Reach out anytime if you need help!

stevenjohns2• 11 years ago

I actually had the same query last year when I pushed a place I work at to use a combination of Yahoo! Pipes and Zapier for our Facebook postings. I asked for a smaller plan (we only need about 600-700 zaps/month at the most) and was just instructed by their support to utilize their reward program[0] which offered an extra 50-100 zaps/month based on what you 'complete,' and it included things like (IIRC) liking them on Facebook or following them on Twitter.

It seems like that program has since been limited, however, as it now redirects to a referral page[1] only. A combination of a couple referrals around the office and performing the required tasks gave us about ~650 monthly "zaps" at the time and we haven't needed more than that so far. Still, I would have happily paid $5/month or something to avoid the hassle (and be able to pay them for their wonderful product).

[0] https://zapier.com/rewards/

[1] https://zapier.com/referral/

jotm• 11 years ago

Aw damn, that kinda sucks, but the project has been dead for a while now. I used it for content aggregation, but moved on to custom scripts, which were way more powerful...

Animats• 11 years ago

Most of what Pipes did can be done locally, in a library. FeedReader for Windows did this.[1] Unfortunately, they recently announced "Get ready for the brand new FeedReader! ... Check our site later this March for a new amazing web service, featuring industry-best RSS search and analysis.", which means it's becoming "cloud based" and probably has ads or worse. "rss.com" already went that way; they charge $6/month to read RSS feeds.[2]

Following RSS feeds on a continuing basis takes a lot of RSS polls. Most RSS feeds do not implement RSS in a way that allows getting only new items reliably.[3] The RSS "etag" mechanism is not reliable. Some sites with multiple servers and a load balancer have different etag values on each server. The "guid" field sometimes changes when the content hasn't changed. My experience is that nothing short of full text comparison eliminates duplicates properly. I wrote an RSS reader which does a MD5 of the text of each incoming message to throw out duplicates. Presumably the "pipes" system did something similar.

If RSS feed servers complied with the standard, there'd be less need for feed aggregation services.

[1] http://feedreader.com/ [2] https://www.rss.com/ [3] http://www.詹姆斯.com/blog/2006/08/rss-dup-detection

sanderjd• 11 years ago

I think your first paragraph speaks to a real tension in the world of software: web applications have enormous discoverability, immediacy, upgradeability, and (often) social collaboration advantages over native applications, but they have ongoing costs proportional to their popularity, so it's harder to run them without ulterior financial motives.

ing33k• 11 years ago

in 2010 I attended Yahoo Hack Day in Bangalore/ India. Thats where I learned about Pipes. This was the only tool needed to create a tech mashup.

bitwize• 11 years ago

Web mashup! Hey, remember the late 2000s?

donnellyp2• 11 years ago

Just a shout out to the awesome Pipes community and some people that really helped it along the way. All the original creators (pasha, JT, ed, daniel), community members and team: hapdaniel, dawnfoster, kentbrewster, mirek, spullara, janluehe, dspark, kevink, brettp, lolo, laurencecoates, earth2marsh, psychemedia, davglass, ssaine, sadaf, ido, nagesh, ameya. I know i'm missing tons more, but just top of my head. You are all awesome.

kentbrew2• 11 years ago

Paul, thanks for keeping it running for so long. Pipes was the prime mover behind "suddenly, everything is mashable."

edho3• 11 years ago

Thanks for all of your work. Wouldn't have been the same without you.

donnellyp2• 11 years ago

Thanks guys, it was such an exciting product to work with, create with and help all the community members. I learned the most from this product.

shavenwarthog2• 11 years ago

I used Yahoo Pipes to find my apartment.

One pipe would grab data from Craigslist, strip out irrelevant items, and send me a text if anything new and interesting appeared. Ditto for Westside Rentals.

This combination worked great. If a lame apartment appeared in CL, I'd edit my Pipes regex to strip the same thing out of WR. In this way I'd get only texts for awesome places.

Vaya con dios, Yahoo Pipes!

bbrian2• 11 years ago

Ah, I did this too! I'd forgotten about it. As I was looking, I'd continually update it to block out streets I wasn't interested in, then eventually I was only getting notifications for exactly where I wanted for exactly how much.

I also used it with Freecycle and got a fridge-freezer! (you had to be quick to get them)

atomicfiredoll• 11 years ago

I was also using it exactly like this until recently! I sent my roommate the pipe and we'd each watch/browse it for new places. It saved us a ton of effort.

bluesmoon• 11 years ago

My website is partially powered by pipes. Looks like I'll have to rewrite.

tomsoderlund• 11 years ago

I'm a big fan of Yahoo Pipes, and sad to hear it's being shut down. Pipes and Visual Basic shaped my view of how fun and easy programming can be.

We're trying to build on this legacy with Weld(.io). See a sneak peek of our programming UI at 0:40 here: http://youtu.be/faG3uuOnqxY

If anyone in the Pipes team would like to be involved somehow, ping me on tom@weld.io

ChrisArchitect• 11 years ago

back when pipes came on the scene it was so out there but it was amazing. We couldn't fathom what kind of magic was making it all work and there seemed to be so many possiblities. Future bright etc. Of course, landscape changed etc...but it was / remains a display of what is possible with the web and kind of the 'idea' of an open web/apis/all that jazz.

I am still using it to repurpose an XKCD feed into a fully formatted feed with images that then is picked up by IFTTT and tweeted out as a tweet with images when new comics go up. I used Pipes just because it was point'n'click and seemed to have all the key components I needed to do the task without worrying about the code-route.

orliesaurus• 11 years ago

Thanks for your service! You've done great!

UserRights• 11 years ago

please open source it, thank you!

kentbrew2• 11 years ago

While working for YDN I was the one who spilled the beans about the top-secret Pipes _callback parameter, which turned your JSON reply into JavaScript. The service fell over instantly, of course, but far from being upset with me, the team was thrilled with the exposure. Viva Pipes!

Any word on whether YQL will live or die?

donatj• 11 years ago

As the one person still using Yahoo Pipes, I'm rather saddened by this. Does this also imply an eventual end to YQL I wonder?

pknerd• 11 years ago

Sad to hear that. I guess Pipes was the first and probably the last innovative product ever shipped by Yahoo!

denimboy• 11 years ago

Yahoo pipes was obviously influenced by UNIX pipes and as such is one of the best demonstrations of flow based programming to date. I really loved Yahoo pipes and it influenced the way I think.

Pipes was a basic web "agent". It made basic programming available to the everyman much like HyperCard. Perhaps is just needed a runtime UI that matched the excellence of its design time interface.

Yahoo is focussing on mobile yet here they had this custom agent building tool that could easily be re-purposed to mobile to make a killer platform for Yahoo users.

Hopefully they will opensource the first generation perl version.

Some of my favourite flow based programming links follow.

Surprised nobody has mentioned noflow node.js workflows:

  https://github.com/noflo/noflo

Also this (untested) will convert your yahoo pipes into Node.js

  https://github.com/neyric/pipes2js

and this one (also untested) will do the same in python:

  https://github.com/ggaughan/pipe2py

There is also python pypes which has a yahoo pipes like frontend with a stackless based backend.

  http://pyvideo.org/video/400/pycon-2011--large-scale-data-conditioning--amp--p
  https://github.com/fullscale/pypes

Other python flow based programming tools

  https://wiki.python.org/moin/FlowBasedProgramming
  http://www.kamaelia.org/Home.html (BBC research)
  http://www.ruffus.org.uk/index.html

Wireit is a javascript pipes like frontend

  http://neyric.github.io/wireit/docs/

As you can see I loved pipes and I can

psadri5• 11 years ago

Correct and thanks for those links.

Pipes was influenced by unix pipes (hence the name). The UI was influenced by National Instrument's LabView and Apple's Automator.

I wish Pipes was launched in the age of containers (e.g. Docker). We had the idea of one click deployment to what we today call a container.

empyrical• 11 years ago

I really hope there's a new service that can pick up where Pipes left off, it was a huge shame how it fizzled out at yahoo.

DonHopkins• 11 years ago

Paul Haberli made ConMan on SGI workstations:

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.78.5...

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.78....

ConMan: A Visual Programming Language for Interactive Graphics (1988)

Abstract

Traditionally, interactive applications have been difficult to build, modify and extend. These integrated applications provide bounded functionality, have a single thread of control and a fixed user interface that must anticipate every-thing the user will need. Current workstations allow several processes to share the screen. With proper communication between processes, it is possible to escape previous models for application development and evolution. ConMan is a high-level visual language we use on an IRIS workstation that lets users dynamically build and modify graphics applications. To do this, a system designer disintegrates complex applications into modular components. By interactively connecting simple components, the user constructs a complete graphics application that matches the needs of a task. A connection manager controls the flow of data between individual components. As a result, we replace the usual user-machine dialog with a dynamic live performance that is orchestrated by the user.

interactiv• 11 years ago

I'm really motived to create an opensource alternative as I was a big pipes user. I set up a repo (obviously empty for now) ,I'll try to come up with a minimal working app before yahoo pipes gets frozen:

github.com/interactiv/pipes

didn't decide yet what serverside tech i'll be using but quick deployement is a priority for me. If anyone's interested.

EDIT: a lot of useful information here about existing projects, thanks.

juliendorra• 11 years ago

I used Pipes as the basis of data mashup classes, with non-developers students that were mostly afraid/bored of anything related to programming. Using Pipes, in just a few hours they would make small, custom data apps. A typical example would be a student building a pipe gathering the weather from one source and trying to fetch the relevant clothes from a retailer with pictures and links. I had a student using the location entity extraction to geocode his favorite street art blog that cited cities in the text but didn't use rssgeo, and then output the augmented feed to a map view.

It was an eye opener for some: they could get, transform and repurpose data, and it could be fun.

Pipes wasn't without its flaws: V1 was buggy and gave us a lot of trouble when 30 students were trying to use it from the same IP, the interface was plagued by the usual issues of visual interfaces (clutter…). Some things were odd, some a little too hard for what they achieved and other were magically easy!

But at a moment when every tech company is saying that "learning to code" is important, it's sad to see a tool that had real educational value disappear. It was a really effective tool for non-developers to learn about data markup standards, to think in term of data flows, to get introduced to the idea of a data or web API. For this particular use in education I'm not sure there is a replacement.

spdustin• 11 years ago

As some deeper comment posted, Huginn [0] is awesome, an open-source connector-of-things like Zapier or IFTTT, but on your own server.

[0] https://github.com/cantino/huginn

narrator• 11 years ago

In the enterprise space this concept is thriving and doing well. For example, SnapLogic has been building enterprise integration technology that can easily tie hundreds of different data sources together using pre-built or custom components using a GUI or programmatically. In order to become a critical part of business infrastructure it needs to be a professionally supported and maintained product. Yahoo Pipes was just an experiment and not a serious business so people never built anything serious on top of it that would pay for the underlying maintenance and infrastructure.

nreece• 11 years ago

Disappointing news. But, if you're just looking at creating custom feeds for webpages, then try out Feedity - https://feedity.com

wiresurfer• 11 years ago

I have been a fan of Pipes for a while. Kudos to @psadri and others who made it possible with the state of web tech back in the days. It was a very well made tool both frontend and backend wise!

I have a emotional connect with Pipes as a tool. Right out of college, and before the whole API thing was pervasive, I had leveraged pipes for so many small projects and hacks. Infact I used it in a Yahoo Hackday hack and it landed me a gig at Y!

More relevant to the present, It has been something I have mentioned to so many people who have come to me discussing UIs for automation/control systems/or to manage workflows, processes. In a way it was what noflow's UI looks like, but years in advance. In a different parallel universe, it may be the way people use APIs, or right big Haddop/Spark/Storm jobs/topologies, and the front end was open-sourced back in the day, with full integration with hadoop for job management.

juxtio• 11 years ago

Sorry to see Pipes being sunset. It had a major design and UX influence on a number of us during the early Web 2.0 days. Kudos to the Pipes folks for a seminal creation.

On a related note, a couple of us have been hacking on a Pipes like user experience around Data Integration and Analytics with Web feeds and APIs -- we had been planning a separate announcement but thought this might be a useful place to post if there are others who are looking for alternatives. Please message me directly - my contact info is in my profile.

llamataboot• 11 years ago

Yahoo Pipes was probably one of the best examples of letting non-programmers be developers. I created a lot of useful pipes back in the day for friends and family. Used a particular craigslist one for years.

xefer• 11 years ago

I always knew this day would eventually come. It's kind of sad.

I had created this years ago to display Twitter data using Pipes, YQL and Google Charts. I haven't had to touch it in years but it's always just quietly worked, and some people seem to like it:

http://xefer.com/twitter

; e.g.;

http://xefer.com/twitter/xefer

I liked that Pipes allowed services to be chained together driven from the browser.

flinty• 11 years ago

There was an awesome pipe which allowed you to get a rss feed of commentators you want to subscribe to. Sadly no more

donretag• 11 years ago

I use Yahoo Pipes to get around Netflix's walled garden.

Netflix does not believe in RSS, even though their Influencer content is not behind a subwall. I use Pipes to parse Netflix's json content and then turn it into a feed. I even created a Pipe that took any influencer ID and automatically created an RSS feed.

The idea was started because Daniel Tunkelang of Netflix was very anti RSS. I showed him, but alas, no more.

I never figured out how to use IFTTT. How can you grab any content from the web, like Netflix's json content? There is no HTTP input as far as I can tell.

stevenjohns2• 11 years ago

I used a combination of Yahoo! Pipes and IFTTT personally. I'm not sure how people are using IFTTT on its own.

For me, I would do the data parsing on Yahoo! Pipes and prepare everything to be triggered, and then the actions would be performed on a service like IFTTT (where the instruction was to just post whatever new item came in the feed).

dtunkelang• 11 years ago

"The idea was started because Daniel Tunkelang of Netflix was very anti RSS."

Um, what? I've never worked at Netflix, and I've never been anti-RSS. And I'm pretty sure there isn't another Daniel Tunkelang. :-)

waldr• 11 years ago

Pipes was certainly an inspiration behind what we've been working on at tray.io - sad to see this come to a close.

Hats off to all those that worked on the product - great to see it remembered so fondly.

bootload2• 11 years ago

Using pipes was interesting and certainly did what it reported. I always capture what

Showing operators: https://www.flickr.com/photos/bootload/385194782/

bootload2• 11 years ago

I remember well using and playing with Pipes when it came out early 2007. The service worked. I mostly played around with my twitter and flicker feeds. Here's some screenshots of what it looked like.

cloning twit user: https://www.flickr.com/photos/bootload/385441178/

multiple feeds into one: https://www.flickr.com/photos/bootload/385433038/

extracting flickr data: https://www.flickr.com/photos/bootload/385101700/ and flickr backend https://www.flickr.com/photos/bootload/2789198106/

extracting twitter data: https://www.flickr.com/photos/bootload/385194782/

clogged pipes: https://www.flickr.com/photos/bootload/384133421/ and https://www.flickr.com/photos/bootload/384128224/

Read the comments on this link to get an idea of how I used Pipes and the logic behind it: https://www.flickr.com/photos/bootload/385101700

gsam• 11 years ago

First Google Reader and now Pipes? What's the problem with RSS and please tell me what's better?

Aissen• 11 years ago

I loved yahoo pipes. I used it to filter high-traffic RSS to do a trivial word-match on content. I am now in the process of moving this to Newsblur, which allows filtering RSS content on tags, authors, or any word in the title. It's actually quite good, although we can't use the same filters for multiple feeds; at least it does the job.

chinpokomon• 11 years ago

So sad to see Pipes go! I absolutely love Pipes and was just raving about it to a coworker this week. Terrific UI and a fantastic idea. When development fizzled out, I tried to find another service as a replacement, but everything I've tried has fallen short. I'm really disappointed as Pipes is definitely one of the best ideas that never caught on line it should. Popfly, while short lived, had the advantage of being able to modify the code behind, but it didn't come close to the fantastic ease of use and tremendous potential for reuse.

FractalNerve• 11 years ago

One of the truly powerful services, I will really miss. Mrs. Meyer squeezes out things at Y! like a lemon.

EDIT: added OSS & Commercial alternatives

free or commercial:

    [1] http://createfeed.fivefilters.org/
    [2] http://www.feedsapi.com/ 
    [3] https://theenginuity.com/search/
    [4] https://zapier.com/
    [5] http://www.elastic.io/
    [6] https://ifttt.com/

open source:

    [1] https://github.com/olviko/RssPercolator
    [2] https://github.com/fogbeam/Neddick/
    [3] https://github.com/cantino/huginn
    [4] https://github.com/fullscale/pypes
    [5] https://github.com/jparkrr/ISyTT
    [6] https://github.com/atask/shifttt
    [7] https://github.com/KLVN/F7_T7_RSSFeed

s0me0ne• 11 years ago

Pipes gui was revolutionary when it came out and to me is invaluable. Shame to see it and rss die out from mass appeal :(

Yahoo Pipes End-Of-life Announcement

Related Stories