Huginn: Like Yahoo Pipes plus IFTTT on your server

https://github.com/cantino/huginn/blob/master/README.md

by ColinWright • 12 years ago

524 92 12 years ago

danso• 12 years ago

Also relevant: How the New York Times interactive team uses Huginn

https://source.opennews.org/en-US/articles/open-source-bot-f...

Most prominently, we used it during our Olympics coverage to monitor the results of the API we built and let us know if the data ingestion pipeline ever grew stale. To do that, we set up a pipeline

FroshKiller• 12 years ago

One of the developers posted about this recently: https://news.ycombinator.com/item?id=7582316

jschulenklopper• 12 years ago

... and in March 2013: https://news.ycombinator.com/item?id=5377651

BTW, a great idea and an impressive side project!

weavie• 12 years ago

This sounds like an excellent project to make use of my raspberry pi.

fasteddie310033• 12 years ago

I am working on a similar project called Taskflow.io that is aimed at more backend business oriented tasks. It can do similar things through an interface flowchart editors where you make the actual flowchart that gets executed. I would still consider it a public beta. I would love your feedback.

johnobrien10103• 12 years ago

Crowded space... We're trying to go the same thing for business intelligence applications with a product we're launching called flowreports.co...

fasteddie310033• 12 years ago

There are other companies doing workflow automation, but their products seem clunky and not aimed at web-services, which companies are increasingly relying on. I want to be the IFTTT of back room office tasks. I want anyone in an organization to be able to create a workflow to automate some mundane process they have at their job. I would love any feedback you have of Taskflow.io so I can get my product to that level. Here is a link for any feedback you might have http://eddie.taskflow.io/start_process/201?return_url=http%3...

zorbo2• 12 years ago

Will this be provided As-A-Service, or will it be a downloadable product that can be deployed in-house? This is exactly what I have been looking for for a while, but there's absolutely zero chance we're going to send any of our business information to a remote service.

fasteddie310033• 12 years ago

It's as a service now. I've had other people express the same kind of concern with data security. We can talk about a self hosted version.

rch• 12 years ago

I've wondered about this quite a bit, since I run computationally intensive analysis on sensitive data, and some of the same thinking would apply in this context.

In brief, I could provide an appliance on something as trivial as a Raspi that updates itself over VPN, and would let you run the services on your own systems. Would that work for you if one of these providers did the same?

Obviously we could do better with a custom system deployed onsite, but the idea is to simplify the process and potentially eliminate cost of getting started; similar to Square sending out card readers.

bkyan• 12 years ago

Hey, I tried it, but I got stuck trying to set up a webhook, as there's nothing in the dropdown. Here is a screenshot:

http://shadowcatcher.mindcast.com/clips/1397535410d5p7t6ut3s...

malanj2• 12 years ago

This looks really awesome for managing an office. We're currently automating things using Google scripts and other custom glue to do things like order food, get feedback on lunch and mail people weekly digests activities. Sounds like this could be a great solution for this.

anentropic• 12 years ago

weekly digestion activites...?

malanj2• 12 years ago

Weekly digests :-) We're about 35 people split in 5 teams. On a weekly basis each team gets a "weekly update" mail containing a google document that gets created off a template doc. The weekly update contains some questions that basically ask the team what they did that week. It's a shared google doc so the team can collaborate to fill it in. Those filled in docs get aggregated into a single PDF and gets sent to everyone on a Monday morning. So everyone stays in the loop with the other teams' progress.

toomuchtodo3• 12 years ago

Writeup please if you do this.

SloughFeg• 12 years ago

Is there an online sandbox anywhere to check it out? A project like this simply calls out for their to be a live demo.

sundip• 12 years ago

http://runnable.com/Ut9idpiQaiM-AACi/huginn-example-setup-fo...

kzahel• 12 years ago

Does this have a companion android/iOS app to upload location data? I really like the idea of self hosting something like this.

N0RMAN• 12 years ago

iOS App (requires RubyMotion) https://github.com/cantino/post_location

platz4• 12 years ago

You could probably get all your data from google as well since it's stored on their servers, without having to upload from your phone

toomuchtodo3• 12 years ago

Does Google provide programatic access to their location history system without scraping?

gibbsnich• 12 years ago

For Android there is https://github.com/jcs/triptracker -- which I did not try yet..

crawfordcomeaux3• 12 years ago

I extended a Find My iDevice API lib about a year ago with the intention of creating a Huginn agent for exactly this, but life got in the way. I also wanted to add a geofencing agent, too.

Unsure if there's a better lib out there, but here's what I worked on...warning: needs lots of love, eg. tests: https://github.com/rickyc/find-my-ios-device

rcyeager• 12 years ago

Another Pipes+IFTTT tool: https://wewiredweb.com

finnn• 12 years ago

Like many of the others that have been posted here, it's not self hosted

psaintla7• 12 years ago

Am I missing something or is this just another rules engine?

perlgeek• 12 years ago

Did you dismiss twitter with the same line?

Twitter is also just another rules engine, with pretty simple rules about which tweets you receive. And yet it's also so much more. It's a platform, and it's a social network. And it's something that many people love to use.

Who cares if it's just another rules engine under the hood?

pestaa• 12 years ago

Not only that, but the interface is always a huge part of any product that's simple under the hood, and I'm going to check it out and evaluate right now...

psaintla7• 12 years ago

Twitter isn't really a rules engine as it doesn't have the ability to create workflows or manage rule priority afaik but let's ignore that for a second.

Rules engines are typically a terrible idea and I say this as someone who has worked at two large corporations, one a bank, where rules engines were heavily used so they could avoid having the larger development staff they really needed. Rules engines fail miserably every single time and eventually have to be replaced.

The problem is that as time goes on people who don't know any better end up writing larger and more complex rules and workflows without an understanding of the side effects those rules generate. The end result inevitably becomes a huge mess that is extremely fragile and nearly impossible to follow.

notastartup2• 12 years ago

what would be great is if each agent was somehow able to obtain it's own ip address.

kej2• 12 years ago

With IPv6 there's no reason you couldn't do this, but what is the use case? I'm not seeing what you could do with individually addressable agents that you couldn't do otherwise.

notastartup2• 12 years ago

Can you elaborate how one can do this with IPv6? What hosts have this? How many IP addresses can you get?

Basically for web scraping. If you had multiple threads and each of them had separate IP addresses, you'd have a better chance than doing it with one IP address.

albertsun2• 12 years ago

The best part of Huginn is being able to self-host and write any arbitrary agents you want.

platz4• 12 years ago

I wish the agents could language-agnostic though.

cantankerous• 12 years ago

Same. I wonder how hard it would be to bake in some kind of local webhook system for that purpose. Haven't looked too closely at this, though.

hnriot• 12 years ago

there are a myriad ways or linking software languages.

albertsun2• 12 years ago

There's already support for agents written in javascript: https://github.com/cantino/huginn/blob/master/app/models/age...

And for more substantial tasks it'd be possible to write an agent in another language and then call it from ruby in some way.

zrail• 12 years ago

I submitted an issue just now to add support for Heroku-buildpack-style agents (just a directory of executables)

https://github.com/cantino/huginn/issues/215

dfc• 12 years ago

I don't know, the documentation is excellent. That is certainly one of the best READMEs I have ever seen. The organization of the setup sction of the README is superb. The author of the README clearly thought about the needs of new users as well as seasoned veterans. More projects need to adopt this general format:

  # Getting Started

  ## Quick Start

  If you are unsure of our project and just want to play around, you can
  get things set up quickly by:

  1.  Clone this repository and…
  2.  Do something
  3.  Do the other thing

  If you need more detailed instructions, have no fear. We are not going
  to look down on you if you are not an expert. We took the time to write
  a setup guide for newcomers: [Novice setup guide][novice-setup-guide].
  Everybody has to start somewhere.

  ## Real Start

  Follow these instructions if you wish to deploy your own version or
  contribute back to the project. There is nothing we hate more than
  README’s that ignore all of the practical concerns related to setting up
  a long term installation. Follow these steps and it will be easy for you
  to keep up with updates to the project and still retain the all tweaks
  you made to suit your idiosyncrasies.

  ## Odds and Ends

  ### Optional features

  Not everybody needs a XYZ plugin or wants to share their every action
  with PQR. You can enable these features by…

  ### Rare Corner cases

  In certain rare circumstances you made need to prevent X or implement Y.

  Prevent X by…

  If you need to implement Y…

yukichan• 12 years ago

Zapier is also good with lots of integrations, but it's a little pricey. Yet if you calculate what your time is worth and include the amount spent on making this work plus customizations, it's probably less. Depends on if Zapier can do what you want.

hyp02• 12 years ago

I always liked the Yahoo Pipes concept... but it didn' seem take off... and I personally found it too limited for everything I tried to do with it. Perhaps it's just another case of the old "visual programming language" is harder than it looks.

I hope Huginn does better. I like their copywriting "You always know who has your data. You do."

shavenwarthog24• 12 years ago

Agreed. I did a multipart Yahoo Pipes project to find my current apartment. It grabbed info from two sites, tossed out the uninteresting ones, filtered it a bit, then texted me if a new apartment in my price/location range appeared.

Very useful, if a little awkward. The Huginn project sounds like a great alternative!

scuba7183• 12 years ago

Just curious, how many hours of work did that take? Do you think it was worth it?

platz4• 12 years ago

Excluding the UI, I wonder if storm is a more robust, if more complex, option to do the same types of things: http://storm.incubator.apache.org/

egonschiele• 12 years ago

You are right that Storm is certainly more robust for large amounts of data. But Storm just provides underlying infrastructure. Huginn builds on top of something like that to add different agents for twitter, weather etc. So afaict, Huginn is an app built on top of something like Storm.

feralmoan• 12 years ago

Storm doesn't naturally support dynamic topologies and is rather resource hungry, which needs a bit advanced planning. I was looking @ Storm for my own pipelining product (bip.io) very early on and shied away as too high an opportunity cost for self-hosting users/devs to be bothered with. On a Rasberry Pi for example, forget about it. Without being able to create dynamic graphs it otherwise just ends up being a simple message bus (anti-pattern).

tectonic7• 12 years ago

bip.io looks very cool. Do you think our efforts should be combined?

nl• 12 years ago

Storm is a framework for coordinating computation. It's not really designed to "perform automated tasks for you online" - although of course you could make it do that.

If you want to run a machine learning algorithm on 100 machines then Storm is what you want. Want a service to check the weather for your location? Huginn looks good.

kirk21• 12 years ago

Where can you get an invite code? http://snag.gy/xh6uk.jpg

tectonic7• 12 years ago

The default invite code is 'try-huginn'

c0nsumer2• 12 years ago

This is a really frustrating name. Hugin is already used for panoramic photo stitching software: http://hugin.sourceforge.net/

This just has another N bolted on to the end and does something completely different.

jmduke• 12 years ago

I'm not sure about the etymology of Hugin, but Huginn is more than likely a reference to Norse mythology (for Anchorman fans, you'd recognize it as "Great Odin's raven!"):

http://en.wikipedia.org/wiki/Huginn_and_Muninn

14113• 12 years ago

It specifically states it in the readme that Huginn is a reference to the raven.

thomasfl• 12 years ago

Will this run on a standard heroku stack? The wiki says it will run on OpenShift and CloudFoundry. https://github.com/cantino/huginn/wiki

tectonic7• 12 years ago

It will run, but the default Procfile spins up 4 processes, so Heroku might be expensive. If someone wants to figure out how to get everything to run easily in one process, that would make free Heroku hosting possible. I run it on a small VPS.

alxndr4• 12 years ago

I would also love to have this running on a free Heroku process.

jayxie• 12 years ago

Exciting stuff, it would be amazing to build an AI layer on top of this that mines your browsing habits (depending on your paranoia settings) and automatically generates agents based on your interests.

zwentz• 12 years ago

This would be very cool for automating parts of AWS. Inclement weather coming? Or an earthquake? Start spooling up servers in another region.

okhan• 12 years ago

I was just building exactly this, only worse. Looks really great.

alxndr4• 12 years ago

Anyone know why this project encourages using a private fork to do contributing development?

"Make a public fork of Huginn. [...] Make a private, empty GitHub repository called huginn-private. Duplicate your public fork into your new private repository[. ...] Checkout your new private repository. Add your Huginn public fork as a remote to your new private repository[. ...] When you want to contribute patches, do a remote push from your private repository to your public fork of the relevant commits, then make a pull request to this repository."

tectonic7• 12 years ago

Just to let you keep any private changes private. Perhaps it's not the best recommendation.

alxndr4• 12 years ago

Ah. It seems unnecessarily complicated for people trying to get started. Perhaps preface it with a note saying something like "if you'd like to keep your commits private, follow this brief guide" so it doesn't seem required?

fujipadam• 12 years ago

This is awesome but is there a tool like this in php? I am looking for a easy visual scraper

Huginn: Like Yahoo Pipes plus IFTTT on your server

Related Stories