JamieLewis

http://i.imgur.com/PnmDT5z.png

I have had some time off this week so decided to work on something a bit different. I have been working on a concept fuzzing framework for security testing. In the screenshot you can see some of the files produced by it - The bottom right is the configuration used to generate the file format (for this case Bitmap, although I have tested a few others like WAV)

Bottom left is a bitmap produced with no defects. The top shot is a bitmaps produced with some random changes - you can see the green bitmap is now corrupted due to a change somewhere in the format.

I've played with a variety of frameworks (Storm and IBM Infosphere Streams are probably the two I have worked with the most) - In my previous job I was in a team which designed and developed our own (and later integrated partly with IBM Streams) I took down many lessons from the initial creation, the later integration and the work following that. ghostream was born out of those lessons.

Mostly I found the follow problems with existing frameworks :

- Tied to an execution environment - Storm with ZooKeeper and Streams with their own custom one. This makes it very difficult to use either for small projects - and limits integration choices when scaling.

- Resource hogs - The IBM Streams environment is a huge resource hog, I never really got that far with Storm but the number of dependencies it required just to get something up and running provided a similar sense of dread.

- Easy to debug/optimization - Streams has the best tools for this at the moment but they are all heavily tied to the execution environment. I'm hoping with ghostream that structures can be built at the protocol level to provide a way to build tools on top - not tied to any particular environment.

Awesome idea.

This week as been all about getting a second version stamped and out (nowhere near a finished version or even 1.0, but a 0.2 version)

I've been working on https://ghostream.com a framework for constructing stream processing systems. Getting a first draft of the website (http://imgur.com/ujsmjJz) and the documentation (although there is still much to do)

And after that, lots of bug fixes and environment clean ups (code coverage pipeline / static analysis etc.) Managed to knock down quite a few little bugs that way.

I posted this on the last thread, but I am definitely still looking for volunteers.

I'm actually currently starting to look for a co-contributor or two, there is so much that I want to get done, and having an extra set of eyes and hands would be great.

I am working on a framework/concept called ghostream (https://ghostream.com / https://github.com/ghostream/ghostream)

ghostream is a stream processing framework, built in C++ - it is a data stream processing framework. Having used numerous commercial ones (most notably IBM's Streams), and building various other part-frameworks (i.e. for time and budget reasons focussed on one particular thing) I has an itch to build an open source one (while storm and samza are definitely related, I feel they fall into a slightly different category - I'm all about the streams :) )

It is still in very active development - but the base is solidified now and the new test framework is starting to shape up nicely. There are a whole list of issues on github + a roadmap waiting to be exploited.

I have already started dog-fooding it in my other personal projects - the most visual of which can be seen here: http://jamielewis.me.uk/posts/2013-11-03-Mapping-Earthquakes....

If this strikes your interest give me a shout: jamie@ghostream.com.

There is a great episode of Black Mirror (All episodes of Black Mirror are great, but this one is more relevant - Season 1 Episode 3) titled The Entire History of Us, it turns the figurative mirror on a world where people do exactly this...

I'm actually currently starting to look for a co-contributor or two, there is so much that I want to get done, and having an extra set of eyes and hands would be great.

I am working on a framework/concept called ghostream (https://ghostream.com / https://github.com/ghostream/ghostream)

ghostream is a stream processing framework, built in C++ - it is a data stream processing framework. Having used numerous commercial ones (most notably IBM's Streams), and building various other part-frameworks (i.e. for time and budget reasons focussed on one particular thing) I has an itch to build an open source one (while storm and samza are definitely related, I feel they fall into a slightly different category - I'm all about the streams :) )

It is still in very active development - but the base is solidified now and the new test framework is starting to shape up nicely. There are a whole list of issues on github + a roadmap waiting to be exploited.

I have already started dog-fooding it in my other personal projects - the most visual of which can be seen here: http://jamielewis.me.uk/posts/2013-11-03-Mapping-Earthquakes...

If this strikes your interest give me a shout: jamie@ghostream.com.

I was bored so I found the public key and your blog and sent you an email :) So that Zero can be a One now.

By the way, the public key on your blog is not very copy and paste friendly - I had to redo the armoring.

But yeah, it is no where near as easy at it should be.

Here is the thing, I don't mind ads - but I do mind pages loading 10+ files of javascript when I am attempting to read an article.

I browse with noscript enabled, and for the most part the web is a much faster and a much, much safer place (https://www.youtube.com/watch?v=Haum9UpIQzU).

I am still amazed when I try to read a simple blog article and am greeted with a blank white page, it turns me right off the site and I go spend my time elsewhere.

I will happily shell out money for content that I like.

I have tried neither of those two...I have heard good things both though, maybe I will make the time to give the a try.

A Javascript IDE would be great, I remember working on some Javascript a couple of years ago and being frustrated at even getting basic autocomplete to work in any tool (Vim was the closest but lacked any context). And +1 Open Source. I look forward to giving it a try.

The IDE is a big pain point of mine.

On my side projects I tend to use Vim for C++ which works fairly well after heavy customization with plugins.

At work I have tried to adopt Vim with Java but Java does not lend itself to simple developments (far too may 3rd party library imports). Eclipse is bulky and slow, doesn't customize well and constantly crashes.

I have heard emacs lends itself better to Java than Vim due to it's extensibility, but devoting the time to learning another application is not really something I want to do.

Vim is too basic for a complete IDE (and it should remain that way it is a text editor after all), Eclipse is too bulky, I want something that is just right.

I'm currently building ghostream (https://github.com/ghostream/ghostream) - it is a data stream processing framework. Having used numerous commercial ones (most notably IBM's Streams), and building various other part-frameworks (i.e. for time and budget reasons focussed on one particular thing) I has an itch to build an open source one (while storm and samza are definitely related, I feel they fall into a slightly different category - I'm all about the streams :) )

It is still in very active development - but the base is solidifying and the new test framework is starting to shape up nicely.

I have already started dog-fooding it in my other personal projects - the most visual of which can be seen here: http://jamielewis.me.uk/posts/2013-11-03-Mapping-Earthquakes...

Twitters streaming API used to work fairly well when I was playing with it. https://dev.twitter.com/docs/streaming-apis that was a few years ago though.

Probably not what you are looking for, but I would love to see more real-time data curation - I would certainly be willing to hand over cash for some decently priced real time data feeds - stock tickers, sports events, raw traffic data, headlines, currency, flights - many of these are available in pull formats, but it would be great to get a push feed.

I read GEB during my final year of university, during winter break. It took about 3 weeks of solid reading (wake up, read, pause and think and clear my mind, do other stuff, go back to reading, sleep, repeat) to get through it. By that point in my life I had already been exposed to many of the topics discussed, and even unwittingly read works that cited GEB, so had been also been exposed to some of the ideas presented as well - I don't however think that matters too much.

It really is, in my opinion, an amazing and clever book. If you can, I would recommend taking a few weeks to really digest it, it is not a book you can read for hours and hours on end - you will need to stop, clear your mind and reflect on some of the points made - at least that is what I had to do.

Be warned, the book will start to mess with you...but it warns you when it does...most of the time.

I really wish people wouldn't give this as much weight has it gets, there are so many problems with the methodology:

1. The raw data (answers, in particular the free response text) has never to my knowledge been published. 2. The algorithm used to select schools and pupils taking the test has never to my knowledge been published. (In fact it differs from country to country...or city..) 3. The test itself is taken in many different languages, which language version of the test is used in which country, has to my knowledge never been published. 4. They compare Shanghai, a relatively well off city with entire countries..without knowing the data in 1-3 we cannot know if this comparison is fair or makes any sense at all. 5. They administer the test at different times of the year relative to exam time...some teenagers would have just got back from summer break, others would have been deep revision, preparing for other exams - I would guess this has a statistically significant impact on performance.

Until they publish all the data so it can be analysed independently (and god forbid replicated), this isn't science or statistics...it's busy work.

Honestly, part of it is motivation. I keep getting distracted with other things.

I expected it to go the 80-20 way, the initial ideas are not new ones, I have been building similar frameworks for a few years.

The time is mostly spent on the more difficult problems like "How should operators hang up when the program is closed?" " Should the program be expected to close?" "Should there be a global context object, or a default one" "Should an operator have a window by default or not"

There is no right answer, so its just a case of sucking it up and implementing something, seeing if it works, and going from there.

Plus the side escapades with earthquakes and space weather and irc channels - although implementing these little things does drive the bigger project.

I am hoping before the end of the year I can hit something I feel comfortable slapping "1.0" on and move onto something else for a while and use it as a base for working on some other ideas I have.

An open source stream processing framework (https://ghostream.com) - it isn't really ready for the limelight yet, but you can find the code base on github (https://github.com/ghostream/ghostream) - final steps on the to do list are a few more example projects, a couple more functional operators and to polish up the documentation.

It has hit the 80-20 scenario. The last 20% is taking 80% of the time :)But it has let me quickly prototype a few little ideas (http://jamielewis.me.uk/posts/2013-11-03-Mapping-Earthquakes...)

Check out this veritasium video https://www.youtube.com/watch?v=ZMByI4s-D-Y - It goes into a little detail on how the kilogram got it's name, it is also very interesting in it's own right.

I pulled the example from the air to make a point, but you are correct, the last 20% is 80% of the work...

See the comment by aaron695 who made a very good point about feedback.

I wonder though if there is some easy fruit at the bottom of the tree i.e. elementary school - where I gather grading is probably a lower burden and easier to formalise - although I am probably again greatly understating the problem.

A very good point that I did not consider, I remember getting very good feedback from my teachers - It would appear I underestimated the size of the problem.

The complaints I hear centre around grading, lesson planning and of course bureaucracy.

Now, I believe that the education system of the majority of the world is fundamentally broken - and I am cheering on the likes of coursera, khan academy and the like to get to fixing it.

But in the mean time...can we take the first of these...grading. At the risk of trying to find a technical solution to everything this sounds like it should have a nice technological solution to it...

I can see some of the issues in English and the like where answers may not be 0 or 1 but if google can instantly translate English to Swahili...someone should be able to find a way to grade English assignments of high school student efficiently.

Or is this a money issue? The technology exists or could exist but there is no funding, so instead we are wasting hundreds/thousands of teacher hours a week while they repeat the same manual chore over and over again?

I would be interested in hearing some teachers comment on this.

Reminds me of this: http://www.ted.com/talks/geoffrey_canada_our_failing_schools...

As mentioned by vitovito your problem is most likely effective communication.

Take this post for example, there is very little structure, the main points are clouded by seques that don't lead anywhere, here is how I might have written it instead:

"Hi everyone,

I have been encountering a problem with several of my clients, both past and current. It seems they become intimidated whenever I start discussing anything vaguely technical e.g. Version control ala GitHub.

Have any of you come across a similar problem in your line of work?"

It might be worth taking a class or at least picking up a book on communication, presentation and debating which should cover many of the problems.

On a separate issue you posted an excerpt from an email. It seems t be that the client would not be requesting information on version control for a website (although that may be the case) - in my experience this is probably a query regarding content management in which a CMS would seemingly be a better fitting solution - this should be taken with caution as I do not know anything about this client and so could be completely off course.

Not a mailing list as such but there is a nice GameDev subreddit which is fairly active: http://www.reddit.com/r/gamedev

There is also the GameDev stack exchange: http://gamedev.stackexchange.com/

This is quite a common teleological argument known as the Watchmake Analogy - https://en.wikipedia.org/wiki/Watchmaker_analogy

Instead of going over the common ground, I wish to present to you some examples of Computer Science (and other areas) finding the exact opposite (that is complex systems often have no defined controller or designer) is true.

Time after time, Computer Scientists, Physicists, Mathematicians and Biologists modeling behaviors of fractals, bacteria, flocking birds, ants, fluid, fish and even people have learned that systems with very simplistic rules can produce insanely complex behavior.

NASA famously developed a program that would design antennas with great success: http://www.nasa.gov/centers/ames/news/releases/2004/04_55AR.... , these antennas worked extremely well, but were created by a computer following a few "simple" rules, but the results were outstanding. There is an entire branch of Computer Science dedicated to researching Evolutionary Algorithms(https://en.wikipedia.org/wiki/Evolutionary_algorithm ), these algorithms produce solutions.

Now if I was to say to you that I was going to start a company with a few workers, each worker could do what they wanted at any time and in 5 years I expected to have to giant skyscraper (which would be constantly cleaned - remember I am not telling anyone to clean!), thousands or millions of workers and a constant supply chain. You would probably call me crazy (mostly for not having any plan at all!), yet almost this exact situation happens all across the world, every single day and has been doing so for millions of years.

Ants. Ants as you may know live in colonies, and generally produce huge underground nests. They are often able to find the shortest path to food and building supplies, the nests are kept free of obstacles and defended to the death. Yet, there is no central controller (the "queen" is a misnomer, all she does is give birth) that indicates where the tunnels go, noone is telling an ant what to do at any other time. The system evolves naturally out of very simple genetic rules and is based on communication between peer ants. Deborah Gordon gave a brilliant TED talk on the subject of how clever ants (or at least collections of ants) can be : http://www.ted.com/talks/deborah_gordon_digs_ants.html

(Note: I am not advocating trying to form a company as described above!)

A few books for those who want to research further into this fascinating area:

http://www.amazon.com/Computational-Beauty-Nature-Exploratio... - The Computational Beauty of Nature - Computer Explorations of Fractals, Chaos, Complex Systems, and Adaptation

http://www.amazon.com/Turtles-Termites-Traffic-Jams-Explorat... - Turtles, Termites, and Traffic Jams: Explorations in Massively Parallel Microworlds - The book that started me in this field

And some Google (other search engines are available) search terms:

- Fractals - Ants / Termites/ Turtles Behaviour - Flocking Algorithms - Cellular Automata

Right now, I am currently designing/writing a game for Android, mainly as an exercise to get me back into game programming.

As a note You must be logged in to vote. And yes, an about page is a good idea, I will get right on that. Thanks for the feedback...

This was great fun to do, and I am planning to keep http://qitika.com/news going, 24 hours is just enough time to build a foundation and get a concept together...the hardest part of course is keeping the momentum going....