fat0wl

This is true but keep in mind that you have to set up a pipeline for collecting the data into HDFS (Storm? batch loading?) & you have to pay for the machines.

So while your analysis is valid, there are more "costs" at play like developer time, cluster maintenance, hardware. I like to play with Spark's ML libraries but am wary about designing projects specifically around them because of this overhead, especially when trying to distribute some API/tech that you'd like others to use.

Not trying to be a downer, I actually wish the choice to go distributed was more of a no-brainer, hah. Would love for some APIs to emerge that could be used locally/distributed transparently without actually having to run a dummy cluster & data migration to run locally.

yeah i do Scala now but same deal. Where I think the OP comment really goes wrong is when he says something like "you're gonna need all that stuff and you're gonna end up re-implementing it yourself". I really don't find that to be the case... when you're developing with libs you are cognizant of your app endpoints and just add whats needed. It's better to consciously add a lib or turn a feature on than to just accept a big ball of yarn that some cowboy framework devs shat out. In the Rails app at my job it takes a ton of debugging just to figure out how many layers of decorators are firing off around every model.

I might even go so far as to say that a lot of where JVM has strength is that contexts are pretty measurable... Spring containers, Akka systems. You instantiate them and hold them basically in the palm of your hand, you understand your scopes. Any technology that allows application scope to slip out of your fingers and intermingle with everything else under the sun is something I'd be wary of.

There is a huge legal difference between getting caught doing drugs vs facilitating the trafficking of mass quantities of controlled substances. I don't think casual marijuana smokers should get all up in arms like "hey man they are just going after everybody now!", do they offer the same support to heroin cartels?

Even if you're against the war on drugs I don't think you should really take it as a personal slight when someone operating on this scale gets arrested. Unless you really 100% believe that distribution of heavy narcotics doesn't damage society.

I feel like this is already indirectly possible with cache systems... yes there is an initial load from DB or whatever, but with Ehcache for example I think(?) all those objects just sit in the JVM, and therefore should be in RAM. If you wrote some app startup batch process to stick every object possible into the cache proactively, I think you'd have essentially what you're asking for.

http://ehcache.org/documentation/2.6/configuration/cache-siz...

I'd like to at least offer some support to this -- I'm not sure I agree with all of your conclusions but since switching to Scrum mentality I definitely notice that the goals of our team have become much less ambitious... Rather than insulating for development cycles that may not yield much for a few of months but in the end equals proper enterprise architecture, the business/devs have found it much easier to focus on small iterative tasks that are much more superficial in the grand scheme of things.

I think engineering processes benefit when they are insulated from direct scrutiny of timelines. Resorting to tackling smaller, neater issues in Agile seems to be a sign that there is a lack of technical leadership/vision. This is not necessarily the fault of Agile (I don't mind Agile, I think projects can succeed with it if approached properly), I simply agree that increasing process rigor seems to be a symptom of an engineering/mgmt team that is floundering.

In short, if your team's problem is that things are getting "messy" and "hectic" because of too much crappy copy/paste code, I have seen how Agile can just deepen that wound while masking it in a veil of "tangible progress" seen in burndown charts. Then, again, if you don't have the right people you don't have the right people. What else can you say?

my company is "totally agile" (they wish) and it gives the management layer something to do cuz they are analyzing JIRA all day coming up with horse-shit charts, so of course they keep pushing all of it like its solving high-level organizational issues in ways we will experience tangibly sometime by 2018...

if you got rid of all that process infrastructure and just had a few programmers discussing proper design patterns now & then our company would be lightyears ahead of where it is now. i think everything becomes cargo cult when you refocus technical people onto processes that discourage technicality to make management feel fuzzy about their stats. people aren't more productive now, they are just gaming the agile system

process planning has its place i just feel like its becoming the focal point of discussion rather than a means to it. bureaucracies will continue to grow, what can you do other than look for a job with a more technical group that is focused on code?

i'm a former academic, i was astronomically more productive in that setting and there were no real process restrictions imposed at all. people need to be cultured into code & self-education, not process. optimize process once there is a good technical core in place, not in lieu of one

(i've also noticed that fake agile just promotes a lack of requirements docs from business... in that case i'd be better off doing damn waterfall with a half-decent reqs doc anyway...)

A doctor would probably classify this as some level of abuse, since medically accepted drinking levels are low (few drinks a week, I think?). Even if its just "1 per day", chemical dependency in order to function is usually seen as an issue. Sure, small doses of coffee & alcohol are well... small, but there are tons of people out there who rely on neither at all, where as the absence of these "small" doses would greatly impact your life. EDIT: I should mention though that I am not really against small-dose self-medication, as the alternative of getting medicated through the "proper system" is probably even more frightening. I do wish I could kick all small-dose habits though, rather than dealing with relapses (boring work without coffee is haaaard) & other compensatory habits.

As for your 10+ beers followed by welcome hangover comment, I believe Adorno addresses this in Minima Moralia and some of his other texts. His basic belief is that people are so alienated from themselves that they treat the weekend (their only time to really be themselves completely) as a respite that will ready them for the onslaught of the next week's work. Basically, the work week and attitudes that have arisen to rationalize it are a system of total self-alienation, with all traces of your "in an ideal world" personality smothered out of you.

http://en.wikipedia.org/wiki/Minima_Moralia

Anyway, sorry if this came off sounding very dramatic. I really am not like deeply worried about you, hah, I'm just saying from a theoretical standpoint what you're describing does not prove your point at all. It's more like you're just using black humor to perpetuate bread&circus mentality.

Do the repo owners get a cut? :)

I tried it on Clojure then imagined how guilty I'd feel buying it knowing Rich Hickey wouldn't get any money but the printers would get lots, hah.

heh thanks for the words of encouragement. i'm going to try now, yes. but basically.... to be fair i'm getting paid a pretty good wage so i really just need some experience to avoid looking like a dabbler. I'm looking at senior level positions now & they seem to want to see that you're really committed to a certain type of code.

I went from research Matlab / dabbling in C# at a dayjob, to RoR, to javascript, to now pretty hardcore backend Java (EE, Spring, high concurrency for last 2 yrs), trying to move into Clojure a bit more. Most well-funded companies don't seem to like polyglots from what I can tell... But yeah I will be looking for the ones that do, plus I am studying C++ to make a bit of a more lateral move into audio digital signal processing / data-mining / machine learning.

I don't know... companies tell me I'm too expensive for the level of experience I have, yet all the high-paid seniors I work with are awful so I don't want to lower my salary expectations unless I can find a company thats actually worth sacrificing for.

yep there are also a lot of situations in enterprise where you don't exactly know how your code will be used in near future (in <5 years it could grow to exponentially more customers using it compared to soft releases to select test customers).

Because of that I'd say.... it's not like I see a lot of people get raked over the coals for it, but there is definitely some organizational disappointment when a silly scaling bug crops up in production[1]. I think as a mature dev it is proper to internalize as many performance-related techniques as possible so that you can write more performant code with each subsequent project. Sometimes it doesn't even affect the amount of initial coding hours, it is just one of the freebie gains that comes with experience.

[1] Many more such bugs would probably occur if the company didn't occasionally get around to load-testing, so that should be part of CI or dev cycle if possible.

Honestly, I'd rather they make it a paid service than a dystopian clusterfuck of advertising espionage.

I'm more likely to quit the wonnnnderful "free" service you describe and defend because it "costs millions of dollars a month to run". Those millions of dollars are just R&D on how to get advertisers more intimate data. If they didn't pay for all that gross research, the infrastructure wouldn't cost nearly as much.

This cost-center point is key. I work for one of the big international electronics corps as an enterprise web dev, but they have no interest in the quality of our work. They don't understand it, they have no belief in codebase maintainability (everything is new development or bugfix, basically as soon as an app is written it becomes frozen legacy code that will sit for 5+ years until a rewrite is finally budgeted).

A lot of the superiors they hire really affect the architectural policies of the company. For example, we can use Spring & jQuery but little else. Now they are trying to write apps with heavier front-end & getting into trouble cuz its a mess of jQuery callbacks with no real js framework. Why? The main architect has never heard of React, Angular, backbone, etc. (not to mention the fact that I'm one of like 2 devs who can actually write js, and unfortunately the other does not possess the equally rare skill: "uses proper design patterns"). You'd think I was joking until you met a bunch of corporate consultant programmers.

Basically, these companies try to boil web development into an old-school "IT" process but the level of intellect is astonishingly low. I've been lookin for a new gig at a tech company for quite a while but it was tough to make a change cuz this is my first gig out of grad school, had to prove myself a bit & build resume.

The only relief is that the corp "cost-center" IT kindof job I have is such a mess that its a good place to study & relax, pace of work is veeeeery slow.

o man i would love to try "brothers" :) does the offer still stand? i am one of those "controller" pc gamers...

(i am "fat0wl" on steam too!)

hah i am jealous of you both... i don't get the kickbacks, i just get one more team of consultants that i have to work with on every project who relish the tiny bit of authority they have now been granted (Message Broker team thinks they're hot shit cuz they are a bottleneck on every project & we can't revise our service interfaces without their permission).

Funny thing is, all the consultants who specialize in these products are awful pseudo-programmers & I end up bypassing their techs (WMB, BPM, etc.) with pure Java solutions whenever possible since the overhead of speaking to them creates like a 10x productivity loss. If the nosy busybodies (awful below average pseudo-programmers within our own group) didn't tattle to mgmt to help them reign us back in, the whole company would be none-the-wiser & we'd get work done at breakneck speed... ALL MOST OF US NEED IS A GODDAMN SPRING OR EE CONTAINER + THE JVM, hah.

this may be true. I use a lot of IBM server/development techs & pretty much all of them are missing the magic they claim, boiling down to "confusing forced development paradigm that bleeds millions from your company in infrastructure and niche developer costs" (Web Experience Factory, Coremetrics, Portal etc.). None of the out-of-the-box-everything-integrates-seamlessly aspect is there, despite that being the whole basis for these monstrous sales prices. In short, many of their products have laughably little value and even non-technical corporate management is realizing this.

(You can tell a lot of the software is sold with IBM thinking "We'll tell them its their fault for a couple of years unless they hire a few of our consultants, then if they realize the product actually sucks hopefully we will have an upgrade ready by then.")

However, whenever I see articles from that Cringely dude (the one referenced repeatedly in this summary) I have to think "eh, maybe.... I'll believe it when I see it". He is pretty much an anti-IBM conspiracy theorist from what I can tell, though of course they are a strange enough organization that a lot of what he says turns out to be true.

http://en.wikipedia.org/wiki/Robert_X._Cringely

http://www.cringely.com/

I think a big issue is that a lot of these rules need to be revised as language features evolve. A lot of productive paradigms are no longer "pure" code.

I see a lot of comments kindof poo-pooing annotations, but one of the better Java devs I know is convinced that annotations are the solution to code readability/overengineering in Java -- in essence, custom annotation processors can replace both the need for interfaces and abstract classes.

One of the biggest problems I see is simply that the schism-ing of modern design paradigms means that debugging tools have to play catchup and therefore make code seem a bit less linear. But the reality is, through IoC/AOP/annotations, developers are often reducing the number or interfaces to traverse and making the code more readable, while at the same time actually making it more generic (your class doesn't have to conform to so many standards if you can tack the annotations on whatever fields/methods you want). Should someone be introducing a proxy layer for every class just in case they need to fit it into a more advanced design in the future? Or would it be easier to just code more literally while language/container designers work on a more seamless replacement method?

In a way, it does just seem like some of these new techniques are a hacky way of forcing FP into OOP. Lots of different design paradigms playing nicely within the same VMs, ecosystems is a nice problem to have though. :)

i did this with an AWS system because we never configured security keys properly -- basically got tired of whitelisting 1 IP at a time for development tasks, especially since I had issues accessing from behind corp firewalls.

client told me they got a big IO bill from AWS & i was deathly afraid it was from bots just probing the SQL server ports. thankfully it wasn't (it was from s3, they are a media heavy site with many users) but it reminded me not to just leave things accessible via internet, cuz even though it seems irrational to think someone would just probe you over & over, the fact is you have to pay an AWS bill, & attackers are just operating from their laptop.... if the amplification is right they can cost you looooots of $$ with no cost to them.

amen. there are downsides to the chaos of consulting that i don't particularly enjoy but the only time i ever spoke to HR was when they handed me my badge, and there is no such thing as a "performance review", reprimands, etc. (lots of annoying memos about corporate policies at this particular client tho). In fact, I talk to my real "boss" about once every few months, in between it is just scrums with PMs

I pretty much just go to work, chill out, and hope I can find a better gig sometime in the near future (I'm on basically an ever-renewing contract but the vendors in our system are horrible about rate increase).

i kindof believe all languages should have some ability to run with a browser-based GUI... why bother with all the hair-brained implementations (Swing, Qt, etc.) when we could just have a "desktop mode" (altered security) version of Chrome pop up to provide GUI. it wouldnt even have to follow same rules as browser really (navigation could be via buttons only, no back/refresh), just be based on html & perhaps js to allow a bit of advanced nav logic (though hopefully not enough to start polluting app logic)

i believe chrome apps may have some steps toward this, though not sure... at 1 point i considered trying to make a browser-based GUI that does all local RPC communication to a c++ app, but i feel like you know.... there should be a nice standard way of doing this out there

amen... i struggle with this a lot & though its cute to think "oh but i get to write code at work & be creative!!" all of the monstrous bureaucracy is very stifling, & my creativity is definitely in spite of my job.

is this "work is play" mentality something that the new generation of office workers actually believes? Given that 9-5 (or 6) is an outdated force-fed corporate notion, people seem to just be paid in a way that is almost directly correlated with how much annoyance they will face daily. The difficulty of the work i face at a corp is laughable.... after 6 months adapting, these jobs are mostly about keeping a straight face while beefing up one of the millions of coding projects that are basically child's play for anyone competent, yet need to be done to keep corp bureaucracy running or keep people clicking for some service.

Basically, my mind reels thinking about the great things i could achieve if i were not in this chair. If you are really "into" society (a yuppie, basically) then you might think that living in the big city, decorating your room just right & going out to see an indie film after work means you're inspired & hitting creative peaks (Brooklyn is filled with people living vicariously like this). The creative geniuses of previous generations would die of laughter, given that most of them truly devoted their lives to excellence in a given field in ways that were basically pathological.

There are rare counter-cases but I appreciate you (peterwwillis) pointing out that many of these examples are a complete crock. "Did you know ______ was a banker when he/she wrote his/her first symphony?" This can mean that ______ was either extremely well-adjusted and working in off hours, or was extremely irate and doing it in spite of his/her job. The anecdote alone has an implied simultaneity that is usually grossly distorted to serve as a success story to inspire continued enslavement of bright, delusional minds.

It's not that I believe EVERYTHING in the world is a corporate conspiracy, but basically you know whether or not you have a shit job that is destroying your creative mojo, and a few flaky anecdotes cant change that... time theft, maybe. :) But even that is tough, since many companies shun remote work ("oh no the developers might complete tasks efficiently & then enjoy a sandwich instead of working"). They wouldnt keep you strapped in the (mental) electroshock chair if they didn't want to come by & press the button.

i understand your point about history but disagree with your conclusion...

at this point its not like Adobe would have to re-architect the system cuz im sure when you click & use a tool it is not going "uhhh photoshop system, man, could you like look into this screen & blur stuff" it is calling a well-defined method with an x/y param. They just need to alias those methods from a simple command shell

thanks -- i didnt realize #1 was the case, & good point about the pipeline loop... I guess since I just use the CLI utils in different ecosystems it doesn't really occur to me how they are written, nor do I really care.

But I understand on principle, for example I work on JVM & hate when ppl bring in random code that make it hard to test across the whole system. Seamless integration of all tools & unified protocols are nice.

are there a lot of applications where this matters? maybe for CLI utils or something?

i mostly make web-apps or data-mining apps, never really understood this criticism of long (1 second type of long, maybe 10 for EE server) startup times for jvm since i can't imagine many apps worth putting a ton of development care into that aren't worth a 1 second wait to run.

i'm not trying to be a smart-ass (i dont believe everything should be blindly forced onto jvm), really just asking why this is such a common concern when in my mind it is an edge-case

heh i am always nervous to use this macro... while i agree it often makes things much simpler & more linear i feel like working without it would probably lead to quicker learning of how to mentally organize/dissect the normal syntax

In enterprise Java there are quite a bit of tools/frameworks designed to do this. While I don't particularly like the one they use at my company (bloated quirky IBM code-gen tool), it makes me see the merits of this approach (everything is basically a single page app & sections of the page are only replaced as necessary -- it is a Portal solution). Between that & seeing the JSF/PrimeFaces components my friend is always messing around with, I see the breakdown as 2 camps really -- People who like to write js vs. people who don't...

For the people who do, pushing data & logic into the client is fun, working with js, working on the actual "single-page app" & updating logic/css/html simultaneously.

For people who are more into static typing / web back-end, & don't particularly like the process of writing js, there is a nice movement toward dynamic pages where the updates are concocted through data-binding & behind-the-scenes ajax (generated by the component libs...). For a lot of enterprise Java devs, the dream of having an ultimate single source of domain logic in the Java layer somewhere lends itself well to this solution. You code typical OO paradigm and just try to integrate those models naturally into the front-end binding.

this is my whole point I've been ranting about -- Spotify absolves peoples' guilt but they are just feeding the middleman, who seems to be taking the WHOLE pie.

i think pure torrents + physical sales would actually result in better money for artists than the same situation + Spotify.

there are tons of good bands that are not on Spotify, even modern (florida noise punk scene is really easy example)

hah great for Pitchfork... i'm sure they are making much more than the artists off this deal.

regardless, they don't cover the bands i like and half the music i like isnt on spotify so i just do it the old-fashioned way using discogs.com, downloads, and some good sites that have "related artist" webs. wouldn't you know it, the best site i know for related artist webs isn't a major industry service, its a torrent site! & they accomplish it trivially through user-edits...

so i guess my point is i think its true that Spotify can be convenient for some people, but it doesnt refute my point that they are not trying to do anyone favors (unless it fits their current branding strategy). Spotify is not some techno-anarchistic's grand vision of the music industry. it's not helping music fans discover more about their tastes, it is generally colluding with industry to guide listeners toward their profit sectors. I wouldn't be at all surprised if there is a complex sequence of kickbacks between labels, Spotify, and the venues themselves, designed to minimize artist royalty/payouts. If you follow the music industry you know the deals for artists are generally pretty bad & I don't really believe in the "but I go see them live!" rationale because some artists don't tour or aren't big arena acts. AFAIK it's really hard to recoup tour costs unless you are playing decent-sized venues at every stop.

i guess it is just hard for me to get into the feelgood mentality of the music scene when i feel like the technology available could allow us to be much more taste/trend/industry-agnostic but instead seems to just be an extension of industry power (Pitchfork being a major industry player as well...), and I generally feel like the quality of the artistic output is in decline so I don't want to buy into most of the over-produced crypto-pop junk they are selling (fans of 60s/70s/even 80s music are accustomed to a much greater wealth of musical diversity). As I mentioned though, I am not into music journalism i just use artist webs. Maybe if there were some tastemaker I really wanted to follow I would see this as more convenient but I just don't experience music that way. Or at most I'd be like "well, convenient for me, sucks for the artist".

I think I mentioned in my previous comment -- it's not that I can even prove the pricing model false, I just know it doesn't do shit for the artists... 1 sale of a self-published physical album is more profitable for most of them then a year's worth of Spotify listens.

i bet a solely torrents + physical sales model would actually be MORE profitable than involving Spotify... Spotify to me is really only a mechanism for absolving people from the guilt of theft, and conveniently it funnels money back to yet another corp.

Cool! As someone with more experience than me in this, would you mind responding quickly to these points? I will give my personal opinions but if you can trump with more info that would be cool too. :)

1. I agree that JMS sounds like a hassle but is that really necessary? I would think that you can batch process data on an EC2 instance, then pick it up in your local code directly using AWS APIs... not sure.

2. I am not so familiar with the Lambda system but I'm also not sure how it would scale db as necessary (item "g" in your list) thus overall processing time would still be bottlenecked by other resources (database IO, for example), no? I agree with your points but in all these cloud-compute scenarios I always wonder "Are we trying to reach a theoretical limit of fastest-possible computation, or just reach some reasonable saturation point close to the natural bottlenecks/throttles of our system integrations?".

3. Having been burned a few times now by over-optimizing when considering cloud I would probably now first consider just picking a slightly oversized EC2 instance and throwing some high-performing code onto it (Java, C++). Dynamic languages + auto-scalable resources (though I'm talking about web hosting in particular now) seems to drain clients wallets more than anything. At this point I'd actually recommend anyone with new web infrastructure to just buy a static instance and write optimized Java rather than trying their hand at auto-scaling Ruby/Python/Node. Do you notice a similar issue with your clients regarding code optimization vs. auto-scaling?

This is a cool thought exercise I suppose, the idea of within a program throwing a particular function to an offsite parallel-compute engine. I just imagine it will complicate platform integration & bleed money given that it is in js (plus API headaches abound, as you mention)... If the same concept of dynamic provision/deploy/process/collect could all be done from a Java/C++ app I'd be very intrigued, but I suspect it is already near possible or achieved trivially through AWS APIs, albeit it probably with a manual deploy involved.

I think I remember this concept in Matlab back from when I did some research in grad school -- basically an instance of Matlab can be setup as a compute server, and the parallel processing functions of Matlab code on other computers can portion out work to it. This is the ideal model in my mind.

1. Write high-performance code in any language with some function that should be happening remotely in parallel.

2. Configure AWS to auto-provision the resources necessary.

3. Execute code, have it behave as if it is all running locally.

Really all these things can already be achieved with SOA, RMI, message queues, etc., the trick is just in making it transparent to the programmer so there is no deploy step involved. With the right spec it could even become platform agnostic (change small config file somewhere to target different cloud platform... would be nice to see a JSR about that in the near future!).