subprotocol

In chrome it says "Loading assets, please wait..." and hangs. but it works for me in firefox

The scariest part of US internet dominance isn’t vendor lock-in, it’s executive branch chaos engineering.

I think the hardest part of estimation often gets glossed over: genuine technical unknowns. Not "we didn’t think hard enough," but cases where the work itself is exploratory.

Note taking is actually a big part of rocket science: Step 1. Take notes on rocket science Step 2. Discover and distill from notes the first principles that help decouple yourself from analog thinking Step 3. Engineer the most efficient rocket that can practically get you there

I appreciate any feedback on this demo, as well as on the broader concept. There are many little areas I have in mind to tweak and improve the model training. My hope is to further refine and improve, and eventually get this working on touch devices and Chromebooks—and just give it away for free.

I'm an IC and can't stand scrum. However I regularly work with teams that are steeped in it. It's not really an issue because I always choose my level of participation when working with one of the scrum teams.

If I think the level of process and granularity is overbearing I say "no thanks" and pull the "Individuals and interactions over process and tools". If there is one thing that I wish I would have learned earlier in my career is that it's really ok to say "no thanks".

Because I'm in a zoom meeting while reading hacker news.

yes, was wondering about this too

This hits home. Great way to learn on a little risk free side project, where you can pivot on a whim. This is how I like to learn.

Unfortunately at work we have SCRUM teams that practice this methodology religiously, but not on side projects. One team spent two months building a buggy functional wrapper around gRPC. When asked why- "We did it because it wasn't functional". The result is gobs of wasted cycles building overly complicated and abstracted garbage, and then further amortizing that over time with ongoing escalations.

I'm not sure how to help them gravitate away from shiny things to focus more on outcomes (simplicity, reliability, speed of delivery, meeting needs and creating value, etc..)

This. I know many folks who have a 30min - hour commute each way (pre-pandmaic). If they lived a bit closer, and spent that time walking to/from work instead of driving, there you go.

Wow yeah, this made my day! It's nice not waiting 10 seconds for a video I don't want to watch to auto start playing (just so I can stop it).

FTA- "We are aware of a bug in Chrome that is impacting how cookies are cleared on some first-party Google websites. We are investigating the issue, and plan to roll out a fix in the coming days."

Does anyone know what javascript library they are using to generate the their sankey diagrams?

It's used everywhere in analytics applications and particularly noticeable at scale. For specific example- We use roaring bitmaps (also from Lemire) in an inverted index to make count queries instant (depending on the query and encoding, top-n queries do 20B events per second per core in our application).

Very cool! I love geeking out on analytics tech and look forward to studying its design further. My take as I see it so far (please correct me if I'm wrong)-

* As a datapoint Pinot/Druid/Clickhouse can do 1B timeseries on one server. AresDB sounds like it's in the same ballpark here

* Pinot/Druid don't do cross table joins where AresDB can. My understanding is these are at (or near?) sub-second which would be a very distinguishing feature. I'm not sure how this will translate to when distributed mode is built out, as shuffling would become the bottleneck. Maybe there would be some partitioning strategy that within a partition allows arbitrary joining or something?

* Clickhouse can do cross table joins, but aren't going to be sub-second

* AresDB supports event-deduping. I think this can easily be handled by the upstream systems (samza, spark, flink, ..) in lambda

* Reliance on fact/dimension tables. - This design/encoding is probably to help overcome transfer from memory to GPU, which in my limited experience with Thrust was always the bottleneck. - High cardinality columns would make dimension tables grow very large and could become un-unmanageable (unless they are somehow trimmable?)

Hi HN! I just open sourced Genetic.js, hope you enjoy!

How big is this dataset (what does each event consist of)?

Standard clickstream data, maybe 50-ish parameters per event.

What sort of queries are you running? How is the data stored?

Depends on the use-case. For sub-second adhoc queries we go against bitmap indexes. Other queries we uses RDD.cache() after a group/cogroup and answer queries directly from that. For other queries we go hit ORC files. Spark is very memory sensitive compared to hadoop, so using a columnar store and only pulling out the data that you absolutely need goes a very long way. Minimizing cross-communication and shuffling is key to achieving sub-second. It's impossible to achieve that if you're waiting for TB of data to shuffle around =)

How many machines / cores are running across?

Depends on the use case. Clusters are 10-30 machines, some we run virtual on open stack. We will grow our 30 node cluster in 6mo.

Maybe Spark doesn't like 100x growth in the size of an RDD using flatMap

You may actually just need to proportionally scale the number of partitions for that particular task by the same amount. Also when possible use mapPartitions, it is very memory efficient compared to map/flatMap.

Maybe large-scale joins don't work well

Keep in mind that what ever happens per task happens all in memory. For large joins I created a "bloom join" implementation (not currently open source =( ) that does this efficiently. It takes two passes at the data, but minimizes what is shuffled.

I use spark a lot and my experience has been quite the opposite. The queries I run against spark are billions of events and results are sub-second.

I could only speculate as to what this users issues were. One difference between hadoop and spark is that it is more sensitive in that you sometimes need to tell it how many tasks to use. In practice it is no big deal at all.

Perhaps the user was running into this- the data for a task in spark runs all in memory, whereas hadoop will load and spill to disk within a task. So if you give a single hadoop reducer 1TB of data, it will complete after a very long time. In spark if you did this you would need to have 1TB of memory on the executor. I wouldn't give an executor/JVM anything over 10GB. So if you have lots of memory, just be sure to balance it with cores and executors.

I have seen spark use up all the inodes on systems before. A job with 1000 map and 1000 reduce tasks would create 1M spill files on disk. However that was on an earlier version of spark and I was using ext3. I think this has since been improved.

For me spark runs circles around hadoop.

May be of relevance: https://cwiki.apache.org/confluence/display/SPARK/Powered+By...

I don't know what you would count as major deployment, but I've deployed a 30-node cluster on HW for running sub-second real-time adhoc queries. I've also run many smaller 10-20 node virtual clusters on open stack. It is a rock solid platform. Our hosted ops loves it because it just works.

The amazing thing about spark is how insanely expressive and hackable it is. The best way I can describe it is this:

* Hadoop: You spend all of your time telling it how to do what you want (it is the assembly language of bigdata)

* Spark: you spend your time telling it what you want, and it just does it

mostly what you need to beat Hadoop is to make something practical, which lets developers be expressive rather than wrestle with overdesigned nonsense.

It would be difficult for an article to convey this. Having used both Hadoop and Spark, practicality and expressiveness are precisely what made me fall in love with Spark. You can do so much with so little code. Don't take anyones word for it, see for yourself. Download the code and run the interactive shell, takes 2 minutes. It was totally mind-blowing for me.

Spark is a wonderful project, I blogged about it just the other day: http://subprotocol.com/2013/06/17/spark-darling-of-big-data....

Spark make doing MR easy. I've used other frameworks on hadoop MR, but nothing compares with the ease with which you can express computations using it. And it does both batch and real-time/streaming. It is a very well thought-out project.

I've used javascript for many many years and, as of 6mo ago, started using Scala (I do a lot of distributed programming on hadoop, storm, and recently spark). Scala is now one of my favorite languages because of Spark (http://spark-project.org/). Scala very nicely blends the functional and imperative styles and it always favors programming without side-effects. Anything you can do in java you can do in fewer lines and you will see fewer bugs.

That said- it's not a superior language, just a different paradigm. With Scala you get very strong typing, functional constructs (so incredibly powerful), matching, options, very clean closure syntax, etc.. You'll eventually fall in-love with _, too.

I haven't used scala.js yet, but here are some downsides to Scala proper: * slow the compile * tooling isn't great (this true of anything jvm to some degree though)

Great question, I wanted to (at least initially) focus on the experience for paying customers, and hone in on creating value for them. Though, I can see how it may very well make sense to do something like a free extra limited account sooner rather than later.

Also, when you signup I pull in your historical data and begin back-crawling immediately. This way you don't have to wait 3 months for the charts, trends, intensity maps, etc.. to fully populate before you take action on the data.

Hi HN, I just launched my startup, Referrerly! Others in a similar space are hyper-focused on SERPS, data-puking backlinks, and completely miss the bigger picture. Referrerly helps you see how your actions and content are resonating online. I wrote a 'back-crawler' that makes backlinks actually meaningful, by filtering out junk/spam and giving you the context around the link. I also collect and aggregate on a whole host of data points, bring them into beautiful charts, trend it for you, etc.. It makes it all very simple to follow and leverage.

With that said, I would love feedback! If you sign up now there is free trial, and you can use this nifty code for an extra 10% discount: analytics

Cheers

"what do I do?"- If you can jump ship ASAP, there is tons of opportunity out there.

I was in a similar situation (not quite as bad), but I couldn't leave because of a business obligation. I survived by insulating myself from the dipshits. We formed an isolated 'blackops' team within engineering that was beholden to nobody but ourselves. We were able to move at lightspeed compared to the rest of engineering.

After a year our small team actually began a major pivot for the company (culture, technology, etc..) and breathed new life into the company. Most of the diphits were jettisoned and the others were moved into supporting roles.

Interesting, sounds like puppet/chef at the OS level.

It all started as play coding. The deeper I went the funner it became =)

Thanks =)

Great question- I was reading about verlet integration and started doing some coding to play with it, and I just kept on going with it. Would love some help with it! First thing I would like to do is abstract out canvas.

Thank you! I have actually not played around with SVG yet. Is it as easy as canvas? I am very new to canvas and found it to be super easy to pick up and run with.