HN user

demian

321 karma

Data Engineer

Posts1
Comments164
View on HN

But that can change quickly, history has lots of examples of how asymmetric power relations between the owners of the means of production and the workforce can end in conflict and a heavy push for a regulatory framework.

I can attest to the fact that people from the US does not dress more casual than people from the Southern Cone of South America (Argentinians/Chileans/Uruguayans).

Hearts on Twitter 11 years ago

I "love" this tweet about how 10 people died on an earthquake last week.

It has the same problem as "Like".

Basically you need to go into this knowing it'll take 3-4 years before you can show any traction.

I hope some visionaries VCs will start thinking in those terms, and start investing in this kind of high-impact life-time ventures.

Isn't this solved by paying a fixed amount, and only paying if the project works and is well done?

The problem with that approach is metrics. How can to specify that the softwere "works"? How can you measure if it's "well done"?

I worked on a project using that approach and most of the time was spent being basically lawyers with specs instead of contracts. Horrible experience.

It seems to me we are in the same place Front-End Web was a couple of years ago: There is an explosion of new tech and frameworks. BUT there seems to be a bump for the adoption of new tech for data engineering, the HIGH cost of migrating terabytes and terabytes of the company's data to a new system.

Migrating and integrating data is incredibly expensive. When you implement a new system you either have to migrate the data from the older system (= expensive) OR having both systems working in parallel one for "older" data and one for newer (= really expensive)

Currently the established risk-averse corporations, the owners of most of the non- social media and internet data, seem to be still testing enterprise Hadoop distributions like Cloudera and Hortonworks for daily enterprise data operations, and maybie have some projects on R&D for harnessing new "types" ok data (like sensor data from a factory floor or high granularity transport data for supply chain).

Still, I hope the best of the new tech can get a place inside the modern corporations that work on important problems like Energy and Healthcare.

Some people argue that Uber and AirBNB are not actually changing the way we use capital on a fundamental economic level, but are instead using new channels in a way, and with a magnitude, never used before.

IMHO I tend not to agree with the hole "Sharing Economy is The Next Capitalist Revolution" thesis.

By which metric would you measure this "efficiency"?

The difference between VC-backed-startups vs traditional R&D departments is that the first one seems more short sighted, and with a tendency towards a narrow subset of IT problems with high scalability and disruptive potential, while the second one works on a wide array of industries and applications where the parent company already has the benefits of economies of scale.

Tool-level data integration is supported by tools like Spotfire, but for analyzing high volume of data you need to mantain a data warehouse using ETL code to integrate the data from all the diferent sources.

For better performance you may also need to implement an OLAP tool.

This is capitalism critique 101.

Marx was a tech writer in a way, he wrote about the emerging power asymmetries between the people that owned the machines and the people that operated them. I know you guys are not big for socialists authors in the states, but you should at least read the classics.

PS: I'm not a communist.

Not really into using darwinian language, but I like the focus on fitness.

It may work better for customer-facing products ("Market Fit" is a thing), but It may be harder to sell it for internal products ("Functional Fit"?)

"Culture of Failure"

Good luck selling that to upper management.

On the contrary, quick iterations are about reducing the risk of building the wrong thing. Of course in the normal "blaming" corporate culture this won't work, because departments are always competing over budget and deliverables that are not "perfect" will be used as ammunition against the department where the product came from.

This is the same problem people faced at the turn of last century.

The solution at the time was a mix between unionizing labor and adding regulation, like anti-monopoly laws, from the public side, and basically "inventing" the service, tertiary, sector and modern consumerism from the private side.

The problem with this solution now is that there is no quaternary sector in sight to absorb the displaced from the automation of the tertiary sector. Some researchers are talking about an "experience" sector that would be inmune, at least for now, to automation, but the volume of the current job market doesn't seem to indicate that it could replace the "service" sector any time soon.

Also the current market solution seems to be sharing economy businesses, which doesn't seem to scale from a social standpoint.

Right, he fits a role in the company's narrative, both for the inside and outside.

He is perceived as twitter "creator/inventor" after all, so I can understand why the employees are thrilled.

Optimize for two variables:

* User Validation or Knowing You Are Building The Right Thing: data flows, ui design, modes, triggers, and so on

* Tech Validation or Knowing You Are Building The Thing Right: unit tests, performance tests and so on.

Focus on selling the devs on this idea.

The rest of the process formalities can be negotiated.

EDIT2: Be careful to include all the dependencies required to fulfill the UV and TV, like scope management in UV. Every part of the process must be justified either as critical for UV or TV.