luispedrocoelho

It's not clear to me if google is allowed to demand that non-google app phones not be called Android or if that would also be considered anti-competitive.

If they must allow Android branded phones that do not come with google apps, then the EC is making it harder for consumers to choose by disallowing the normal use of a brand to signal what it is that you are buying when you buy something. People may buy an Android cheap phone thinking that they are getting what is now an Android phone and end up with a shittier product.

Under a normal market, these OEM-branded phones would be sold as "ShitPhone OS 12.0" and it'd be clear that you have the option between paying slightly more for Android, a lot more for iOS or go with ShitPhone 12.0

As a matter of law, the EC may be right, but as a matter of user experience, the idea that the Android market suffers from not having enough OEM-installed software on devices is a bit of a stretch. In all android phones/tablets I have seen there are a few shitty OEM apps that I have never found to be even close in quality to the google versions.

The EC claims that google have "denied European consumers the benefits of effective competition in the important mobile sphere", but I fear this will mean that cheap Android phones will have a shitty OEM-branded webbrowser and some random search engine link.

But he did not purchase health insurance, he purchased travel insurance.

It seems to me that the whole thing is "I bought travel insurance and I thought I would be getting global health insurance, but it turns out it's really travel insurance."

I guess because the NHS is paid through your taxes it may not have been so blindingly obvious to someone in the UK that anything similar to what he is asking for would have cost at least "300£ per month" (and that is assuming he is young and healthy) instead of "300£ for 18 months".

In reality, most claimants are going to spend much less than 1% of that £5 million limit before they get flown home. It only really applies to people who are at death's door and end up with a long stay in intensive care, or need a fully-staffed medical flight to get them home.

Exactly, but you're saying it like it's a bad thing instead of "this is just what travel insurance is and why it's so cheap."

I never claimed it was in bad faith and I don't think it was. Considering incentives and conflicts of interest is perfectly normal and does not mean that someone is operating in bad faith.

I just claim that it's reasonable for the company not to take it at face value as the doctor is not impartial and may have been considering a different standard for "not fit to fly." I certainly do not take it at face value. Could he have been put on a plane with extra medical assistance, for example? That may still have been cheaper than surgery.

I think what the guy did was the best choice if he can afford it, but I also think it's not unreasonable from the company to say that it was not what travel insurance covers.

The incentive to say so is pretty clear: if he flies away, not only does the doctor lose a 15,000 costumer, he can also be liable if anything does go wrong.

Maybe he was indeed in no shape to fly, but he should have argued it properly in those days. Even now, it seems more like "it would have been awful to fly economy" and not "it would have been dangerous".

To be sure, travel insurance is very often a very good idea.

But it's travel insurance not "whole-world, any hospital, health insurance" (which I am sure you can buy, but not at the same price).

From his telling, the company was not very communicative when he contacted them, but, fundamentally, saying he should fly back to the UK for treatment and they'll cover the flight does not seem unreasonable.

He complains that "unless you are injured so horrifically that you cannot survive another moment without life saving surgery, odds are they’ll just tell you to fly home and have the surgery for free." But, to me, this is exactly what travel insurance is (which is why it's so cheap). Expecting fancy Singaporean hospitals unless it is strictly necessary seems totally unreasonable.

If they were not so communicative, he also basically decided to not get back to them on the vital "is he fit to fly?" issue for a few days (he had been OK to fly to Singapore after all), until after the surgery. There were a few days to go until the surgery, so that was enough time to get him back to the UK. Also, why hadn't he gotten in touch before? He texts pictures to friends, books flights, &c, and does not call the insurer or even ask somebody else to call on his behalf?

For what it's worth, I think he made the right decision in getting treatment in Singapore at his own expense, but that was still his decision.

The reply in the uber blog is less "angry tweet" than "good peer review": it points to a very specific methodological weakness which has bearing on the final conclusions.

For some things, we do. But databases are not magical and setting up a good table/index system &c is also work and there is overhead.

Thus, if we are talking about having (for example) a webservice where queries have a form that is known apriori, then it's a good solution. If you have output data from your processing that you will be slicing and dicing in different ways which you cannot predict ahead of time, then, they are not appropriate.

(Loading Terabytes of data into a database takes a while too).

`isin` is worse in terms of performance as it does linear iteration of the array.

Reading in chunks is not bad (and you can just use `chunksize=...` as a parameter to `read_csv`), but pandas `read_csv` is not so efficient either. Furthemore, even replacing `isin` with something like `df['id'].map(interesting.__contains__)` still is pretty slow.

Btw, deleting `interesting` (when it goes out of scope) might take hours(!) and there is no way around that. That's a bona fides performance bug.

In my experience, disk IO (even when using network disks) is not the bottleneck for the above example.

1. No, that function was the bottleneck, by far, and I can tell you that >10,000x was what we got between the initial version and the final one.

2. I don't care about faster at this point. The function is fast enough. Maybe there is some magic incantation of pandas that will be readable and compute the same values, but I will believe it when I see it. What I thought was more idiomatic was much slower.

I think this is more of a case of "the problem does not fit numpy/pandas' structure (because of how the duplicated indices need to be handled), so you end up with ugly code."

That is the _FAST_ version of the code (people keep saying "of course, it's slow", when it's the fast version).

Here is an earlier version (intermediate speed): https://git.embl.de/costea/metaSNV/commit/ff44942f5f4e7c4d0e...

It's not so easy to post the data to reproduce a real use-case as it's a few Terabytes :)

*

Here's a simple easy code that is incredibly slow in Python:

    interesting = set(line.strip() for line in open('interesting.txt'))
    total = 0
    for line in open('data.txt'):
        id,val = line.split('\t')
        if id in interesting:
           total += int(val)

If I can fit the code into numpy-like structure, then Python is typically fine.

The issue is when I cannot.

That may be the case. However, my point is that we started with a rather direct implementation of a formula in a paper. This was very easy to write but took hours on a test set (which we could extrapolate to taking weeks on real data!).

Then, I spent a few hours and ended up with that ugly code that now takes a few seconds (and is dominated by the whole analysis taking several minutes, so it would not be worth it even if you could potentially make this function take zero time).

Maybe with a few more hours, I could get both readability and speed, but that is not worth it (at this moment, at least).

*

The comment about the benchmark data being large is exactly my point: as datasets are growing faster than CPU speed, low-level performance matters more than it did a few years ago (at least if you are working, as I am, with these large data).

You are commenting on the variant of the code that is fast enough that it doesn't matter.

"spending an hour figuring out what arcane incantation I need to pass to np.einsum to get the operation I want"

Yes, I have also had this experience and I hate how in the end, the code is very hard to read, while the for loop was probably trivial.

I think this is part of my argument: as datasets grow faster than single-core CPU speed, performance matters more and more.

Yep, my Haskell usage is "conduit all the way down".

I even wrote up a few utilities to make use of multiple threads while working at a high level: https://hackage.haskell.org/package/conduit-algorithms-0.0.7...

OP here.

Speed is the main motivation, but total time is TimeToWriteCode + TimeToRunCode.

Python has the lowest TimeToWriteCode, but very high TimeToRunCode. C++ has lowest TimeToRunCode, but high TimeTowWriteCode. Haskell is often a good compromise for me.

Also, with Haskell, it can be very easy to take advantage of 20 CPU cores, while I don't have as much familiarity with high-level C++ threading libraries.

There are variations where this works out in google's favor, but there is a big difference to the ebay situation where you just keep relisting the same item until it sells for a "good enough" price: each ad impression is only sold once, there is no relisting.

Not as difficult, but consuming large amounts of energy on short notice is also not trivial.

"connect a resistor to the end of a power cord" is supposed to make it sound trivial, but operating the resistors needed for absorbing the extra power on the grid is a large scale operation with non-trivial safety concerns. Also, you need to own and maintain all that equipment, &c &c.

At some cost to portability, IIRC, on Linux you can ask the operating system to keep pages in memory. Other systems will probably have similar functionality.

OP here.

Use redis if you can. Use something like diskhash if you cannot afford the extra overhead of sqlite/redis.

In my case, the alternative we were using before is an in-memory hash table built at startup from a text-based representation of the data. It was pretty good and worked very well up to millions of entries, but at the current scale we work with, it takes least 10 minutes at startup were necessary to build up the thing and it used ~200GB.

Yes, it does give you a lot more and is great.

Unfortunately, on my initial tests, it just was not fast enough.

Look at the source code license, it's MIT licensed.

We are a publicly funded research organization, so even our "work code" is open source.

OP here.

I agree and my phrasing may not have been exact. I meant that if you need to pick a number for a whole group of people, then you look at the average.

This can be in the context of industrial production where you have thousands of workers and are forced to have fixed shifts (because you cannot run the assembly line without all the stations filled).

Or, in more modern example, a public sector bureaucracy where you cannot be very flexible as these things are negotiated with a union as whole.

All these things happen in academia. Nowadays, many people regularly work from home for the quiet (I do 1 day a week from home).

I have a PhD already (from an US institution), but there are elements that are pretty specific to academic research: it's fairly competitive (and wins are not that constant) and it's very individualized. It's also one where we have almost complete control of our time. It's more like being a freelancer, albeit with a fixed salary in an often bureaucratic environment.

I agree with you 100% that most work is not like that at all.

(Btw, most PhDs in Europe do not have required classes or, if they do, they tend to be pretty easy; it's the research component that's hard)

Which is a good reason not to work 60 hours regularly over the long term, but not a reason to never put in a 60 hour work week or even a few in a row.

Elsewhere, I've written how it may be better to switch between periods of very long hours and downtime rather than a steady pace:

https://metarabbit.wordpress.com/2014/02/26/average-work-wee...