HN user
GCP is pretty pleasant overall. The API, command line, and UI are all pretty good. Vast majority of functionality is supported in all three places too. Like all providers, they have a few services that suck but overall Google has done a great job at developer experience.
There are several use cases where ML can help even if it isn't perfect or even just better than random. Here is one example in NLP/search.
Let's say you have a product search engine and you analyzed the logged queries. What you find is a very long tail of queries that are only searched once or twice. In most cases, the queries are either misspellings, synonyms that aren't in the product text, or long queries that describe the product with generic keywords. And the queries either return zero results or junk.
If text classification for the product category is applied to these long tail queries, then the search results will improve and likely yield a boost in sales because users can find what they searched for. Even if the model is only 60% accurate, it will still help because more queries are returning useful results than before. However you don't apply ML with 60% accuracy to your top N queries because it could ruin the results and reduce sales.
Knowing when to use ML is just as important as improving its accuracy.
If you can write a SQL query or a set of SQL queries to do your transformation, then you can use DBT. DBT doesn't do transformation itself rather it helps you manage all the dependencies between your SQL models. Whether you can use SQL depends on your data and database/warehouse functionality. For example, JSON parsing support is pretty good now in many databases and warehouses. If your objects can be represented as JSON, then you could write SQL via DBT to parse the objects into columns and tables.
DBT has two main innovations. First, everything is a SELECT statement and DBT handles all the DDL for you. You can handle DDL yourself if you have a special case too. Second, the ref/source macros build a DAG of all your models so you don't have to think about build order. There are other innovations but those are the main ones.
You can give it truly pure SQL in both models and scripts, and mixing in Jinja if you need it for dynamic models. But I'd recommend at least using ref/source.
I think using a data warehouse as your data lake or lake house is optimal. Even for data that isn't relational. Storage is so cheap now and is decoupled from compute costs for several providers that I don't even give it a thought. You get a fast, scalable SQL interface which is still nice and useful for non-relational data. Then all, or most, of the transformations needed for analysis can be pure SQL using a tool like DBT. In my experience, it greatly simplifies the entire pipeline.
For my search use case, documents are mostly single topic and less than 10 pages. However I have found embeddings still work surprisingly well for longer documents with a few topics in them. But yes, multi-topic documents can certainly be an issue. Segmentation by sentence, paragraph, or page can help here. I believe there are ML-based topic segmentation algorithms too, but that certainly starts making it less simple.
Google has a distributed embedding matching service in preview: https://cloud.google.com/vertex-ai/docs/matching-engine/over...
I guess it depends on what you mean by "simple". The algorithms are complex but there are good tools that implement them. I would imagine smaller companies would use off the shelf tooling, and I would argue that is simpler. Vector embeddings are so unbelievably powerful and often yield better results than classical methods with one of the good tools + pretrained embeddings.
Specifically for search, I use them to completely replace stemming, synonyms, etc in ES. I match the query's embedding to the document embeddings, find the top 1000 or so. Then I ask ES for the BM25 score for that top 1000. I combine the embedding match score with BM25, recency, etc for final rank. The results are so much better than using stemming, etc and it's overall simpler because I can use off the shelf tooling and the data pipeline is simpler.
Yes I know for sure. Postgres search is essentially an easier to use regex engine. If you have a recall-only use case and/or a small dataset, then that works great. As soon as you need multiple languages, advanced autocomplete, misspelling detection, large documents, large datasets, custom scoring, etc you need Solr or ES.
I disagree based on the trends, especially in the US where few, if any, new coal plants are being built and most existing ones are scheduled to be converted to natural gas or shut down. But let's assume that a significant amount of coal will continue to be burned over the next 20 years. Do you think we should stop innovation in other sectors until we are off coal?
Coal is on its way out. We don't have to wait for the perfect energy solution before we can improve agriculture.
If you are going to put creds in git, at least use sops.
I grew up in Cincinnati. You are correct, however traffic is bad enough now that a subway or light rail would make a big difference. Unfortunately it is unlikely the voters would approve a tax increase to support that.
I personally try to make decisions using probability. I understand where you are coming from but there is one factor missing in your analysis: this situation is literally life and death. That changes the math a bit to something more akin to "better safe than sorry" in my opinion. We have found a guaranteed path out of this mess. There may be other faster paths that save more lives, but it could also end up killing millions more too. I'm all for experimenting but I take issue with making the experiment the policy when lives are on the line.
We have no data for mRNA vaccines except those trials. We don't know if the historical data on traditional vaccines are applicable here. As such, it is also possible that a single dose isn't good enough or a single dose is good enough but only for a couple months. We just don't know and it would be way worse if it turns out we have to re-vaccinate everyone because we were impatient.
Yeah I agree, I would like to see some percentage be lottery based. I think you should still need to write a proposal, but if the proposal isn't selected then it goes in the lottery pool. There are two reasons to still write the proposal: helps organize the researcher's thoughts and shows the researcher is serious.
Where do you think those calories come from?
no dietary need for sugar
Can I assume you mean simple sugars that are added for taste? If you literally meant what you said, well you will die without carbohydrates (sugar).
I think a better analogy would be living near an airport like I do. Depending on the day and weather, even my small airport is quite loud and annoying. My neighbor's motorcycle is also louder than the planes, except the occasional military jet, but that noise is far less frequent than the planes. I think drone delivery will mean everyone in cities and suburbs will feel like they live near a small airport on a clear day. I too am not looking forward to the flying drone future. I think we should be working on reducing noise as much as possible, not create more in the name of slightly more convenient consumerism.
I think Citus can be used effectively for that scenario now.
The post makes it sound like it is cheap to get started, but 32 ETH is currently ~$20K. If you already have the necessary ethereum then it is cheap.
In practice, it rarely turns out that way. I have to deal with large, mature Java codebases for some of my work. The good thing is that the code does just about everything well and rarely breaks. The downside is that when something does break, and I have to debug the code. At some point in the Java world, best practice became building abstraction on top of abstraction on top of abstraction. And often these abstractions just call the next abstraction. Well that makes finding the offending line of code extremely difficult and time-consuming unless you are an expert of the codebase. Had the exact same code been written with less abstractions, debugging would be a lot easier.
I am not against abstractions, but I think they lead to hard to read/debug code when overused. I think they need to be used wisely rather than the default.
"Here is why the consensus is _______" is not argument by authority, correct.
"The ________ theory/hypothesis is fringe, the consensus is ________" is argument by authority.
See the difference? Phrasing matters in this case. All too often it is phrased the second way. I don't expect someone to literally enumerate all the evidence for or against something, but there is a middle ground between that and just saying "well consensus is ______".
Most people on HN are aware that the consensus, a form of argument by authority, is dark matter. It is also known that the dark matter consensus has problems despite billions spent on research over decades. When there is an experimental result that contradicts the leading theory, that is interesting and worthy of media attention. If this experimental result was only published in popular media, then I too would likely ignore it. However it was also published in a major journal. Like you said elsewhere, that doesn't mean it is right. But I think it is worthy of attention and discussion and should not be dismissed out of hand because it doesn't agree with consensus.
Argument by authority is bad science too.
You have all that DIY too, except maybe shock bills, but it's your problem to fix. Like with everything, extremes are bad and all managed or all unmanaged are both bad. My personal guideline is use off-the-shelf services if they are cheaper than DIY, including labor, or if DIY provides no competitive advantage and off-the-shelf is cheap enough. But if DIY is much cheaper or gives a competitive advantage, do it yourself.
McKesson is handling distribution for Moderna, and one of the main goals of Operation Warp Speed was to ramp up production using companies who are good at it so the vaccine developers wouldn't necessarily have to do large scale manufacturing themselves.
Word bigrams without stopword removal on the title would help in this case as well.
Usually managed systems cost less if you factor in salaries. Like everything, it depends on the specifics though.
One dose of MMR vaccine is 93% effective against measles, 78% effective against mumps, and 97% effective against rubella. Two doses of MMR vaccine are 97% effective against measles and 88% effective against mumps. MMR is an attenuated (weakened) live virus vaccine.
From:
https://www.cdc.gov/vaccines/vpd/mmr/public/index.html#:~:te....
Yeah. What I do is create a view that does all the joins then the middleware just needs to do "SELECT * FROM my_view". If the DB has good JSON support, I will also convert the data into an ES index request with SQL so the middleware becomes even simpler.