jakebol

Imagination applies also to how we organize society and “rights” we hold over nature and natural resources. Reforming outdated water law is the true fix no pipeline needed.

The southwest isn’t really over populated with people, it’s overpopulated with cows which require tremendous land and water resources.

138 MW is also nameplate capacity multiply that by effective utilization rate for ex solar (~20%) and it’s meaningless compared to the scale of the project.

Oil products are a fractional distillate of a barrel of oil. How are you going to pave the roads all these EV’s are going to drive on, or produce the plastic they consume (EV’s require ~40-50% more plastic)? If gasoline demands softens it doesn’t necessarily mean that other oil product demand will decrease at similar rates. Oil production declines over time so you need constant development even in a declining consumption scenario, and I think we are heading into a world where domestic supply will command a premium.

How is ancestry.com a startup at this point? It was founded in 1996.

Just to note that language "duplicates in sparse arrays" doesn't make sense, if you allow for duplicates it is no longer an array by definition.

Most every (analytic) RDMS database system can model sparse arrays. A sparse array is modeled by defining a clustered index on the table "array" dimensions and defining a uniqueness constraint on that clustered index. This works well with columnar storage because the data needs to have (and assumed to naturally have) a total sort order on the dimensions. Ex. Vertica, Clickhouse, Bigquery... all allow you to do this. TileDB allows for efficient range queries through an R-Tree like index on the specified dimensions.

Most real world data though is messy and defining a uniqueness constraint upfront (upon ingestion) is often limiting, so for practical use cases this gets relaxed to a multi-set rather than sparse array model for storage, and uniqueness imposed in some way after the fact (if required).

Unfortunately "its just geography" is kind of one of the talking points for not really addressing the problem. Although true, concerted reductions in pollution have happened when there was political will to make it happen (mostly through the federal gov. / EPA clean air regulations).

Ogden and Provo are some of the worst offenders for per household air pollution emissions. Like many western cities they have longish commutes (everywhere) in large cars (trucks / suv's) with a high number of cars / household and almost non-functional public transport system. For the Salt Lake Metro area, per capita carbon emissions doubled between 1980 and 2015 because of increasing sprawl. Air regulations here are spotty for personal vehicles and I'm guessing almost non-existent for commercial vehicles. Oh and the state governments solution to this is to push a publicly subsidized "inland port" that will bring increased truck and rail traffic to the valley. The leaders of these tech companies are starting to point out that terrible air pollution for parts of the year is hurting recruitment so it seems like as the money flows into this sector maybe there will be political will on the state and local side to address some of these issues.

this is true (and a property of all? cloud formats Delta, Hudi, Iceberg, parquet, etc.)

I was referring more to the fact that the cloud vendors can co-design their infrastructure and software to support their database services.

There are a lot of issues though with S3, latency, poor performance for small reads / writes, timeouts, api rate limits, api costs, and consistency issues poorly understood by third party developers.

A "thick-client" also doesn't perform well unless that client is located on a node in the same region. I think as with everything it works well in some cases and not well in others.

This is a good description, except that TileDB (the open source client) is not transactional but eventually consistent at least for S3 and other object stores.

I like your point about consuming S3 cleverly, it's often difficult to get good out of the box performance from S3 so abstracting that to the degree possible is good for end-users. The cloud vendors though are always one or two steps ahead of companies that build upon their services. AWS Redshift for instance already can pre-index objects stored on S3 to accelerate queries at the storage layer. It's difficult as a third party vendor to compete with that.

Underrated it is not, just search for Cottonwood Canyon traffic jams to see what skiing really is like here when the snow flies. 30 minutes no traffic, can easily be 3+ hours now.

Moral of the story was that his dreams came true?

https://twitter.com/mikeandersonKSL/status/12577783769258024...

they already provide this as a service

Funny that he mentions Salt Lake / the Salt lake valley as the only other experience with significant levels of air pollution. I often wonder how "silicon slopes" companies are able to attract people here, they must never interview during the winter (particulate) or summer (ozone / smoke).

Made the jump from OSX and for scientific users who don't to endlessly chase a working laptop setup PopOS and it's nvidia driver support (on a thinkpad) has been fantastic, highly recommend as well.

usually original investors get stock in the split-up companies correct? In the long run the aggregate value of the split up / smaller basket of independent companies could be larger than the original quasi monopoly.

TileDB, Inc. | Full-Time | REMOTE | Cambridge, USA | Athens, Greece | https://tiledb.io

TileDB is a technology for storing and managing enormous volumes of structured data on the cloud. TileDB efficiently stores data generated from variety of domains (genomics, finance, imaging, geospatial, etc) in a novel unified format as sparse or dense multidimensional arrays. Users can efficiently access this data via a growing number of language APIs, interfaces and our upcoming hosted service. TileDB, Inc. has raised $4M in seed funding over the past 18 months.

We are looking for frontend and backend developers to help develop a scalable service to make data storage, management, and computations easier for practicing data and domain scientists. A few features on the roadmap include designing a web admin console for data management, monitoring resources in real time, integration with JupyterLab and handling user profiles, billing, encryption and key management, and scaling / management of serverless compute resources.

We are primarily seeking:

- UI / UX Engineer (Javascript / Vue.js / Golang / K8s)

- Backend engineer (Go, CGo, K8’s)

* TileDB GitHub organization: https://github.com/TileDB-Inc

* TileDB has been featured on HN: https://news.ycombinator.com/item?id=15547749.

* TileDB blog post: http://bit.ly/2SjkeYR

* TileDB talk at PyData 2018: http://bit.ly/2BfbyJ3

Our headquarters are located in Cambridge, MA. The candidates must be US citizens or permanent residents located in the US, or Greek citizens to be located at our subsidiary in Athens, Greece.

Apply today at https://tiledb.workable.com !

My Father works as an engineer for the VA, the hidden costs of this transition is the need to rewire almost the entire VA hospital system to comply with cerner networking requirements at the cost of $100's of millions per hospital before the software can begin to be deployed.

The other hidden cost is that these ancient software systems are extremely efficient in terms of cpu / memory utilization compared to modern equivalents so the hardware requirements go up considerably which impacts power / energy efficiency and direct deployment costs for replacement systems. Not saying modernization is not a win in the long term but there are direct and indirect costs with these big transitions.

TileDB, Inc. | Full-Time | REMOTE | Cambridge, USA | Athens, Greece | https://tiledb.io

TileDB is a technology for storing and managing enormous volumes of structured data on the cloud. TileDB efficiently stores data generated from variety of domains (genomics, finance, imaging, geospatial, etc) in a novel unified format as sparse or dense multidimensional arrays. Users can efficiently access this data via a growing number of language APIs, interfaces and our upcoming hosted service. TileDB, Inc. has raised $4M in seed funding over the past 18 months.

We are looking for frontend and backend developers to help develop a scalable service to make data storage, management, and computations easier for practicing data and domain scientists. A few features on the roadmap include designing a web admin console for data management, monitoring resources in real time, integration with JupyterLab and handling user profiles, billing, encryption and key management, and scaling / management of serverless compute resources.

We are primarily seeking:

- UI / UX Engineer (Javascript / Vue.js / Golang / K8s)

- Backend engineer (Go, CGo, K8’s)

TileDB GitHub organization: https://github.com/TileDB-Inc

TileDB has been featured on HN: https://news.ycombinator.com/item?id=15547749.

TileDB recent blog post: http://bit.ly/2SjkeYR

TileDB talk at PyData 2018: http://bit.ly/2BfbyJ3

Our headquarters are located in Cambridge, MA. The candidates must be US citizens or permanent residents located in the US, or Greek citizens to be located at our subsidiary in Athens, Greece.

Apply today at https://tiledb.workable.com!

TileDB, Inc. | Full-Time | REMOTE | Cambridge, USA | Athens, Greece | https://tiledb.io

TileDB is a disruptive technology for storing and managing enormous volumes of structured data, adopting the best ideas from columnar and spatial database research to support fast updates, compression, and interoperability with scalable cloud object storage backends. TileDB efficiently stores data generated from variety of domains (genomics, finance, imaging, geospatial, etc) in a novel unified format as sparse or dense multidimensional arrays. Users can efficiently access this data via a growing number of language APIs and interfaces (C/C++, Python, R and SQL data engines).

TileDB, Inc. has raised $4M in seed funding over the past 18 months. We are looking for frontend / fullstack developers to help us with designing and implementing a scalable service to make data storage, management, and computations easier for practicing data scientists both on-prem and on the cloud.

We are primarily seeking:

- Frontend / fullstack engineers (UI / UX with Vue.js, Go, K8s)

* TileDB GitHub organization: https://github.com/TileDB-Inc

* TileDB has been featured on HN: https://news.ycombinator.com/item?id=15547749.

* TileDB recent blog post: http://bit.ly/2SjkeYR

* TileDB talk at PyData 2018: http://bit.ly/2BfbyJ3

Our headquarters are located in Cambridge, MA. The candidates must be US citizens or permanent residents located in the US, or Greek citizens to be located at our subsidiary in Athens, Greece.

Apply today at https://tiledb.workable.com !

Even after such infrastructure is built, a > 0 multiplier assumes the economic impact of the project would be greater than the lifetime costs of upkeep and maintenance of the infrastructure (and current and future taxes to support such infrastructure + interest on debt payments used in its initial construction). And excludes the opportunity cost of allocating that future capital to other projects. Ex. NY still throws money into the canal project, even though the canal has long since been economically unviable (but politically it is still viable and so it remains...).

TileDB, Inc. | Software Engineers | Cambridge, MA or REMOTE (US) | https://tiledb.io

TileDB, Inc. is looking for new team members to help develop the future of big data storage. For emerging and growing fields such as genomics, earth science, imaging, and financial analysis, the volume of data is increasing and TileDB is meeting the challenge head on. Data in these areas is well represented as multidimensional dense / sparse arrays, and the open-source TileDB data management software is a novel, cutting edge solution for array storage.

TileDB, Inc. is backed by Nexus Venture Partners and Intel Capital.

Our team combines software engineering expertise with scientific domain knowledge to develop the open-source TileDB array data management software, and we are looking for talented and motivated engineers for the following positions:

- Full Stack Web Developer: Production experience with Kubernetes, payment gateways, and React or Vue.js for frontend work.

- JVM / Spark Engineer: Deep experience with Spark, Scala, Java, native JVM extensions / JNI, Maven packaging and the SBT build system.

- Bioinformatics Software Engineer: Strong C++ skills, experience with R or Python. Genomics data engineering / analysis with FastQ, VCF, BAM formats, bcftools, htslib, GATK, Hail.

Additionally interested candidates with strong C++ skills and experience developing database / data management solutions are welcome to apply.

TileDB GitHub organization: https://github.com/TileDB-Inc

TileDB has been featured on HN: https://news.ycombinator.com/item?id=15547749.

Our headquarters are located in Cambridge, MA. To cope efficiently with the different time zones and hiring processes, priority will be given to candidates that are located in the US, and are US citizens or permanent residents.

Apply at https://tiledb.workable.com or contact us directly: careers@tiledb.io

Subsidized rates are ~5-7%. I know some of my sisters private loans are higher than 7% (I think somewhere around 10%). My parents gave her absolutely terrible advice about taking on the loans that she did, the millennial generation needs to be told the truth about the real life implications in servicing that much debt.

https://studentaid.ed.gov/sa/types/loans/interest-rates

Not necessarily, if you don't want to take the time to submit a full CV, feel free to send us a message outlining your background / experience / interests.

The company is HQ'd in Cambridge, but (all) the rest of the team is remote.

Jake from TileDB, Inc. Performance wise I would look at the referenced paper in this thread which provides benchmarks for various workloads. As to what advantages TileDB may offer you that is problem dependent, esp. compared to dense simulation output data which is the use case HDF5 was designed for. If you have specific suggestions for ways to improve HDF5 for your use case we would love to hear about them.

Absolutely! Drop us a line at hello@tiledb.io and tell us a little more about the problem you are trying to solve and we can go from there.

Haha, fully agree! It's good to separate MUMPS the language from MUMPS the storage layer / database which was innovative and pioneering in many ways. I don't think we need to re-hash MUMPS the language :)

Jake from TileDB, Inc. Engineering as a discipline is conservative (for good reason). The tools and the processes they use change slowly. Matlab is still entirely relevant both in the sciences and in industry. There are people with huge amounts of domain knowledge (who may only know Matlab or Excel) that are increasingly called upon to analyze and interpret larger amounts of data. These "old farts" are the people engineering, designing, and debugging our modern world. Empowering people with domain knowledge to answer data driven questions is what the democratization of data science is all about. There is tremendous value in building bridges across communities and across generations here.

I say this as someone who helped in small ways to develop open source alternatives to Matlab.

Jake from TileDB, Inc. In addition to the differences pointed out in the paper, I think SciDB and TileDB are very different philosophically. SciDB is architected very much like a traditional RDMS while TileDB is much more lightweight. SciDB encourages you to use their own query language (AQL), TileDB wants to integrate and extend the high level tools you already use (Python, R, etc.) with as little overhead as possible.