Smith42

The Multimodal Universe (MMU) pools together 80TB+ of data from over 30 astronomical surveys into one place. Crossmatching (linking observations of the same object across surveys) is its killer feature, but until now it required downloading hefty chunks of data to local disk. We got tired of needing a cluster just to run a crossmatch, so we gathered in the UniverseTBD and Hugging Science Discord servers to fix that. We've converted the MMU to the parquet-based HATS format so that you can use the LSDB and Hugging Face ecosystems to crossmatch from a laptop. The datasets are here https://huggingface.co/collections/UniverseTBD/multimodal-un.... No bulk downloads are necessary, and 4GB of RAM is enough even at Gaia scale.

It's always been this way ever since the first industrial revolution.

So write it! Shouldn't be much extra to add to the AGPL licence?

We investigate the potential constraints on LLM scaling posed by the availability of public human-generated text data. We forecast the growing demand for training data based on current trends and estimate the total stock of public human text data. Our findings indicate that if current LLM development trends continue, models will be trained on datasets roughly equal in size to the available stock of public human text data between 2026 and 2032, or slightly earlier if models are overtrained. We explore how progress in language modeling can continue when human-generated text datasets cannot be scaled any further. We argue that synthetic data generation, transfer learning from data-rich domains, and data efficiency improvements might support further progress.

That really isn't the case, and I am not sure how you could arrive at that unsubstantiated conclusion.

Abstract:

This work presents AstroPT, an autoregressive pretrained transformer developed with astronomical use-cases in mind. The AstroPT models presented here have been pretrained on 8.6 million 512 × 512 pixel grz-band galaxy postage stamp observations from the DESI Legacy Survey DR8. We train a selection of foundation models of increasing size from 1 million to 2.1 billion parameters, and find that AstroPT follows a similar saturating log-log scaling law to textual models. We also find that the models' performances on downstream tasks as measured by linear probing improves with model size up to the model parameter saturation point. We believe that collaborative community development paves the best route towards realising an open source `Large Observation Model' -- a model trained on data taken from the observational sciences at the scale seen in natural language processing. To this end, we release the source code, weights, and dataset for AstroPT under the MIT license, and invite potential collaborators to join us in collectively building and researching these models.

$15k!

"Large Observation Model" has a nice ring to it

If you are interested in this also check out EarthPT, which is also a time series decoding transformer (and has the code and weights released under the MIT licence): https://arxiv.org/abs/2309.07207

What's new with SXMO? Haven't been keeping up since 2021. Is there a stable phone to run this on now?

Wanted to share the code release of EarthPT, a model that predicts future satellite observations in a zero shot setting! I'm the first author so please shoot any questions you have at me.

EarthPT is a 700 million parameter decoding transformer foundation model trained in an autoregressive self-supervised manner and developed specifically with EO use-cases in mind. EarthPT can accurately predict future satellite observations across the 400-2300 nm range well into the future (we found six months!).

The embeddings learnt by EarthPT hold semantically meaningful information and could be exploited for downstream tasks such as highly granular, dynamic land use classification.

The coolest takeaway for me is that EO data provides us with -- in theory -- quadrillions of training tokens. Therefore, if we assume that EarthPT follows neural scaling laws akin to those derived for Large Language Models (LLMs), there is currently no data-imposed limit to scaling EarthPT and other similar ‘Large Observation Models.’(!)

Code: https://github.com/aspiaspace/EarthPT

Paper: https://arxiv.org/abs/2309.07207

Wishful thinkin buddy

Anyone have a paste of the article? There is a paywall

Check out EarthPT! https://arxiv.org/abs/2309.07207

This isn't the first foundation model for time series, see EarthPT from last month: https://arxiv.org/abs/2309.07207

Yep we are running out of text data, see https://doi.org/10.1098/rsos.221454

Please give it a read! It begins from first principles (Rosenblatts perceptron!) and builds from there so you might find it more general than you expect.

I wanted it to reach a more general audience, as the review is very general in itself (but maybe the original title does not reflect this as I thought)! There are alternating sections concentrating on the astronomy and the deep learning sides.

Author here! We explore the past, present, and future of deep learning in astronomy. We predict that GPT-like foundation models will make a huge impact on the field, and that astronomy is ideally placed to supercharge open source large language modelling (Section 9).

My favourite excerpt, where we propose foundation model-powered scientists:

Autonomous agents are no longer science fiction; task-driven autonomous agents powered by the simulacra of a foundation model are capable of solving very general tasks when given only a high-level prompt by a human operator [305,306]. One could therefore imagine a semi-automated research pipeline, where an autonomous agent with astronomical knowledge is given access to a set of astronomical data through an API. The agent would be prompted with a high-level research goal (such as ‘find something interesting and surprising within this dataset’), and would then take steps to achieve this task. These steps could include querying research papers for a literature review, searching a large multi-modal astronomical dataset to find data that supports a theory, evoking and discussing its findings with additional simulacra, or spinning up simulations to test a hypothesis [307]. While the agent operates in the background, the human researcher would be able to provide high-level interpretation of the results, and would be a steady hand providing guidance and refinement of a more general research direction. In this way, an astronomical foundation model would provide the tools to make all astronomers the principal investigator of their own powerful ‘AI lab’

I wrote a literature review on applying neural networks to astronomical problems -- I found that using applications really helped to iron out what is going on in the networks! Here's the link to the review https://arxiv.org/abs/2211.03796

Since this is pytorch it should run on cpu anyway. What am I missing?

Can I ask how old are you? I like your way of thinking

In recent years, deep learning has infiltrated every field it has touched, reducing the need for specialist knowledge and automating the process of knowledge discovery from data. This review argues that astronomy is no different, and that we are currently in the midst of a deep learning revolution that is transforming the way we do astronomy. We trace the history of astronomical connectionism from the early days of multilayer perceptrons, through the second wave of convolutional and recurrent neural networks, to the current third wave of self-supervised and unsupervised deep learning. We then predict that we will soon enter a fourth wave of astronomical connectionism, in which finetuned versions of an all-encompassing 'foundation' model will replace expertly crafted deep learning models. We argue that such a model can only be brought about through a symbiotic relationship between astronomy and connectionism, whereby astronomy provides high quality multimodal data to train the foundation model, and in turn the foundation model is used to advance astronomical research.

You can also generate similar scale astronomical images with GANs!

I worked on a model that could do this last year:

Paper: https://arxiv.org/abs/1904.10286

7.6 gigapixel image: https://star.herts.ac.uk/~jgeach/gdf