I trained a 113M-parameter earthquake LLM from absolute scratch

https://github.com/jiazhe868/nanogpt-seis

by jzsfg • 11 days ago

12 5 11 days ago

jzsfg3• 11 days ago

Hi HN,

I am a geophysics researcher and PI making a transition into the ML/AI space. Over the past year, I noticed that while many students and researchers use LLMs, the actual pretraining lifecycle remains a black box to them. Most tutorials start with a clean Hugging Face dataset and end with a trainer.train() call. I wanted to demystify what actually happens in the trenches, so I built nanoGPT-Seis.

This isn't an attempt to build a SOTA foundation model. It is a teaching repository where every single step of the LLM pipeline is implemented and explained block by block, scaled down to run on a single workstation or two A30 GPUs.

I started by writing a concurrent BFS crawler to pull open-access seismology papers via Crossref and Unpaywall. The yak-shaving here was real. To avoid the classic O(N^2) deduplication nightmare, I implemented a MinHash LSH pipeline to cluster near-duplicates before doing exact similarity checks.

Instead of reusing the GPT-2 tokenizer, I wrote a byte-level BPE tokenizer from scratch with a 16k vocabulary, purely to understand how domain-specific compression behaves. For the architecture, I wanted it to mirror modern Llama-style decoders, so the repo includes from-scratch implementations of Grouped-Query Attention (GQA), Rotary Positional Embeddings (RoPE), SwiGLU, and an integration with FlashAttention-2.

One of the most interesting engineering challenges was figuring out the actual scaling behavior. I built a small IsoFLOP scaling-law harness. By implementing Maximal Update Parametrization (muP), I was able to tune the learning rate once on a tiny model and cleanly transfer it across a 90x compute sweep to find the compute-optimal frontier.

A fun empirical finding on data mixing: when I trained the model purely on my scraped research papers, it became highly fluent in "academic register" but generated repetitive gibberish when asked to write plain prose. Injecting a general-text mix (about 76% FineWeb-Edu and Wikipedia) completely restored its plain English fluency while preserving its domain sharpness. It was a very stark demonstration of the specialization versus fluency trade-off.

The repo has all the code, the exact VRAM mathematics for sizing batches before OOMing, and the reasoning behind each architectural choice. I would love to hear your thoughts, answer any questions about the math or the PyTorch engineering, or just chat about the transition from physical sciences to ML.

curiousObject2• 11 days ago

I understand you have created this for teaching - to help other researchers use LLMs better.

But, in its current state of development, is it of any use in earthquake preparedness or immediate response to an earthquake?

jzsfg3• 11 days ago

Not yet in its current state.

Right now, this is purely a pre-trained base model. It has learned the statistical distribution and vocabulary from seismology research papers, but it only knows how to predict the next token. Without Supervised Fine-Tuning (SFT) and alignment (RLHF/DPO), it cannot act as a reliable assistant or follow instructions.

However, proving out this end-to-end pipeline was the necessary first step. Now that the infrastructure is in place, the roadmap is to scale up the parameter count and apply SFT to build something genuinely useful for post-earthquake response.

The vision is that in the critical minutes after a major earthquake, when we only have sparse, rapid telemetry (magnitude, location, affected area), an aligned domain-specific LLM, coupled with a RAG (Retrieval-Augmented Generation) system, could instantly retrieve structurally similar historical events. It could help researchers and responders rapidly synthesize the likely faulting mechanisms, evaluate the localized tsunami potential, and assess the aftershock hazard based on physical constraints.

In disaster response, hallucination is unacceptable. That's exactly why I wanted to explore building a domain-specific model from the ground up, where we can strictly control the data mixture and eventually use RL to penalize it for violating physical laws, rather than just prompting a generic off-the-shelf model. So it's not ready for the emergency room today, but this is the foundation to get there.

curiousObject2• 10 days ago

That rapid disaster response seems a good practical target to start with. Good luck.

I trained a 113M-parameter earthquake LLM from absolute scratch

Related Stories