Major shake-up coming for Fermilab, the troubled U.S. particle physics center

https://www.science.org/content/article/major-shake-coming-fermilab-troubled-u-s-particle-physics-center

by dyslexit • 3 years ago

247 144 3 years ago

h2odragon• 3 years ago

One of my great joys as a teenager exploring electronics was the opportunity to clean out an old storage closet at some Fermilab facility with a friend. This was in the late 80s, we got several 1950s oscilloscopes and much other gear of similar era.

The room had been overlooked in the usual surplus process for years. My friend was related to someone working for the janitorial services company, and they'd been told to clean that room out and throw the stuff away. So what we got to do was help with that, and put anything we liked in my car instead of the dumpster.

We were stripping stuff there in the parking lot to save space. Crammed that car full of junk. It was a truly wonderful day.

DeathArrow2• 3 years ago

One man's junk is another man's treasure.

ghaff3• 3 years ago

I have a bunch of stuff in my house that I know there are people who would be happy to have it and maybe pay a few dollars but I live in a fairly small town and I'm certainly not about to go to the trouble of boxing things up and shipping them. I can get rid of some things by just leaving them at the end of the driveway with a free sign but doesn't work for everything.

rektide• 3 years ago

So the adage goes but sometimes one man's junk just becomes another person's hoarding.

jeffbee• 3 years ago

As a teenager I got to work and study at Fermi for two weeks under a DoE summer program. There was one kid from each US state and territory, and a few from other countries. We worked in the shops assembling the D0 and Leon Lederman tried to teach cosmology, which was both an incredible privilege and ultimately futile. I was eating in the cafeteria the day the Texas Supercollider was canceled and I don’t think there’s ever been a sadder crowd of physicists anywhere.

I wonder if they still do that summer program.

jjtheblunt• 3 years ago

Fermilab was built from late 1967 and opened in 1969 so that must have been really old stuff.

Similar to your story, I got to take high school physics extra classes there, and it was awesome, like being in a Star Wars set with entirely normal parents who worked there and who could teach us really interesting physics (classical mostly, and relativity).

mycall• 3 years ago

I tried taking apart one of those old oscilloscopes. After the first screw, I heard a few nuts and parts drop behind it. I knew it was never going to be the same again. Complicated instruments.

ithinkso• 3 years ago

This is tangential at best but I was in Chicago,US once for a business trip and we had weekends off so we did some sight-seeing and one Sunday late afternoon we were close to Fermilab so we decided to take a look.

The building was empty but open, there's a museum upstairs but when we got lost between floors we would just walk between cubicles that were clearly in use during the week, it was super wierd and cool at the same time. We didn't see the accelerator of course but still, we saw some control rooms etc..., to this day I'm not sure if we broke in or what

toomuchtodo• 3 years ago

I’m not sure if it’s still there, but if you walked into the cafeteria the remote ops center for the CMS detector and the LHC was on your left hand side through a glass wall and door.

My leadership chain was…interesting but it was incredibly cool in my mid 20s to lunch with high energy physicists and others in this space.

https://cms.fnal.gov/remote-operations-center/

(worked on CMS data taking for a year in the mid 00s)

Kon-Peki• 3 years ago

However, Fermilab may not deserve all the blame, says a theoretical physicist who requested anonymity to protect relations with DOE. For example, he says, after the lab finally hammered out an excavation contract with Thyssen Mining, months passed before DOE approved it. “I’m not sure whether it’s really the lab that has a problem, or if it’s DOE that has a problem and is blaming the lab.”

If you need anonymity to talk to a reporter, you already know that DOE has a problem.

chollida1• 3 years ago

If you need anonymity to talk to a reporter, you already know that DOE has a problem.

I know nothing of the issues going on but this just isn't a universal truth. Most people would speak anonymously when talking about their employer and that doesn't mean there is anything wrong with their employer.

Which employers allow their employees to speak poorly of them in public without permission to speak about the company?

ghaff3• 3 years ago

In general, companies are pretty sensitive about employees talking to reporters. And certainly employees who aren't media trained and talking about things that aren't explicitly public. Depends to some degree on the employee and the topic, but generally talking to media outside of official channels can easily get someone in trouble even at relatively open companies.

Nullabillity• 3 years ago

Translation: poor leadership is endemic.

Spooky23• 3 years ago

No civil service government employee should openly talk to the media, ever.

bagels• 3 years ago

Government should operate in secret?

frumiousirc2• 3 years ago

US nat. lab. employees are not gov't employees.

prpl2• 3 years ago

Fermilab has been in trouble for a while. They did not diversify enough at the tail end of Tevatron and (I think) are mostly working on DUNE, which isn’t going great. You can’t have a lab that big with mostly a singular focus which is being executed poorly.

ccooffee2• 3 years ago

Your comment reminded me of a Youtube documentary about "the missing American particle collider"[0]. The documentary touches on some Fermilab drama, but most of the content is aimed at politics (and in particular, how presidential politics is unfortunately intertwined with funding for science mega-projects in the US).

[0] https://www.youtube.com/watch?v=ivVzGpznw1U&list=PLAB-wWbHL7...

lumost2• 3 years ago

This may be an interesting example of a “zombie” organization. Fermilab exists to push the boundaries of particle physics.

We stopped wanting to do that, or at the very least outlay the capital to do that. We likewise don’t want people at fermilab doing other things with their time. Meaning that they don’t have much to do but go through the motions.

We have a few of these in science, it would arguably be better in the long run to refocus our science budget on problems we’re interested in solving and some “risk” budget for different ideas.

dmix• 3 years ago

DUNE wiki https://en.wikipedia.org/wiki/Deep_Underground_Neutrino_Expe...

The video of the project is pretty cool https://youtu.be/nv13DswIKr8

Shooting neutrinos through the earth 1300km in 4ms

pfdietz3• 3 years ago

I guess the Holometer doesn't count as diversification. >.>

scheme271• 3 years ago

They're also the CMS tier 1 site for US and possibly north america

ccooffee2• 3 years ago

I found this webpage[0] (and those it links to) to be a good bit of context on what this all means. (And then ran off to do a bunch more searching!)

tl;dr: CMS (Compact Muon Solenoid) is one of two major detectors of the Large Hadron collider. Fermilab was intimately involved in designing CMS. Fermilab is also a "tier 1" computation center for processing LHC data (possibly restricted to CMS data?). Per this factsheet[1], there is one other "tier 1" compute farm in the US, though it isn't named in particular. I'd guess Lawrence Livermore, but I have nothing more than a hunch.

[0] https://cms.fnal.gov/

[1] https://www.fnal.gov/gridfest/pdfs/uscomputing_factsheet.pdf

Caligatio• 3 years ago

This makes me sad.

I grew up in the Chicago suburbs and got accepted into the Saturday Morning Physics Program (https://saturdaymorningphysics.fnal.gov/) at Fermilab. Many of the lectures went completely over my head at the time but I was still in awe of the particle accelerator on campus. I actually got lost driving to a different building one Saturday and accidentally drove around the ring (with several people blindly following me). Their neutrino experiment, which was planned to shoot neutrinos to Minnesota (MINOS program), was on the imminent horizon and the staff were excited... apparently that was ended in 2016.

I guess it's a hard life being known for a particle accelerator when you get dethroned by CERN and then shut down your ring.

1letterunixname4• 3 years ago

Some categories shouldn't be privatized like medicine, military arms, basic science, road maintenance, and emergency rescue operations. Elon isn't going to work on particle physics without a direct commercial purpose.

luckylion4• 3 years ago

"Road maintenance" shouldn't be done by private companies that are given contracts by the government? Why not?

photochemsyn2• 3 years ago

It really depends on where responsibility begins and ends. It might make a lot more sense, financially, for the government to manage maintenance using supplies and equipments purchased from private companies, with the people working on the roads being government employees instead of employees of the contractors. However, for specific jobs - say building a new bridge, etc. - it might make sense to have that be a complete private contract, and then, after it's built, maintenance becomes the government responsibility.

In general however, there's too much opportunity for corruption if contracts are just blindly given to private interests by politicians. Iraq and Afghanistan reconstruction contracts were a good example of how bad that can get.

1letterunixname4• 3 years ago

"Please insert your FriendlyCard to unlock the street in front of your house."

And toll turnpikes.

"Public-Private Partnerships" are neoliberal fantasies of privatized, for-profit monopolies performing government functions while taxing working people. Most often, there is some sort of corrupt and unethical favoritism and revolving door between government official(s) and the vendor corporation.

kingstoned• 3 years ago

Private does not mean for-profit. Funding and performance are two different things. Road construction is funded by government, but performed by private companies.

rqtwteye• 3 years ago

Typically huge profits are made when government contracts are managed by private companies. You see this in defense, space (SpaceX broke this) and big construction projects.

edgyquant• 3 years ago

Who decides what “basic science” is?

myself2482• 3 years ago

For the longest time, scientific research performed two significant functions beyond its stated mission of discovering things:

1: It kept a workforce advancing, both at the labs themselves, and at all the suppliers and manufacturers who fed into them. Those mega-budgets weren't buying mega-yachts for the lab directors; they flowed into mega-upgrades for industry, who could score "juicy government lab contracts" and advance their capabilities, which then paid dividends for the rest of the sector.

2: It supplied national prestige. The best and brightest from all over the world would fall over themselves to work in labs here. The moon landing shattered records for the most-watched TV broadcast ever, and a billion people knew America was on top of the world.

Both of these had significant trickle-down effects, which I don't think were fully appreciated until we let them rot on the vine.

I think "science" in the sense that GP uses it, should refer to any research that ticks both of the above boxes.

alangibson3• 3 years ago

The real problem is that theoretical particle physics had hit a local optimum that they can't break out of. String theory hasn't panned out, quantum gravity hasn't panned out, and no one really knows where to go next.

pa7x13• 3 years ago

This is a complete misunderstanding of the state of physics. The problem is not theoretical physics, if anything, the problem lies in our technological/engineering prowess that has not been able to follow our advances in physics.

The Standard Model has some issues but it's, in principle, a valid effective description of all the fundamental forces minus gravity. The issue is that quantum effects of gravity are negligible until the Planck scale, we have no technological means to get even close to those levels. That is, physicists have been able to provide an understanding of the universe that for all we know for certain might be valid up to the Planck scale, which is where we know for sure new physics must appear. And because our technological ability is lagging much farther behind they must probe that new physics blindly, without any experimental evidence. This is an incredibly difficult thing to do.

Imagine that the ancient greeks managed to discover Quantum Mechanics. They lay down the correct Schrodinger equation, and understand the superposition principle, Heisenberg uncertainty principle, etc. They know it can potentially explain the atom but lacking sufficient engineering prowess they cannot really test the theory and verify its validity. This is the situation physics is in, we have mathematically consistent theories of quantum gravity but we cannot know if it's the correct description of the universe because experimental evidence is inaccessible and might be inaccessible for centuries to come.

alangibson3• 3 years ago

The problem is not theoretical physics, if anything, the problem lies in our technological/engineering prowess that has not been able to follow our advances in physics.

That used to be my view, but it turns I was wrong. The standard model was complete in the 1970's, and everything beyond it since then has been fruitless. Axions excluded, the other kind of axion excluded, sterile neutrinos excluded to make just a few.

Even Roger Penrose thinks we're on the wrong track with quantum gravity.

Check out Sabine Hofstetter's voluminous output on this issue for a better breakdown than I can give.

paganel• 3 years ago

follow our advances in physics.

I'd say physics without the related "technological/engineering prowess" (i.e. the experiments to back said physics up) is just wishy-washy fancy-sounding maths.

blululu• 3 years ago

Sure blame the experimentalists. If the theorists are so smart maybe they can use their immense brilliance to draw on their chalkboards with femtoscale precision. All joking aside, while you are correct that we don't really have the technological ability to advance on a better theory of Quantum Gravity. But theorists need a rich experimental backdrop for their work to be meaningful. Since this is lacking, theoretical physics has produced a glut of plausible models that all arrive at similar answers from very different first principles. This leads to the sense that none of them are decisively right or wrong and the pursuit of yet more models is a meaningless project. Yes the root problem is with the experimentalists, but the theorists suffer the consequences.

sampo2• 3 years ago

The Standard Model has some issues but it's, in principle, a valid effective description of all the fundamental forces minus gravity.

Yes. Theoretical particle physics has done a wonderfully good job in explaining everything it can. But now the job is done. Should move on and do something else. But because the institutions are too established, and past glory is too good, they don't know how.

makerdiety• 3 years ago

Beautifully put together conception of the current state of technology and science.

dt23• 3 years ago

Back in 2015 when I still thought I'd be a physicist, I was a summer intern at Fermilab on the MicroBooNE experiment. The sense of excitement and teamwork on the 10th floor was something I dearly missed when I went to CERN the year after.

Every day at 3pm was "coffee and cookies", and my colleagues and I would join the line filled with physicists of all ages and from across the lab, to grab a cookie or maybe two. On Fridays the coffee turned into wine and the cookies into cheese.

The second floor, where the coffee was, had a rotating art installation, which at some point included an acrylic box filled with water. One day this box went from being empty to containing a live goldfish.

I hope Fermilab retains this sense of magic that I have found in so few other places.

peteradio2• 3 years ago

Yea well cern had beer on tap ... For lunch!

ntsipina• 3 years ago

It still does!

Source: Currently at CERN.

brnt• 3 years ago

"beer"...

/ducks

nukeman• 3 years ago

Perks of a stand-alone national lab I guess! My workplace (Savannah River Site) has a strict no-booze policy. Although people frequently bring donuts!

pknomad• 3 years ago

That's awesome! I work over at ESnet (located w/ LBNL) and got to interact with some folks over at SRS. Y'all are cool bunch :)

unethical_ban3• 3 years ago

When I last worked in person in IT, a few members of my team would get up at 2pm to get coffee and chat about work and life. Different people went in the mornings. It was really useful to team-build and feel decent about work.

People who say "I don't have friends at work" with pride are completely foreign to me. I don't understand how people are supposed to have any joy at work, or get over comms barriers, when there is no opportunity to get to know coworkers as people.

dylan604• 3 years ago

I can have lunch and water cooler chat friends at work, but rarely do they turn into invite over to bbq on the weekend friends. There's a big difference.

You can be friendly, but when something happens at work and one of you no longer works there...things get weird. Those weekend bbqs that you had with work friends no longer happen, or they keep going for a bit, but conversations are awkward since the still employed are no longer free to discuss or the no-longer employed harbors resentment that isn't exactly hidden. What kind of "friend" was that after all?

peterangular2• 3 years ago

Just responding to some of your points - sorry if this comes off as abrasive.

a few members of my team would get up at 2pm to get coffee and chat about work and life

If I stopped programming arbitrarily at 2:00pm when my hours are 9:00am-5:00pm I expect to be let go (unless I'm taking this as my lunch period). Especially to socialize over coffee. As a production-level IC I just can't do that ethically, and perhaps this is a personal fault... but I'm paid to keep my nose down and develop software solutions, not to socialize with colleagues. If we need to meet over work it's best to get that on the calendar so there's some expectation of formality - not in some adhoc social hour/afternoon.

People who say "I don't have friends at work" with pride are completely foreign to me.

I don't say it with "pride," but I have 100% learned to put distance between my personal and professional lives. Example: every time I have had to deal with suicide scares it's always been a professional relationship/colleague of sorts. It's been three times now, and for some reason I seem like a trustable enough person that people put this on me. I hate it.

Then, there's people who I have thought were close friends who completely ghosted me after moving to a different company - who knows, maybe it's me. But I've heard lots of people express similar frustrations and it's just pushed me to consider every work friendship a "work friendship." I have more equitable and consistent relationships when I make friends outside of my profession, I invest my time and energy there.

Finally there's the emotional investment. When it comes to employment I am not part of a "family," and I'd really appreciate that sort of language to stop. Right now. I am selling my time and labor to a business to solve problems and fulfill operational duties that they require of me. At any point, employer or employee can terminate that relationship because "at will." This is surely not a "family," and I personally believe it is not an ideal spot to sink your social energy. I've seen people have their careers pulled from underneath them after over-investing socially into the company - they do not do well. I'm not here to emotionally invest during my 9-5, I'm here for business. Unless it's a requirement of the job (sales, vendors, talent aq, etc.) I am not putting myself out there socially/emotionally.

---

Sorry to share some anecdotes but I think there are very valid reasons to put distance between your work relationships and your personal ones, purposefully erecting a buffer. I think it's very valid if someone wants to keep their work relationships just that: work relationships.

Pokepokalypse2• 3 years ago

I've lost too many friends to mass-layoffs over the years. I just can't do that shit anymore.

DangitBobby• 3 years ago

Ha, based on this comment you could be one of my classmates! Did you do undergrad in TN?

samstave• 3 years ago

Every time stories appear about the Particle Physics Center, Fermi Lab - they seem to show up in waves.

an1sotropy• 3 years ago

well that's the article-wave duality in action.

pfdietz3• 3 years ago

OW OW OW

photochemsyn2• 3 years ago

DOE should really be spending more money on condensed matter physics than on high-energy physics, at least the former has more practical applications and research can be done with smaller budgets, and the latter has plateaued in many ways.

Neutrinos are pretty fascinating, it's true, but the price tag's pretty high. DOE could instead be financing solar PV research, battery research, etc... although then the politicians would likely cut their budget under pressure from the investor-owned utilities and fossil fuel exporters.

rsfern• 3 years ago

DOE already funds a ton of research programs in battery tech, PV, and other renewables. I mean it would be great if they could fund even more, but you make it seem like some kind of politically untenable topic when it’s not

analognoise• 3 years ago

They should give it to Northrop Grumman instead - if it's 12 years late and 10x over budget, everyone will get awards, like what happened on JWST.

yummypaint• 3 years ago

My Fermilab story: when I was designing my dissertation experiment it became clear that i would need ~$100k in silicon strip charged particle detectors, and that no company would ever microbond the thousands of connections for me due to the low volume. Fermilab gave me detectors that had been QC rejects from the outer barrel of the CMS detector at CERN for free, and bonded them to my boards for essentially materials cost. For my purposes they worked perfectly. The microbonding machines and the wonderful people associated with them are still to the best of my knowledge the only viable place in the US to have 9x9 cm silicon autobonded in small volume.

killjoywashere2• 3 years ago

Reminds me of this story:

https://www.latimes.com/local/education/la-me-caltech-glassb...

There are continuous technical workforce challenges in science because the methods keep advancing. If only there were money it. Instead we have engineers, scientists, and physicians driving Uber and we outsource and offshore the entire scientific supply chain.

clivefx• 3 years ago

I don't know if that is true. Stuff just changes. To solve similar problems to this, my lab developed sapphire metal bonding techniques that could be run with CNC machined parts. Everything became nice and reliable and repeatable.

We lose techniques all the time but gain new ones. I designed a scientific instrument once that had a long precision bore through it. I designed it with the belief that I needed clearances for a large stiff boring bar to make the feature.

I showed the prints to my machinist, and he told me they were just gonna put it on the five axis and hit the bore on both sides with an end mill. That it was as accurate as how the old guys did it.

It was.

veqq• 3 years ago

we have engineers, scientists, and physicians driving Uber

Why do you believe this?

queuebert• 3 years ago

Not driving Uber, but working for big tech and finance making money without advancing civilization.

dugmartin• 3 years ago

One of my favorite memories from thirty years ago in college was having small group breakfast with a few other students through the honor's program with Leon Lederman, the Nobel winning physicist that was the then Director of Fermilab. He had a lot of great stories about how big government funded research worked and was very charismatic.

I toured Fermilab later in college while we where near there for an ACM programming competition. The one thing I remember from the tour was a big red button labeled "Start" next to some huge experiment. The tour guide saw us looking at it and said something like "You don't want to touch that". I've often wondered if that was real or just a prop they used for tours.

Btw, I'll never forget what Leon said when asked by a physics student about what the most important thing to learn or study as an undergrad. He said, "learn to write".

RedCondor• 3 years ago

Theoretical physicist and Nobel Laureate Steven Weinberg had this to say on [the SSC's] cancellation:
>Spending for the Superconducting Super Collider had become a target for a new class of congressmen elected in 1992. They were eager to show that they could cut what they saw as Texas pork, and they didn’t feel that much was at stake. The cold war was over, and discoveries at the SSC were not going to produce anything of immediate practical importance.

https://redsails.org/concessions/#education-and-research-its...

Animats• 3 years ago

They're in trouble because they didn't have a few people who knew how to manage excavation contracting.

pasttense01• 3 years ago

I think it is a poor idea for the Department of Energy to being hiring businesses or organizations to be managing the National Laboratories. This means there are three levels of management: the DOE, the managing organization/business (such as the University of Chicago) and the lab itself. Eliminate this middle layer--so that management is either done by the DOE or the laboratory.

frumiousirc2• 3 years ago

I expect many scientists would quit working for a lab if they had to become direct gov't employees and those who are non-US citizens would be even more excluded than they are now. Both results would degrade the quality of the scientific output of the labs and not necessarily save any gov't expenditure. I also suspect the contractor layer gives some continuity and smoothing of the rocky year-to-year games played by executive and congress critters.

bmitc3• 3 years ago

A big issue at these labs is that there's this idea that people with PhDs think only others with technical PhDs can manage things. Management takes a certain set of skills, and quite a lot of PhDs have zero experience outside of academic environments with managing things.

“We did not write a very good contract for the excavation,” ... “There were all kinds of loopholes in it, and the excavation company made an awful lot of money off of us.”

Another thing that strikes me is, why is construction in the U.S. so corrupt and such a money pit? I don't think I have ever heard of a large construction project being delivered on time and on budget. I understand construction is very difficult, but it just seems out of control.

dreamcompiler2• 3 years ago

https://youtube.com/watch?v=dOe_6vuaR_s

bmitc3• 3 years ago

Thanks for the video. I'll watch in full later. But it is interesting that he opens with The Big Dig in Boston. There have been several counts, including prosecutions, of fraud in that project. It wasn't just because it was complex. There was rampant fraud and corruption. After some skimming, it didn't appear that that video covers that.

I have also read anecdotes about that project. For example, stories of contractors tricking out their personal trucks and tools and booking it to the protect.

josh2600• 3 years ago

Threads like these make me really happy to be a Hacker News reader. This is a slice of life that is very far afield from my own but one where I appreciate the attention to detail and correctness that holds the whole damn thing together.

I guess what I'm trying to say is that the hacker mentality extends far beyond software. The hacker mentality is a way of life.

vuciv1• 3 years ago

It’s worth noting that these large-scale projects, like the Long Baseline Neutrino Facility and Deep Underground Neutrino Experiment, are incredibly complex and have no real historical precedent. The sheer scale and ambition of these endeavors make it nearly impossible to accurately predict timelines and budgets.

It's easy to criticize Fermilab for cost overruns and delays, but we must remember that they are pushing the boundaries of scientific knowledge and technology. The nature of cutting-edge research means that there will always be unknowns and challenges that cannot be anticipated.

gammarator• 3 years ago

The "iron law of megaprojects": Over budget, over time, over and over again.

https://arxiv.org/abs/1409.0003

iampivot• 3 years ago

That is one very depressing looking building! As if it was built in North Korea!

blueline• 3 years ago

Wilson Hall is the total opposite of depressing for me. Especially the interior is quite beautiful:

https://3.bp.blogspot.com/-UOpsfRLRr_I/VbhWuR4OeBI/AAAAAAAAB...

https://live.staticflickr.com/3800/33033871090_f658526696_b....

The art and architecture around the lab has a surprisingly rich history dating back to its first director who was also a sculptor/artist: https://history.fnal.gov/historical/art_arch/art_architectur...

Major shake-up coming for Fermilab, the troubled U.S. particle physics center

Related Stories