quadruplebond

What do you mean bare metal? Because all of the big DOE computers are running Linux. Users will probably use hardware specific libraries (like CUDA/ROCm) and occasionally write some hardware specific asm, but none of the big computers are running without a POSIX OS.

They got back to me after about 5.5 weeks (it was a no, but nicely worded). I think they are just really busy.

If dod research labs are like NNSA research labs part of why 0.4 FTEs can be considered significant investment is because overheads are quite high. My experience is that for every dollar a FTE makes their project is paying an additional 2-3 dollars in overhead. That means your $130K/yr FTE is using 50% of your $1M/yr budget.

I'm not sure what the typical cost for an FTE are in industry, but I believe they are lower than 3-4x base salary.

Not all positions require a clearance, but most citizens eventually get one. Life as a foreign national can be difficult if you are at SNL. And if you are from a sensitive country it is even more difficult.

send me an email at <my-user-name>@gmail.com if you want to chat more,

I am in a software role, focused on future architectures and HPC, WLB is good in our department.

Job security is really good IFF you are an FTE, if you are a postdoc or an LTE then meh.

Changing departments requires some work, but at least on the open science side it's easy to switch projects. You keep the same manager, but just work on different stuff. If you are in a super specific mission area it might be harder to change work.

If you are interested in HPC at Sandia just email me and lets chat more, HPC is where I currently sit and know the most people.

Happy to answer specific questions if you have them.

It is going to depend on your org and some other stuff. How much do you want to work on the high side etc.

Nvidia was probably under contract for this before mining blew up to where it is today. Provisioning for these machines starts 5-10 years before they ever come online.

Partly because we don’t want to be locked into using Intel for both of DOE’s open science machines. In the lead up to the exascale or near exascale machines (2012-now) I don’t remember hearing much about AMD, so presumably Intel and IBM were the only vendors supplying general purpose CPUs.

Yes, but rumor has it that Intel will still be the supplier for Argonne’s next computer. Which if everything goes to plan will be the first exascale machine.

Sort of, but Blue Gene tech was always an Argonne thing, Oak Ridge has been the one doing GPUs for awhile now. Unless something weird happens Argonne and Oak Ridge will always be running different types of computer since we don’t want to be locked into one specific vendor or hardware configuration.

Most machines will allow for back filling if your jobs are short enough and small enough. But that will still leave some of the machine under utilized.

Unfortunately you would be disappointed. top is only going to display what's running on your current node, not the whole machine. There is probably some sort of global top, but just logging in and running top in your shell will only run top on the head node. That head node might still be beefy, but only as beefy as a large shared memory machine can be.

Think of it as a much higher bandwidth PCIE. You still have to think about data locality, the same way you do with cpu cache levels. It just makes the problem of not having data on the gpu less painful than before.

Depends on what the computer is designed for. Imagine you want to parallelize up to about 16 nodes or 1000-2000 cores of a new xeon. And your code is not optimized for accelerators. Well you might build a large machine with hundreds of nodes, but with the intent to run many 16 node jobs at a time. The largest machines though mostly are aiming for jobs that use a significant fraction of the total machine and are chasing peak performance so accelerators it is. Also I didn't look too closely at the link, but some machines that aren't technically accelerators like K computer or the old IBM Bluegene computers are closer to general purpose accelerators than they are to fat xeon nodes.

Probably something that looks like Red Hat. All these codes will use some form of message passing with MPI being the most common. Finally to get on node parallelism some codes will hand write cuda for their problem and others can get away with calling Nvidia libraries or using something like new versions of OpenMP.

Pretty sure that Phi as you know it is largely being left. Argonne's Aurora exascale machine was suppose to be Xeon Phi, but it has been postponed till 2021 and will likely use some other tech. Also look at the top 500 list from late 2017. Trinity the NNSA computer at #7 is Phi based but only gets about 50% of theoretical peak.

This looks like it might be nice, but regular coupled cluster doesn't really have very many sparse matrices so mapping things into Blas3 is fine. Also if you go the reduced scaling route and use PNOs or OSVs everything is mapped into matrix matrix for CCSD anyways.

Things like this exist but there are two issues. They aren’t part of the standard and Fortran compilers are better at optimizing built in Fortran abstractions than c++ compilers are at optimizing user made abstractions.

I suspect it’s the lack of standardization that’s really the issue.

Because you need to modify or extend it in some nontrivial way.

Yeah small and skinny matrix multiplies are an active area of research.

Sure but that will likely be much much slower.

Eigen isn’t good enough? You can offload some stuff to MKL with it also.

It’s partly the power of library authoring that is moving things I think. To my knowlegdge most post lapack tensor algebra is all C/C++ also. I don’t know Fortran but I am not sure if it has the same flexibility when it comes to generic code and things like writing things like template expression math codes.

Finally the fact that groups like Facebook and google are writing their machine learning code in C++ shows that 1 they find it useful and 2 plenty performant. This kinda became a response to the comment above yours sorry.

There is a lot of electronic structure in c and c++. For SMP there is psi4 and Orca also Garnet Chan has a new python C++ package. Then some widely used integral code generators generate c or c++ (libcints, Libint2)

On the parallel front MPQC has been around a long time and is C++.

I don’t use Fortran but I agree as of today and probably post 90 Fortran is nice enough but people moved to c++ for things like polymorphism and templates and pointers among other things. You could probably also rewrite nwchemex in Modern Fortran just fine. I’ll give you one example of why C++ might be easier though. Next gen linear algebra libraries are being written in c/c++ such as elemental and dplasma also Cyclopes tensor framework and probably others. When the libraries you want to use are in c/c++ the that can be the easiest path forward.

I agree, it is very project/field dependent. There are times when a painful redesign might pay off in the future, but I am sure there are also times when it is a waste of effort. In your case a redesign might have been a waste, you are in the best position to judge this.

Not necessarily. It was definitely bad if they want to become academics and needed to publish papers. But if they knew from the start what they were getting into and wanted to transition from physics to software development then it might not have been so bad (although there might have been a more optimal path).

A bad advisor wouldn't care what his students wanted, but a good advisor might still have students work on this kind of project as long as they were aware it wasn't going to help them get a tenure track position in the future. If they wanted to go work at a national lab doing HPC work and programing it might have been plenty ok for their career (This is what I am transitioning to now) or if they want to go work for a hedge fund or apple it might also be an okay option.

Potentially your "rival groups" will have a long term advantage now though. This might be bad for the graduate students that did the work, but could be good for the professor and group over the next decade.

Graduate students need to think in terms of 4-6 years while forward looking professors might want to think in terms of 5-15 years.

Just a grad student, soon to be postdoc, so not a big wig on ECP.