The governor of NJ went live on TV to request COBOL programmers

Must not be willing to pay enough or be hard to work with. At the right price, 100 people should not be hard to find. Of course, they probably would not be in this position if they were willing to put money into the system or keep people on retainer at least.

dreary_dugong• 6 years ago

I recommend taking a look at the other discussions linked. Unfortunately the situation is more nuanced then "just pay them more."

renewiltord• 6 years ago

I tried reading the other threads but they are not very information-dense. It's mostly people arguing about whether governments are good employers.

Would you mind sharing a quick summary if you've synthesized the position?

sharkweek2• 6 years ago

My dad, when I was in elementary school, started a small commercial janitorial business that he runs to this day, even though he’s mostly retired.

He always told me how it was helpful to have an unsexy but useful skill set, and basically do the work that nobody else wanted to do to ensure a long career.

papeda• 6 years ago

Curious, how does a small commercial janitorial business survive for so long? My naive understanding as a non-business-owner is that small entities can survive or even do well if they offer a product that's hard or not worth it for big entities to replicate at scale. Some examples might be art, or a nice feeling of "buying local" from people you know and like, or a niche product whose market is just not big enough for giants to bother with.

A janitorial business doesn't seem to be any of those things. So it seems like an area where a big company would dominate (because, say, a lot of 1000 janitorial carts is way cheaper than a lot of 10). But, no?

1123581321• 6 years ago

I don’t know why^, but the largest janitorial/cleaning companies that serve ordinary businesses^^ are franchise companies. Local independents just have to compete against branded local companies.

^I suspect it’s because janitors have access so trust of the individuals involved matters more than cost or having access to a pool of interchangeable employees.

^^There are large contractors that serve really large facilities, but that business isn’t necessary to win to be viable.

lumpsti• 6 years ago

I guess like with many job. You can put a little heart into it and care about the customers. This is something that is hard to replicate in scale for big cleaning company. Not many cleaning companies do that. And those who do are loved deeply by their customers.

ridaj• 6 years ago

There aren't very meaningful benefits to scale in that business. It's labor intensive, and no critical central infrastructure is needed. Capital costs (those janitorial carts) and supplies are a lot smaller than labor costs.

sharkweek2• 6 years ago

I honestly think it has everything to do with having started it in the early 90s and just keeping his customers happy for 30ish years. He has several big local commercial clients from his early years that still use him because they know he or one of his staff will always show up.

Also I think the bigger players in the space, ABM-types, aren’t interested in commercial space smaller than XX,000 sqft where my dad has been able to fit in nicely.

amelius• 6 years ago

How hard would it be to automatically convert that COBOL to e.g. C++ or Rust? I suspect not that hard, since COBOL does not even have a heap afaik.

AndrewDucker• 6 years ago

And then you have a bunch of code that's even harder to understand than the original, working in ways you don't expect from the language it's now in.

And not working with the normal build or debug tools on the mainframe it runs on.

bpicolo• 6 years ago

Presumably, hard:

https://aws.amazon.com/blogs/apn/automated-refactoring-of-a-...

downerending• 6 years ago

A high-fidelity translation between almost any two serious programming languages sound like a very difficult task. The first 95% might indeed be quite easy, but the last 0.1% would often turn out to be extremely difficult. And if the software "matters", you have to achieve 100%.

ape4• 6 years ago

Lots of things don't have a 1:1 match. For example Cobol has a PICTURE clause for describing a record. It uses 1 character symbols like in fscanf() but there are different options. Like binary coded decimal.

retrac• 6 years ago

It can be automatically translated. But then all you've done is convert a bunch of IF A < B THEN GO TO STEPTWO into if (a < b) { goto steptwo; }

And with a lot of extra unreadable stuff to support things COBOL does but the target language doesn't.

The biggest problem with COBOL codebases is that COBOL (at least back in the day) doesn't offer much in the way of structured programming or standard libraries. Horrifying spaghetti code with one-off local solutions for common tasks is the norm, especially on some ancient endlessly patched code.

gregjor4• 6 years ago

Not true, COBOL has as much structure as any other language, if the programmers wrote their code that way.

The main problem with a direct conversion is COBOL has database operations in the language, mapping data directly to variables (records). All modern languages you might translate COBOL to use libraries to perform database operations, and may not even have an off-the-shelf implementation of the pre-relational data structures COBOL was originally designed to work with.

eej712• 6 years ago

The ignorance of cobol is not the blocking point.

Ignorance of the details of the NJ system is the great challenge. Its likely a huge legacy system with all of the attendant warts, histories, strange choices, and odd limitations that come with that. Given that these systems are built up through years of teams of people that come and go - it will likely take a long time to address the real issues.

thaumasiotes• 6 years ago

The ignorance of cobol is not the blocking point.
Ignorance of the details of the NJ system is the great challenge.

This seems very plausible. But on this analysis, what's the point of trying to hire "COBOL programmers"? We've just stipulated that the COBOL is easy. Onboarding an experienced COBOL programmer is only a trivial savings over onboarding someone who's never heard of COBOL; they'll both have to learn the system -- but the pool of experienced COBOL programmers is much smaller than the pool of "everyone".

isaachier• 6 years ago

From what I gather, it's an issue of floating point precision. See more here: https://medium.com/@bellmar/is-cobol-holding-you-hostage-wit....

Blackthorn• 6 years ago

Universities stopped training because it was not—glamorous.

It also doesn't pay very well.

Bnshsysjab3• 6 years ago

If it’s anything like legacy public service code in my country it’s not just that, it’s also dealing with terrible organisational culture, d-grade devs, c-grade contractors and corrupt management that let it get that way (as millions are dumped on the contractors, as it turns out).

Some days I wish an ultra virus would take out computers as we know it so we’d be forced to start again with what we know now, namely not creating these monolithic nasty monsters.

chx• 6 years ago

Yes and no remote work as far as I am aware of these things. It just sucks. If it wouldn't suck, I would've went into it: I like retro and COBOL is not a hard language to learn, really, it is not.

grogenaut2• 6 years ago

Cool we'd have a bunch of new monsters based on a super old version of nodejs instead.

moron4hire• 6 years ago

Yeah, did my time working in consulting for state gov bureaus. Some of the worst working environments I've ever been in. Asshole, down-right criminal managers. IT actively pirating software. Computers literally falling apart.

Not sure you could pay me enough to go back. Maybe for something ludicrous, like $3mil/yr and mutual termination at any time. I could probably put up with it for 4 months.

These places that "can't find COBOL programmers" need to stop blaming universities and start taking a harder look at themselves.

csdreamer7• 6 years ago

This. HN and other places love to regurgitate this was the reason people didn't learn cobol.

The real reason is that most companies that don't bother to move away from either physical or emulated cobol systems just do not care.

They certainly do not care enough to pay market wages.

27182818284• 6 years ago

In the YouTube clip, the URL http://covid19.nj.gov/ but there is still as of yet no mention of COBOL as far as I can see.

If anyone has the actual links that include the job description, etc, I'd be curious.

macintux• 6 years ago

I don’t have a direct link for you, but this topic has popped up several times in the past few days and you might find something more substantive.

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

blasket2• 6 years ago

8 seconds in he says "we need Cobalt programmers" and repeats at 27 seconds https://www.youtube.com/watch?v=HSVgHlSTPYQ

blasket2• 6 years ago

https://www.youtube.com/watch?v=HSVgHlSTPYQ

8 seconds in he mentionds "Cobalt" & also at 27s

Waterluvian• 6 years ago

Generally speaking how long would it take to mobilize any competent senior software developer as a COBOL developer?

And could you imagine the DPA being invoked to govern what kind of _software_ output a company produces?

I know I'm being far fetched but I'm thinking about the hypothetical of the government just saying, "Apple, you're doing government payment software now. Here's your contact. Get to work."

gregjor4• 6 years ago

I learned COBOL from the (then) standard text by McCracken back in the early 1980s. It’s not hard, a week or two for someone with programming experience.

The bigger problem is domain expertise (or lack of), and coming up to speed on the standards and idioms used in projects like this. Requirements and documentation likely out of date.

ellius2• 6 years ago

I'm a younger engineer, so I say this with a degree of humility, but is trying to modify these monstrous legacy systems on the fly during a worldwide catastrophe really the most responsible or intelligent solution? Is there no way to authorize some kind of emergency system built with modern tooling to offload the system burden and then process it later through the system of record? If I were trying to deal with this problem, my approach would be the following:

- Organize the legislature(s) to pass some kind of emergency act that gives protection around PCI compliance and that sort of thing to new engineering teams

- Hire business and technical experts, fast

- Build a system on AWS that can handle massive scale and start offloading the most important business functionality to that system (things like registering for aid, fraud prevention)

- when things begin to return to normal, start processing requests through the original system, deal with problems, whatever

I know all of this is not easy. Legal and compliance stuff is tough, and I know the domain itself must not exactly be simple. But this is an emergency, and it calls for emergency measures. Trying to hack your way through an ancient code base on the fly seems like it might be doomed to failure.

gregjor4• 6 years ago

I think you underestimate the scope and scale of systems like this. And maybe overestimate the success rate of rewrites, made worse when the developers have no domain experience and a rushed schedule. Under the best conditions rewrites fail more often than not.

Maintaining legacy code isn’t always easy, but it has a big advantage over a rewrite: it works and satisfies requirements to some non-zero percentage, whereas imaginary code built with “modern tooling” does not work or meet any requirements. COBOL and old business systems aren’t that hard to understand.

chberry• 6 years ago

I completely agree. Rewriting a system on the fly is absolutely going to fail. There are no requirements and without the domain knowledge, there is no way that anything could be completed correctly in any reasonable time. This is a project that should have been scheduled years ago.

Some level of technical debt will exist in every organization. The problem is that NJ never had prioritized this work and now they have an emergency and they are looking for volunteers. They should absolutely should be paying for this work.

ellius2• 6 years ago

I wasn't imagining a full rewrite, more like a "holding area" that could defer some of the load. Again I may be totally wrong about this—I have never written a COBOL system, and if there is some plausible way to make it effective for this problem in a reasonable timeframe, then I understand why that would make sense. I understand all of your reasoning and agree with it in general. I know rewrites are very expensive and often ineffective. I'm just skeptical that the system can be modified quickly to deal with this new load. But again I am not an expert and could be totally wrong, and I'm open to being convinced.

blululu• 6 years ago

>needs 100's of programmers - TODAY

If you haven't done it already, please read Fred Brook's timeless classic 'The Mythic Man-Month' (1975) https://en.wikipedia.org/wiki/The_Mythical_Man-Month

Nine women can't make a baby in one month - classic wisdom that should be understood by anyone trying to organize and plan (like a governor).

The governor of NJ went live on TV to request COBOL programmers

Related Stories