July 29, 2019No Comments

The cost of fixing COBOL bugs

COBOL is entrenched as a vital part of the global economy, but its diminishing trained workforce is causing the cost of maintaining COBOL-based applications to rise precipitously. We estimate that the current cost is roughly $1.02 billion per year. Check our numbers and learn more about why these costs will continue to rise.

I began working with mainframe programming languages in 1990 while I was in college. I started with FORTRAN and assembly languages, and I knew about COBOL but wasn’t exposed to it until much later.

While FORTRAN and assembly languages now have their niche uses, COBOL is entrenched as a vital part of the global economy. And now the cost of maintaining the nearly 60-year-old language is rising precipitously.

A little history

The Common Business Oriented Language (COBOL) was developed as a stop-gap measure during the second Eisenhower administration to create a portable mainframe programming language for the Department of Defense. It was based on the FLOW-MATIC compiler, which was recognized as the first English language data-processor compiler and was designed by Rear Adm. Grace Hopper.

COBOL was adopted by the business world starting in 1960, and because of its simplicity and reliability, COBOL-based applications remain entrenched in mainframe-dependent industries such as government and finance. The Social Security Administration (SSA) and Internal Revenue Service (IRS) rely on approximately 110 million lines of COBOL code combined daily. An estimated $3 trillion a day and 90% of all ATM and in-person financial transactions are handled by COBOL-supported systems.

I predict that the last mainframe will be unplugged on March 15, 1996.
~Stewart Alsop II, InfoWorld magazine, 1991

The bad news is that some of these applications are nearing 50 years in age. And although COBOL remains vital to many critical systems, this prominence has not resulted in a stable supporting workforce. The population of experienced COBOL developers declines at least 5% every year because of retirement. The population’s average age is roughly 55 years old, and the language’s absolute uncoolness has motivated the majority of university computer science programs to drop their COBOL classes entirely. The trickle of newly trained COBOL programmers are coming from community colleges, technical schools, and programs run by private companies such as IBM.

The continuing importance of COBOL-based applications and the diminishing trained workforce is causing the cost of maintaining COBOL-based applications to rise.

How often do defects arise?

You might ask, “Well, if COBOL is so reliable, why do we need lots of programmers? Fewer bugs naturally means fewer engineers.”

That sound perfectly logical, however, in my experience, due to the lack of a sufficient supporting workforce and the spaghettified nature of today’s COBOL-based applications, they are not modified and supported as well as other applications programmed in more popular languages.

Think about all the patches, additions, and technical debt that have built-up since these applications began surfacing in 1960, as well how many developers contributed to those applications but are no longer around to address issues and answer questions.

Due to the uncertainty of downstream impact and to reduce the risk of breaking critical applications, I believe many COBOL engineers duplicate hundreds or thousands of lines-of-code when making changes or repairs because they do not completely comprehend the COBOL code in their system. As a result, changes are made, tested, and implemented in an environment of uncertainty and heightened risk.

And when the flaw is discovered, the repair cost includes:

  • Consultants or outside IT contractors
  • The development team’s time away from building new features and products
  • Salaries of internal personnel tasked with fixing the bug or supporting the project
  • Application downtime
  • Lost business opportunities

The Y2K scare is just one example. While it ended up being much less daunting than many estimates, roughly $320 billion was spent worldwide evaluating and fixing systems.

But the price of fixing COBOL code includes more than just repairs and downtime. An organization’s security, reputation, and market position can be affected by one time-bomb defect. When an August 1, 2012, software failure caused Knight Capital LLC to create thousands of trades per second on the New York Stock Exchange, the company lost between $440-460 million in 45 minutes. By the next day, Knight’s stock had fallen 75% and within the year the company was acquired by a rival.

What is the cost of fixing COBOL code?

So, what is the actual cost of fixing COBOL code?

For the sake of simplicity, let's analyze the cost based on salaries and time spent supporting COBOL-based applications, but not developing new programs.

Here are our assumptions:

If we conservatively estimate the cost of maintaining COBOL-based applications by only including the developers working solely on maintenance, it’s roughly $1.02 billion per year. Here’s how we get there.

20,000 COBOL developers
x 42% solely doing maintenance
x $81,000 per year                           
= $680.4 million per year

In my experience, you have to add another 50% of the development cost for quality assurance (QA) testing, which would bring the total to $1.02 billion just in the U.S.

$680.4 million
+ $340.2 million for QA testing     
= $1.02 billion per year

If anything, these numbers are low because we didn’t consider the companies doing both maintenance and new development in COBOL.

The cost of fixing COBOL v. alternatives

Another approach to handling applications written in legacy programming languages is to modernize the code – translate the COBOL source code to a modern language such as Java.

Despite the astronomical maintenance costs, it doesn’t necessarily make sense to modernize COBOL-based applications. Hundreds of millions of dollars are spent each year on COBOL transformation projects, and more than a few have ultimately failed.

The risks associated with a COBOL transformation project are significant. Dave Brown, Systems Architect at The Bank of New York Mellon, said in 2012 that his bank had 343 million lines of COBOL code. Transforming a code base of that size and complexity would take years and hundreds of millions of dollars.

Just ask the executives at the Commonwealth Bank of Australia, which undertook a planned AU $580 million (US $413 million) legacy core banking technology transformation in April 2008. The project was finally completed in August 2013 at a final cost of AU $1.1 billion (US $783.871 million) – almost two years behind schedule and $370 million over budget. And this was a successful legacy system conversion.

The COBOL programming language is not destined for retirement. Organizations can plan on continuing to spend a high percentage of their IT budgets maintaining legacy systems, especially in the financial and government sectors. An April 2017 article by NextGov.com claims that the federal government spent roughly $90 billion in 2017 maintaining legacy systems, which was roughly 80% of its entire IT budget ($90 billion in 2017).

Until new technology comes along to enable organizations to better understand their COBOL-based applications, unravel their complex and spaghettified code, and extract the embedded technical, business, and regulatory knowledge buried within, this trend will continue. Whichever approach you take, right now the only solution is to throw time and money at the problem.

Originally published on April 24, 2019, by Greg Brueggeman.

Greg Brueggeman is the Director of Product Management at Phase Change Software. You can reach him at gbrueggeman@phasechange.ai.

July 12, 2019No Comments

COBOL is dead! Long live COBOL!
A living collection of COBOL articles

Despite its age and multiple reports of its impending death, the Common Business Oriented Language (COBOL) remains responsible for a large portion of the world’s daily financial transactions – credible estimates include as much as $3 trillion per day and roughly 90 percent of all ATM and in-person financial transactions.

COBOL was first published in January 1960 by the Conference on Data Systems Languages (CODASLY), who based it on the first compiler developed by Rear Admiral Grace Hopper and her team at Remington Rand in 1952. It’s designed to develop portable business applications that could be run on systems developed by multiple manufacturers.

It remains vital to the world’s financial systems because of its simplicity and reliability.

One measure of its importance is the number of news and commentary articles published in reliable industry sources that repeat a common theme, namely that the programming language is ancient, nobody wants to use it, but it’s so vital to the financial and government sectors that it won’t go away – COBOL is dead! Long live COBOL!

Once or twice a year a new piece pops up and we typically pass it around the office, discuss new information or opinions it reveals, and archive it.

Recently, one of our shrewd colleagues suggested we post links to these articles here on our website so others in the small but influential COBOL community can reference them.

So we did. We’ll update this page when we discover new COBOL media pieces. If we’ve missed something important, email terickson@phasechange.ai.

Long live COBOL!

Articles
2018

Quartz Obsession: COBOL
What's going to happen when all the Baby-Boomer COBOL developers retire?
June 28, 2018
by Justin Sablich, qz.com
https://goo.gl/C7Ykv1

In digital transformation top banks are leading
Instead of ripping and replacing legacy systems and code, which can be prohibitively expensive and time consuming, some banks are maintaining these systems and wrapping customer engagement systems around them.
April 3, 2018
by Tom Groenfeldt, Forbes.com
https://goo.gl/W7WHHM

It’s Cobol all the way down
COBOL-based systems continue to run much of the world’s financial systems. But its supporting workforce is retiring and efforts to convert these applications to modern programming languages are expensive and time-consuming.
April 2018
by Glenn Fleishman, Increment.com
https://goo.gl/QpnUFa

2017

COBOL is everywhere. Who will maintain it?
Many of the world’s financial institutions and U.S. government agencies, such as Homeland Security, Department of Veterans Affairs, and Social Security, rely on COBOL-based systems, but a shortage of programming talent and education institutions that provide programming courses is on the horizon.
May 6, 2017
by David Cassel, The New Stack.io
https://goo.gl/InzR48

Trump said government has one 40-year-old IT system. It actually has at least 10.
A list of 10 U.S. government computer systems that are at least 40-years old. Three of the systems run on COBOL code.
April 12, 2017
by Frank Konkel, Nextgov.com
https://goo.gl/BSu37t

Banks scramble to fix old systems as IT 'cowboys' ride into sunset
Organizations that rely on Cobol-based applications have a hard time replacing retiring programmers and support personnel, which has given veteran developers opportunities to continue working, even after retirement.
April 9, 2017
by Anna Irrera, Reuters.com
https://goo.gl/TcxjXa

2016

Why it’s time to learn COBOL
Acquiring COBOL programming skills might be a wise career move. Hundreds-of-billions of lines of COBOL code are still in use and many universities have stopped offering classes in the 50+ year old language.
April 1, 2016
by Paul Rubens, CIO.com
https://goo.gl/Jkuj7G

2015

The inevitable return of COBOL
The looming shortage of COBOL programmers will inevitably lead to COBOL programming once again becoming an in-demand skill set.
July 6, 2015
by Ritika Trikha, HackerRank
https://goo.gl/EuSH6c

2014

CIOs should prepare for lack of Cobol (Yes, Cobol) developers
While the demand for talented and skilled Cobol programmers remains steady, the programming language’s lack of popularity has shrunk the available talent pool. As the existing Cobol support workforce ages and retires, companies are resorting to novel strategies to acquire and train staff.
October 2, 2014
by Sharon Florentine, CIO.com
https://goo.gl/1XA1KG

Cobol is dead. Long live Cobol!
CIOs that rely on Cobol-based systems keep developer staff as long as possible while others prefer new hires with multi-language capabilities over Cobol-specific or Cobol-only skills.
October 2, 2014
by Gary Beach, Wall Street Journal CIO Journal blogs
https://goo.gl/HmH5K8

All the rich kids are into COBOL – but why?
COBOL isn’t sexy or even that popular. But the basic tenants of supply and demand remain true – if there are still a lot of COBOL applications running critical systems and not a lot of programmers interested in learning the 50-year-old programming language, then brushing up on your COBOL skills might make it easier to find a job earning more money.
September 17, 2014
by Matt Asay, readwrite.com
https://goo.gl/AZNFPi

The government’s COBOL conundrum
The U.S. federal government’s Office of Personnel Management released its “Strategic Information Technology Plan” for revamping the agency’s IP operations. Part of the plan discusses the office’s plans for maintaining and eventually migrating away from the roughly 60-million-lines of production COBOL code that enable the agency to meet a number of its regulatory requirements.
June 2, 2014
by Nicole Blake Johnson, FedTech magazine.com
https://goo.gl/Sqeo6b

2012

Brain Drain: Where COBOL systems go from here
Not only does losing experienced COBOL programmers hurt many companies’ ability to maintain its mainframe systems, but it also means the loss of the programmers’ deep understanding of the business logic. A number of organizations are teaming with private businesses to educate younger programmers and team them with experienced developers before it’s too late.
May 21, 2012
by Robert L. Mitchell, CIO.com
https://goo.gl/8UKvPg

Cobol brain drain: Survey results
Results from the Compuworld survey on Cobol use in business and government, which showed that nearly 50 percent of respondents had operational Cobol-based systems and large number continue to develop new business applications with Cobol.
March 14, 2012
by Staff, Compuworld.com
https://goo.gl/oJdB3D

The future of COBOL: Why it won’t go away soon – Part 2
When thinking about maintaining or replacing their COBOL systems, companies must consider the employee angle. Can they continue to hire COBOL programmers when experts forecast that a major COBOL skills gap in on the horizon, and is that enough of a reason to rip and replace?
Date: January 11, 2012
by Brian Bloom, IT World Canada
https://goo.gl/ksDWkZ

The future of COBOL: Why it won’t go away soon – Part 1
COBOL-based systems will not be going away anytime soon because of the millions of invested man-hours and dollars already spent to develop these mainframe programs and the enormous predicted replacement costs. There’s also the fact that we don’t have anything better enough to make the change.
January 10, 2012
by Brian Bloom, IT World Canada
https://goo.gl/cRjaPk

Fun

*All images are copyrighted by their respective owners

Are you a COBOL programmer?
November 4, 1997
by Scott Adams, Dilbert.com
http://dilbert.com/strip/1997-11-04
Are you a COBOL programmer?

 

The Holy Grail of [programming] technology!
June 10, 1994
Scott Adams, dilbert.com
http://dilbert.com/strip/1994-06-10
It's the Holy Grail of programming! dilbert cartoon

 

 

 

 

updated October 30, 2018
by Todd Erickson

*Todd Erickson is a tech writer with Phase Change. You can reach him at terickson@phasechange.ai.

July 10, 2019No Comments

Why is COBOL cool again?

Discover why the recent spotlight on COBOL systems and the shortage of qualified COBOL programmers aren’t due to a lack of qualified engineers, it's due to a lack of knowledge.

Reuters and The New Stack recently published articles about COBOL, an often-overlooked programming language that was developed before John F. Kennedy became the 35th President of the United States.

At Phase Change, we pay attention to legacy systems and their challenges. So, why was a mainframe language developed in 1959 suddenly the topic of multiple news articles?

The U.S. government developed the common business-oriented language (COBOL) in conjunction with Rear Admiral Grace Hopper and a coalition of industry and higher-education envoys. It's simplicity and portability have stood the test of time, and are the main reasons why 60-year-old COBOL applications continue to play a critical role in finance, banking, and government operations. That plus the inertia that characterizes large, critical systems.

Organizations like the Department of Veteran Affairs and large financial companies, such as Bank of New York Mellon and Barclays PLC, are examples of the types of institutions that rely on COBOL applications for nearly $3 trillion worth of daily transactions. But they’ve used COBOL for decades, so, that doesn't explain the recent attention.

It's because the engineers that maintain COBOL-based systems are leaving the workforce, there aren't qualified developers available to replace them, and these institutions are freaking out. The COBOL brain drain is threatening the organizations that economies are built upon.

Brain drain refers to how departing software engineers leave with all of their system and domain knowledge supposedly locked away in their brains. That knowledge is thought to be lost from the organization forever.

The average age of a COBOL programmer is somewhere between 45 and 60 years old and they are retiring. The problem is that few programmers are interested in replacing them, and the availability of COBOL training resources has dropped precipitously because it's just not a cool language anymore.

We won't repeat all of the statistics that show how much COBOL code is still in use and how important those systems are. Read the Reuters and The New Stack articles, which both mirror a series of comprehensive feature articles published by ComputerWorld in 2012. The metrics and themes haven’t changed much.

You can also follow the "official" COBOL Twitter account, @morecobol (spoiler: it's clever).

Basically, these companies have three options to deal with COBOL brain drain, and all involve high risks. First, they can simply replace their COBOL systems with systems built on more modern programming languages. That project took the Commonwealth Bank of Australia 5 years and $749.9 million, which was 30% over budget. The risk associated with implementing such a massive new system has kept most financial institutions from doing it.

Second, they can engage consultants like the Cobol Cowboys, or hire and train new programmers to support their COBOL systems. This option also involves a great amount of risk because companies have to find engineers that have the skills and interest to support COBOL applications, and then hope they can unravel the layers of modifications and system integrations that accrue with five decades of maintenance, usually with little documentation.

Third, they can completely stop modifying core systems that nobody understands, but are too critical to risk changing or replacing. The USDA faced that choice.

It's not a people problem

But from our perspective, the issue is not a human-resources problem. The companies that rely on COBOL-based systems don't lack the right people, they lack the right knowledge.

If the new engineers assigned to work on COBOL-based applications could access the departing developers' system and domain knowledge, or better yet, all of the programming and domain knowledge imbued into the system from prior engineers, imagine how much easier it would be for them to comprehend these complex systems. It would be like having a personal mentor always available — even while the previous engineers are off enjoying retirement.

That's why this is a knowledge problem and not a people problem.

And it's a huge opportunity. Unlocking the encoded knowledge that's trapped in COBOL systems will give large institutions the knowledge they need to make informed decisions about their legacy systems.

Learn more at CodeCatalyst.ai.

Originally published on May 25, 2017, by Todd Erickson and Elizabeth Richards.

Todd Erickson is a tech writer with Phase Change. You can reach him at terickson@phasechange.ai.

Elizabeth Richards is Phase Change's director of business operations. You can reach her at erichards@phasechange.ai.

Code Catalyst

Golden, Colorado

+1.303.586.8900

info@codecatalyst.ai

Legal
Privacy Policy
Terms of Use
Copyright © 2019