Software problems related to FAA’s $2.1 billion En Route Automation Modernization (ERAM) project are threatening to delay the agency’s multi-billion-dollar Next Generation Transportation System (NextGen) initiative.
In its simplest form, ERAM will process flight radar data, provide communications support and generate display data for air traffic controllers at all 20 FAA Air Route Traffic Control Centers (ARTCC) in the United States. ERAM is designed to replace functionality embedded in the decades-old Host upper airspace traffic computer network, including its backup and ancillary services, with an open, supportable and technologically modernized en route automation environment. Key programs written in obsolete Jovial and Basic Assembly languages would be replaced by Ada to bring top performance and high reliability.
Initial deployments of ERAM, notably at the Salt Lake City ARTCC, by prime contractor Lockheed Martin, have revealed problems with its software system, causing the full deployment of the system to be pushed back from its original deadline of 2010 to 2014, and boosted the budget millions of dollars beyond initial estimates. “As in any highly complex software program, when operationally deployed, modifications are necessary. Lockheed Martin has made modifications, per FAA direction at the agreed upon cost and schedule,” Fran Hill, director of ERAM Program at Lockheed Martin said in a statement to Avionics Magazine.
Noting Lockheed Martin’s “unique expertise” in the en route automation environment, FAA intended to award it the ERAM upgrade in 2002 as a sole source, 10-year contract. However, Raytheon protested, causing FAA to establish an inquiry that subsequently upheld the protest, with Raytheon becoming a Lockheed Martin team member when the $2.1 billion contract was finally awarded in 2003.
The inquiry report revealed, however, that neither FAA nor Lockheed Martin appeared sure how the upgrade was to be accomplished, other than it would be an “incremental decomposition,” a term that evaded clear definition. Three officials gave three different views, while FAA’s integrated product team lead for en route said she did not understand how it could be accomplished or how often one would have to go into the system at any or all of the ARTCCs to perform the incremental modifications. “There’s nothing practical about this,” the FAA integrated product team lead testified. “This is the most complex thing that the agency will ever undertake.” Her ERAM product team lead also did not know how the “fairly complex technical task” of decomposing old software to get it to run on a new platform in a modern language might be performed.
FAA did not conduct a risk assessment to examine the potential costs and benefits of incremental decomposition, nor did it know exactly what had to be incrementally decomposed, although it did acknowledge that “this will necessitate multiple transitions of deployed ERAM functionality.” Yet, as the integrated product team lead explained, one must know what the software is in order to decompose it.
Unfortunately, the Host system had been built in stages, with new functions and capabilities progressively added. Consequently, it consisted of a set of separate hardware and software components physically interfaced together, but without a common design, infrastructure or software environment: a software “bowl of spaghetti” as both FAA’s integrated product team lead and the associate administrator for research and acquisitions described it to the inquiry.
Host software had been (and still is) enhanced to provide site specific functions used to meet local requirements, contained in libraries of national and local patches. But most ARTCCs do not use the same set of patches, resulting in a unique Host “build” for each site. In FAA’s words, “One of the greatest risks for the successful implementation of ERAM does not lie with the replacement of well-documented capabilities but that decomposition will fail to capture the unique ways in which the field has applied the capabilities of Hostsfunctions to their specific operations. For example, the use of national and local patches, unique airspace issues, management of external interfaces (and other ways) in which the controllers ‘manipulate’ the Host to make it do what they want.”
The end result has been a continuing source of problems at the first test installation at the low-activity Salt Lake City (SLC) ARTCC, where aircraft mis-identifications, failed handovers and a large number of other issues requiring controller “work arounds” have arisen.
The initial problems were described in Congressional testimony in November 2011 by the Government Accountability Office (GAO) and, separately, by the DOT Inspector General (IG). The GAO estimated ERAM would not be operational before 2014 and would be overspent by $330 million. The IG felt it could extend to 2016, and be $500 million over budget. “ERAM continues to face a substantial risk for cost growth, schedule delays and performance shortfalls as the program is deployed to more complex sites. These risks will grow as FAA and its contractor continue to add new capabilities while attempting to resolve problems in earlier software versions,” according to the DOT IG testimony.
But the true bottom lines are even worse. Operationally, Automatic Dependent Surveillance-Broadcast (ADS-B), Data Communications and System Wide Information Management (SWIM) each require a functioning ERAM to support NextGen.
Financially, it’s equally grievous. In 2007, the system was tested at FAA’s Atlantic City Technical Center in New Jersey, met agency specifications, and was accepted by FAA, although problems reappeared afterwards at Salt Lake City and seem to be continuing. Reportedly, the differences in performance were ascribed to the lack of testing at the Technical Center against real, rather than simulated, aircraft targets. Unfortunately, FAA acceptance means all costs above the original contract are on FAA’s tab, including continuing support of the nationwide Host system.
“ERAM’s problems are the result of a number of fundamental programmatic and contract management concerns, and prolonged problems with directly impact the cost and pace of NextGen,” according to the DOT IG testimony.
According to Lockheed Martin, ERAM has run continuously at Salt Lake City since October 2010 and in Seattle since December 2010. Three additional sites Denver, Minneapolis and Albuquerque, N.M, achieved initial operating capability in December. Chicago achieved initial operating capability in January.
“As we continue to work toward deployment at the remaining air route traffic control centers, there will be unique operational conditions at each site, and we will work with the FAA to make any modifications, if needed,” Hill said. “During all operational runs, ERAM has run safely without negative impacts to air traffic control operations that have resulted in operational error or airline/passenger delays.”
Hill said ERAM stakeholders, including Lockheed Martin, FAA and the National Air Traffic Controllers Association (NATCA), have learned a “great deal” from the work done in Seattle and Salt Lake City. The FAA has instituted a benchmarking process at each site to allow adaptation data and software changes to be prioritized for site operations. In addition, NATCA is involved in defining the system changes early in the system engineering process to ensure that all changes meet the end-user needs. NATCA is also involved in testing of all new system releases in the factory, and a new phase of testing has been added where new system releases are tested at multiple sites to enhance the opportunity to find changes and issues earlier in the lifecycle.
“End-user involvement in all phases of the engineering and deployment lifecycle is a key ‘lessons learned’ that has been institutionalized in an excellent set of process changes,” Hill said.