Comair on Christmas Eve discovered the risk of using computer software long past its prime when its crew tracking system crashed under the weight of too many crew changes that needed to be made because of a major snowstorm.

The Cincinnati-base regional unit of Delta Air Lines [DAL] did not resume its full schedule of 1,160 daily flights until Dec. 29. And now, the U.S. Department of Transportation (DOT) is investigating Comair's holiday operations.

Regional Aviation News surveyed other regional carriers and discovered some old software still operating at other carriers, but none apparently as old at the 1986 SBS International crew tracking software used at Comair.

One carrier, Horizon Air [ALK], uses a version of SBS Crew Check that had a close call last year during a snowstorm, but did not come close to hitting its limitations. Another carrier, Mesaba [MAIR], was forced to cancel flights in December 2000 when its software failed. It has since upgraded its systems.

"If you survey the airlines, there is a general level of dissatisfaction with their systems whether they built it themselves or bought it from someone else," said Warren Luce, who markets crew planning software for Advanced Optimizations Systems (AOS). While there is a level of dissatisfaction with the software, Luce told Regional Aviation News, "I am sure there is no willingness to pay. This is not the time to consider changing one of your primary systems. I have never seen the price tag, but I am guessing you are talking in the tens of millions. Every crew member potentially would need to be re-trained."

AOS does not compete with SBS, now a unit of Boeing [BA], and Sabre Airline Solutions [TSG]. Its pre-flight planning programs are used with both SBS and Sabre day-of-operations crew tracking software as well as the proprietary programs developed by individual airlines. "We are finished about three weeks before a flight," Luce said of his product.

"Every single airline we work with knows and appreciates the old legacy systems that have frailties. From our position we don't see that today's technologies are being used. No one has started from a clean slate and used the capabilities and rich features of today's data base management systems. No one has wiped the slate clean and started with a new system.

"What has happened almost everywhere is that the old system have been added to. They have been improved and upgraded. The old systems have been carried forward with new software and enhancements," Luce said.

"There a lot of real-time options that were not considered when these systems were designed and built. There are new requirements for data collection and the union contracts continue to grow in complexity. It is not a simple thing at all to update these programs. It is not a trivial thing to start over. It is a massive thing taking a lot of time and money."

Since DOT has begun probing Comair's operations, Comair will no longer discuss its Christmas operations. Based on the limited information released prior to the DOT announcement, computer experts have speculated that if Comair spent a few more cents - literally a few more pennies - to alter the 1986-vintage computer software more than a decade ago, its limits would have been increased tenfold from its capacity of 32,000 crew assignments. To its credit, Comair did realize that its software needed to be replaced. Last June, Comair announced it was purchasing new crew tracking software from Sabre Airline Solutions (CRAN, June 14, p. 7). Neither Comair nor Sabre would say when the new software will be ready.

Both Luce and Steven Clampett, president of Sabre Airline Products and Service, said it is very rare for an airline to switch the basic core program at the heart of a crew tracking system.

Comair's sister regional, Atlantic Southeast Airlines (ASA), has used the Sabre CrewTrac since 1997 - pre-dating its acquisition by Delta.

In the last three months, Clampett said, Sabre has sold additional crew tracking software components to ASA and Pinnacle Airlines [PNCL]. Worldwide, he said, 115 carriers use Sabre's crew tracking products, including 55 carriers in the U.S.

A Boeing spokesperson said that SBS could not talk about its programs due to the on going DOT investigation. The company declined to talk about its market share.

While spending millions in product development each year, Clampett said Sabre makes at least one "significant" improvement in each software package each year. Over the course of the year, he added, there are other minor improvement releases to the programs.

While Sabre makes annual improvements, he said, most airlines are "taking some update every other year. Few go very far beyond that." A few carriers are "aggressive" and buy the annual program improvements, he noted.

Some Sabre products are modules that can be easily plugged into its own systems or competing systems. For example, crew pairing software can be adapted to run with a variety of crew tracking programs.

However, Clampett said there is little "mixing and matching" of vendor products in the core systems that track crews on a real-time basis.

Apparently part of Comair's problem is that it failed to have back-up program or a mirror location of its crew tracking software. The program required all the data to be manually re-entered. Back-up and recovery solutions vary with each carrier and are not part of a standard package, Clampett said.

The software that is based on a Unix server has more back-up options than those programs operating in a PC-only world. Sabre sells its programs for both computer environments.

"We sell solutions where we will host the server here and we do have a disaster recovery program," he added. "To date, only a small number of carriers participate in this program."

Even if a corporate family of carriers - American Eagle and American Airlines [AMR], for example - had common programs that could in theory provide backup protections, the back-ups may not be effective because they don't share the same work rules, Luce said. "At one level every one would be the same, but its not. The devil is in the details. You cannot underestimate how specific and unique each carrier is. It is tailored to a lot of very special need."

Survey Notes

Horizon Air: It currently uses SBS Trac while its sister carrier, Alaska Air, uses SBS Maestro. "We can function with the product as it is," said Horizon spokeswoman Jen McSkimming, but "we are looking for something more modern."

During a snowstorm in early 2004, McSkimming said Horizon experienced its worst-case scenario. The maximum assignments the carrier's software handled that day was 10,000 trips. "We were not in harm's way."

Mesaba Aviation: When Mesaba built its new headquarters in 2003, it included a new operations center and updated its Sabre Crew Trac software at that time. In addition, it built a duplicate operations center in a separate building with a mirror copy of the Sabre software to avoid any downtime in case the primary computers crashed, said spokesman David Jackson. After Mesaba experienced the software crash in 2000, he said, it upgraded its software at that time.

Pinnacle Airlines: Comair wasn't the only regional carrier grounded by the Dec. 23 snowstorm. Three inches of ice accumulated in Memphis, which caused Pinnacle to cancel two out of three banks of flights that day, said spokesman Philip Reed. The SBS crew tracking software performed perfectly that day, he said.

Pinnacle upgraded its software two years ago in anticipation of growth. The program can handle 139 aircraft. Pinnacle currently has 114 of the 139 Bombardier [BBD] CRJ 200s that Northwest Airlines [NWAC] will be placing with the carrier.

Next month, Reed said, Pinnacle will link its crew tracking software with Flite Trac, which tracks the aircraft, so that two programs can share real-time information.

Unlike some other carriers, Pinnacle keeps its crews assigned to the same aircraft and track the paired crew by the aircraft's tail number. With its aircraft scheduled "to touch a maintenance base" every four days, Pinnacle uses its Kronos software to develop its crew pairing schedules.

Independence Air [FLYI]: When Flyi needed establish new computer systems last spring as it set up Independence Air, it did not need to update its crew tracking software, said spokesman Rick Delisi. Flyi's flight schedule is much less complicated - thus less taxing - now than it was a year ago when it operated from five hubs for both Delta and United Airlines [UALAQ]. Independence only operates out of Washington Dulles.

When the carrier last tested it back-up system, Delisi said Bornemann Crew Trac software was fully restored within one hour.

ExpressJet [XJT]: Unlike most other carriers, ExpressJet and its former parent, Continental Airlines [CAL], use an in-house program developed for them by Electronic Data Systems [EDS]. "The system was last updated the week of Dec. 13 and it is typically updated weekly for any maintenance type items and to add or change crew information," said spokeswoman Kristy Nicholas.

Air Wisconsin: When Air Wisconsin upgraded its computer servers last August, it upgraded its Sabre Crew Trac software. "Air Wisconsin has implemented both fully redundant hardware and software to manage its system operations center," said spokeswoman Kelly Lanpheer. The carrier routinely tests its business recovery/continuity plans. In a November test, the information technology crews were able to fully recover to a new sever in less than four hours.

American Eagle: Both American Eagle and American Airlines operate the same system, Crews. "There are back up plans in the very unlikely event that system would go down," said spokeswoman Lisa Bailey. Crews was developed by Sabre when AMR owned Sabre, but it is no longer considered a Sabre product. 

>>Contacts: Warren Luce, AOS, (732) 390-8460; Steve Clampett, Sabre, (682) 605-2252; Jen McSkimming, Horizon, (206) 431-4672; David Jackson, Mesaba, (651) 367-5264; Philip Reed, Pinnacle, (901) 348-4257; Rick Delisi, Flyi, (703) 650-6019; Kristy Nicholas, ExpressJet, (713) 324-4755; Kelly Lanpheer, Air Wisconsin, (920) 749-7638; Lisa Bailey, American Eagle, (817) 963-1602.<<