Boeing Outsourced Software to $9 Programmers, Probe Expands Beyond the Max

-edited

Boeings problems continue to mount. The FAA discovered more problems and a DOJ probe expands beyond the Max.

New Uncommanded Dive Risks

Last week the FAA issued statement on a new risk that Boeing must mitigate. The new risk does not involve the MACS but could lead to similar results according to the Seattle Times.

The Federal Aviation Administration discovered that data processing by a flight computer on the jetliner could cause the plane to dive in a way that pilots had difficulty recovering from in simulator tests, according to two people familiar with the finding who asked not to be named discussing it.

While the issue didn’t involve the Maneuvering Characteristics Augmentation System linked to the two accidents since October that killed 346 people, it could produce an uncommanded dive similar to what occurred in the crashes, according to one person, who wasn’t authorized to speak about the matter.

David Learmount, consulting aviation-safety editor at Flight Global and a former Royal Air Force pilot, said details of the new issue are sketchy but it’s possible that it could further delay the MAX’s return. “The implication is that this is different software in a different control computer that’s presenting similar symptoms,” he said. “When you control an aircraft with computers, which we do now, you’ve always got potential for problems.”

Boeing agreed with the FAA's findings but has not yet presented a fix to the FAA.

DoJ Probe Expands to Dreamliner

The Seattle Times reports DOJ probe expands beyond Boeing 737 MAX, includes 787 Dreamliner.

Federal prosecutors have subpoenaed records from Boeing relating to the production of the 787 Dreamliner in South Carolina, where there have been allegations of shoddy work, according to two sources familiar with the investigation.

The subpoena was issued by the Department of Justice (DOJ), the sources said. DOJ is also conducting a criminal investigation into the certification and design of the 737 MAX after two deadly crashes of that jetliner. The 787 subpoena significantly widens the scope of the DOJ’s scrutiny of safety issues at Boeing.

The grand-jury investigation into the MAX has been cloaked in secrecy, but some of the Justice Department’s activities have become known as prosecutors issued subpoenas for documents. Allegations relating to the 787 Dreamliner have centered on shoddy work and cutting corners at the company’s South Carolina plant.

Prosecutors are likely looking into whether broad cultural problems run throughout the company, according to the third source and a person in South Carolina, who also spoke on condition of anonymity because of the sensitive nature of the matter. That could include pressure to sign off on faulty work to avoid delays in delivering planes to customers, the source said.

The entire [Dreamliner] fleet was grounded in January 2013 after two battery-overheating incidents: a battery fire on an empty 787 parked at the gate at Boston airport, then a smoldering battery on a flight in Japan that forced an emergency landing. The FAA lifted the grounding in April 2013 after Boeing modified the jets with beefed-up batteries, containment boxes and venting tubes.

In the 737 MAX investigation, prosecutors appear to be getting information from someone with inside knowledge of the plane’s development based on the questions they are asking, the third source said.

$9 an Hour Programmers with No Aviation Experience

Shoddy work and cutting corners? Uh ... Yes.

Bloomberg reports [Boeing’s 737 Max Software Outsourced to $9-an-Hour Engineers](Boeing’s 737 Max Software Outsourced to $9-an-Hour Engineers)

It remains the mystery at the heart of Boeing Co.’s 737 Max crisis: how a company renowned for meticulous design made seemingly basic software mistakes leading to a pair of deadly crashes. Longtime Boeing engineers say the effort was complicated by a push to outsource work to lower-paid contractors.

The Max software -- plagued by issues that could keep the planes grounded months longer after U.S. regulators this week revealed a new flaw -- was developed at a time Boeing was laying off experienced engineers and pressing suppliers to cut costs.

Increasingly, the iconic American planemaker and its subcontractors have relied on temporary workers making as little as $9 an hour to develop and test software, often from countries lacking a deep background in aerospace -- notably India.

In offices across from Seattle’s Boeing Field, recent college graduates employed by the Indian software developer HCL Technologies Ltd. occupied several rows of desks, said Mark Rabin, a former Boeing software engineer who worked in a flight-test group that supported the Max.

The coders from HCL were typically designing to specifications set by Boeing. Still, “it was controversial because it was far less efficient than Boeing engineers just writing the code,” Rabin said. Frequently, he recalled, “it took many rounds going back and forth because the code was not done correctly.”

Double Dividends

Not only did Boeing benefit from cheap coders who did not know what they were doing, Bloomberg notes that Boeing’s cultivation of Indian companies appeared to pay other dividends.

Boeing won several orders for Indian military and commercial aircraft, such as a $22 billion one in January 2017 to supply SpiceJet Ltd. That order included 100 737-Max 8 jets and represented Boeing’s largest order ever from an Indian airline, a coup in a country dominated by Airbus.

Grounded for Cause

The Wall Street Journal reports Boeing 737 MAX Likely Grounded Until Late This Year.

Boeing Co.’s troubled 737 MAX fleet is expected to stay grounded until late this year as a result of the latest flight-control problem flagged by U.S. air-safety regulators, according to people briefed on the issue.

The setback, at the very least, is expected to prompt additional disruptions to airline schedules across the U.S. and overseas as some 500 of the planes remain idled for months longer than previously projected.

During simulator tests of certain emergency procedures, FAA pilots uncovered a potentially dangerous situation they hadn’t encountered before, according to people briefed on the issue. The crux of the problem, according to the Boeing official and company messages to airlines, is that if a chip inside the plane’s flight-control computer fails, it can cause uncommanded movement of a panel on the aircraft’s tail, pointing the nose downward.

Tests of the emergency procedures to cope with this so-called runaway stabilizer condition, the official said, revealed that it would take average pilots longer than expected to recognize and counteract the problem.

Darn those Simulators

When you use actual flight simulators instead of iPads problems turn up. But all along Boeing has insisted and still insists iPads are all the pilots need to train.

No New Parts Needed

“We believe this can be updated through a software fix,” a Boeing official said.

Of course it does.

It could take many more months if the 737 fleet needs new parts.

What Can Possibly Go Wrong?

Boeing took a base 1964 design, overloaded it with huge engines making the aircraft unstable, then depended on poorly designed software that cannot easily be overridden to keep the plane from nosediving in crashes, while insisting training can take place on an iPad.

What could possibly go wrong with that set of cost-cutting decisions?

Unfortunately, we just found out.

Yet, even after the second crash, Boeing begged the FAA to keep the plane in service.

Mike "Mish" Shedlock

Comments (56)
No. 1-22
shamrock
shamrock

Being in the software development business, I can attest that having developers who suck on the team is way worse than having no developers at all.

Thalamus
Thalamus

You need to hit those numbers as management to get those bonuses.

JL1
JL1

Somebody in Boeing leadership said : "Hey, let's outsource software development to India and we will save money " Every other executive at Boeing agreed. Everything went fine for a bit and profits increased and executives got bigger bonuses and more money from their options because the failures and problems were hidden with workarounds. Then reality entered and Boeing brand and image and products Were destroyed. Every executive at Boeing should be FIRED and then sued and asset-stripped to their underwear by shareholders.

JL1
JL1

How many other American companies have massive problems and liabilities currently hidden that have come from saving a few dollars by outsourcing software development to India or bringing cheap indian programmers to USA on H1B-visas who are NOT More talented than American programmers they are just cheaper and allow company executives to extract more bonuses and options from the company just like happened with boeing?

mark0f0
mark0f0

Meanwhile real US citizen engineers, software, computer, hardware, sent their applications to Boeing for jobs. And were ignored. While Boeing hired a firm that no self-respecting talented US citizen would even bother to apply to, to staff the positions. Effectively creating a self-fulfilling prophecy that foreign nationals were required for jobs "Americans wouldn't do".
The solution is to ban the H-1B visa. It has destroyed so many US citizen STEM jobs, and left so much great talent "on the couch" or "in the basements" of the nation.

mark0f0
mark0f0

@shamrock But you don't build good developers without letting them develop experience. Not saying that grads should go straight from college, to programming rocket ships. But the industry expectations are somewhat out of whack from reality, and unfortunately its the foreign nationals who have nothing to lose and lots of reason to lie about competency who get the jobs. The executives love how cheap they are.

Webej
Webej

S H O D D Y

One word says it all. Look to the leadership to find out how shoddy creeps in everywhere. It seems the emblem of our times. Shoddy leaders with shoddy morals and shoddy vision, everywhere throughout the West.

It's also more than shoddy. It's fraud. Not resubmitting the design to the FAA after changing the limits (4× earlier design) of MCAS is fraud. And the relationship between the FAA/Boeing, also shoddy and arguably fraudulent. As always, there will no charges or trials.

conscript
conscript

Ah yes! And when the $9 an hour outsourcing {engineering} finally gets to effect fund managers, bankers, medical doctors, on and on, everyone will FINALLY understand why the middle Amerika {those steel workers , coal miners, general factory workers, etc} voted for Trump. A non-adiabatic economy only works for few at the expense of many others.

Sechel
Sechel

$9 an hour programmers doesn't concern me. Not supervising them and having good testing procedures would. I only skimmed the article but not seeing anyone is alleging that software wasn't tested and undergone a thorough code review etc

Carl_R
Carl_R

In an earlier life, I was a systems designer and systems programmer, and a good one. I concur that a bad programmer is worse than no programmer at all. Nevertheless, I have no problem with $9/hr programmers from India. I don't care what they make. I don't care where they are from. I do care if they are competent. A bad programmer is a bad programmer, whether they make $9/hr, or $90/hr.

The problem here is bad code, not who wrote it. Blaming it on $9/hr contract programmers is all too easy. Where wrote the specs? Who supervised the programmers? Who tested the code? Who tested the entire system? There is far more than enough blame to go around.

Grumblenose
Grumblenose

I've worked with Indian programmers and I know what they can do. I wouldn't let them within 10 miles of anything mission-critical.

Greggg
Greggg

9 dollar per hour engineer - Human Resource Department Translation:

Student engineer in 3rd year with no job experience.

Casual_Observer
Casual_Observer

There also needs to be blame put on the move to South Carolina. Boeing moved there for tax breaks and cheaper labor and a right to work state. Lindsey Graham is at the center of this. This is where the road to lower regulations leads. You reap what you sow.

Mish
Mish

Editor

I worked over 20 years in programming. I made AVP at Harris Bank now BMO after 2 years.

We hired many contractors. The best were well-paid. The worst were not.

The problem is management. They assumed that at the bottom level, a programmer is a programmer is a programmer.

I was in finance. The best contractors knew finance. The worst did not know squat.

Boeing is aviation. I suspect the best know something about what they were doing but most don't.

Expecting to get quality code out of poor designers and programmers is a huge mistake.

Yes, this is a management fuck-up but I seriously doubt those $9 programmers were any good.

KidHorn
KidHorn

I worked for a company that setup a development farm in India. It wasn't to replace US workers. Rather, it was to augment US workers. At least that's what the CEO told us. I managed a group or 6 or so in Bangalore. Not only were they extremely sub par. I spent my days fixing their mistakes. Their culture is not suited for team work. Every one of them was trying to claw their way to the top. In the end, we hired maybe 25 people in India and 1 US worker lost their job. I don't know the exact economics of it, but I suspect we lost money.

BillSanDiego
BillSanDiego

The problem is not the programmers. You could have the programming done by Americans being paid $150/hr and the planes would still be accidents waiting to happen.

The problem is that the engineers designed an airframe that was aerodynamically unstable and was essentially unflyable, and rather than spend the time and money to fix the problem by redesigning the airframe to make it stable in flight, they spent far less money to impart a false stability by means of computer controls to override that aerodynamic instability. If the computer fails for any reason, bad programming being only one of many reason a computer can fail, the aerodynamic instability of the aircraft itself causes the airplane to crash.

msurkan
msurkan

Mish is incorrect to say that Boeing’s changes to the original 737 made the aircraft “unstable”. The new versions of the 737 (such as the MAX) are in fact very stable. The problem is that the changes to the 737 changed the flight characteristics of the aircraft which should have required retraining and certification for pilots. In order to avoid requiring pilots to retrain on the new 737s (which customers don’t like having to pay for), Boeing created software to mimic the behavior of old 737s from the controls.

It is this classic 737 software mimicry (to avoid having to recertify pilots) that is the real problem. This software mimicry doesn’t always work and relies on sensors which can be faulty (e.g. such as having a bird hit an air speed sensor). It is telling that the pilot instructions for handling flight problems is to disable the mimic software and fly the plane in a native mode. Unfortunately, very few 737 pilots have experience with how to handle the plane without the mimic software, and they only get the chance to try when something is really bad, which is the exact worst time to start learning how a new model 737 really behaves.

Carlos_
Carlos_

The problem is not the 9 an hour programmers. An airplane has multiple SW units. The 9 an hour may have been involved writing SW for the entertainment unit. We just do not know. I worked in SW developing real time or near real time SW for control systems. It is totally different to say SW for financial applications. A control system piece of SW will normally have a very detailed requirements document, a very detailed code review phase, a very stringent unit test phase and finally a system test. If during those phases something is discovered then you will need to go back and do regression testing. All those phases normally capture faulty code. In essence code that does not match the requirement. Also, SW for control systems is written in heavily structured language such as Ada which I think Boeing still uses. This avoids coding errors. 9 an hour programmers do not know Ada and Ill be surprise if they can write C or C++. I think they are being used as scapegoats. I think the plane is a bad HW design that can not be completely fixed with SW. In other words more HW fixes are needed. As for the 787 well that is built in a nonunion plant as far as I can tell. I know this is not an audience that likes unions. However, unions do have a role keeping business manager in check.

stillCJ
stillCJ

Editor

When an airliner becomes unflyable because a computer chip fails, we have huge problem in the industry. Lately I have been encountering similar problems in excessively complicated automotive computer systems, which always end up being very expensive to fix, and few places even know how to fix it.

NormGriffin
NormGriffin

Hi Mish, in my opinion, this issue is more of a failure of the design team writing the software function than the salary of the programmers. Before a programmer saw a line of code, the scope of the project and function should have been well defined. Apparently , they forgot what action the software should take to insure the angle of attack sensor was giving the correct information and what to do if it had failed. The fact it happened twice , causing two crashes, it’s unforgivable.

mark0f0
mark0f0

any quality internship pays more than $9/hour for a US citizen.

frozeninthenorth
frozeninthenorth

Wow guys, wow! You all seem to believe that only American Software engineer can write good code...I disagree! Certainly, it's hard to believe that $9 /hrs software engineers of high quality exist. When we hired Indian, Mexican and other software companies (almost all for data processing) we found the error rate excellent. As to writing great software, that's got nothing to do with the country and EVERYTHING to do with the actual engineer writing the software!

However, Boeing was is search for the ultimate cost-cutting and share buyback program. Rather than investing in the company, they decided they didn't need the capital and returned it to their investors. By some happy coincidence, the share price rose from $150 to $450...who knew (sarcasm intended)