I was surprised to read about the Therac-25 accidents, not because I don't believe that a software mistake could kill someone, but because I had never heard about it. It makes sense to learn about it in an ethics class, but I feel like a case study like this should be important to introduce to CS students early on in their studies, just so it's at the back of their mind. They should be aware that what we do and what we create has the potential to have important and significant ramifications to those that use it. It doesn't have to be a big, in-depth thing, but it should at least be mentioned, perhaps, when we start learning about code and start to encounter bugs.
That all being said, Therac-25 was a radiation therapy machine introduced by Atomic Energy of Canada Limited (AECL) in the 1980s following similar units donned with the same name (but different number). The difference between the Therac-25 and its predecessors, however, was that the Therac-25 featured safety measures that were implemented through software rather than hardware. Because of the hardware measures previously in place, AECL overlooked a bug in the original software that was ignored because the hardware took care of the problem the bug created. This would prove to be a deadly mistake. The bug allowed the radiology technician to set the machine incorrectly if they make a mistake in typing at a critical point in the machine's aligning of the magnets. If the magnets were not properly placed, a "Malfunction 54" appeared, which indicated that the machine did not know whether there was going to be an underdose OR overdose of the radiation. This message, however, apparently occurred relatively frequently and did not appear in the manual, so technicians tended to ignore the error when it occurred because it seemed trivial. However, when the magnets were not set and the patient received the radiation, the doses were at hundreds of times greater than normal, resulting in the deaths of at least 3 people and serious injuries of 3 more. One challenge that software developers working safety-critical systems face is a very thorough checking of their code, testing every single possible case, and making sure everything works in the expected way. It is crucial that when code has the potential to kill or seriously injure someone, lose or severely damage the equipment/property, or cause environmental harm that they take great care to make sure that these systems do not fail. I think that they should approach these very carefully knowing the magnitude of impact a mistake or bug can have. When accidents happen, I do not think that it is only the software engineers at fault. There are many different people working on a single project; it will probably almost never be just the software engineers who originally designed the code. One must also lay blame on the people who test and review the code because they are the ones who need to make sure they catch these mistakes. There should be a very structured way of combing through code to prevent mistakes caused by bugs or unknown input. However, because we are only human, it is difficult to catch every single thing. People who are going to be using products that have safety-critical systems should be aware of the potential for problems in the software so they can anticipate and do their own research on things. This way, they can make an educated decision about whether or not they want to go through with it. That all being said, it's hard to prevent accidents (because by their very nature, they are not foreseen or intentional), but if they are things that occur because of something we can control, we must take every action to prevent these things from happening.
0 Comments
Leave a Reply. |
AuthorJulianna Yee. Archives
March 2018
Categories |