A Systems Engineering Review of the Mars One & M.I.T. Debate 
by: Oscar Mathews, F-18 Systems Engineer, NAVAIR
If you are a true space nerd/geek, then you may know by now that Sydney Do and Andrew Owens, the two highly credentialed students at M.I.T. (mega street cred!) took to the debate stage at the 18th Annual Mars Society convention Aug 16th, squaring off against CEO Bas Lansdorp of commercial space non-profit Mars One and Chief Engineer Barry Finger of Paragon Space Development Corporation. The purpose? To debate a seemingly simple question, “Is the Mars One Mission Feasible?”…the MIT stance premised and supported by their team’s report: “An Independent Assessment of the Technical Feasibility of the Mars One Mission Plan,” funded through a NASA research grant supporting efforts to advance space logistics. (The PhD students are completing research through the MIT Space Logistics Project http://strategic.mit.edu/, otherwise known as MLOG, run by Prof. Oliver de Weck).
Now I welcome MIT’s interest, yet it would seem awkward, to many experts in engineering professions, to debate feasibility outcomes without a discussion (however cursory) of risk acceptance. That is to say…does the question, “Does feasibility have a purely subjective answer, without risk?,” make any sense in our real world often marked by personal and highly subjective measures of risk acceptance? After all…in life, what is really safe?
The answer I will attempt to highlight, supporting the nascent Mars One one-way mission strategy, is that the answer of feasibility, despite attempts made to objectify it, remains inextricably linked to the assumptions taken to arrive, seemingly carefully, at the resultant subjective conclusion, which in all cases depend on total risk level acceptance, as well as other supporting data. So let us analyze risk level acceptance, supporting technical data by both sides, and various unaddressed lines of reasoning in the debate.
To begin, if you haven’t seen the debate I highly encourage you watch it at the Mars Society YouTube channel here:
Based on the paper here:
Which led to the very predictable media frenzy, all similar to the (surprisingly tame) TechTimes review below:
MIT outlines why the Mars One mission will fail
The Mars One mission that hopes to put colonists on Mars by 2024 will fail, at least according to MIT researchers…
And more breathless media confirmation bias post-debate from esteemed news sources here:
Mars One Is Still Completely Full of Shit
After watching a two hour debate on the feasibility of the Mars One mission last night, I think I finally understand…
Mars One Torn To Shreds In MIT Debate
Photo credit: It's not looking good for Mars One's proposed colony. Bryan Versteeg/Mars One. Whatever your views on…
Two MIT students just schooled a company trying to send people to Mars
YouTube/Mars One Two MIT engineering students just faced off with a private company that wants to send people on a one…
Mars One (and done?)
In 2012, the "Mars One" project, led by a Dutch nonprofit, announced plans to establish the first human colony on the…
All very similar articles (cough). Media around the globe have announced Mars One’s latest imminent demise following its resounding defeat in the open field of debate, err, battle (…debattle?), but the careful viewer and reader may perhaps glean something of an opposite opinion.
Dr. Z’s opening and explanation for the value of open and honest debate is timeless advice, though some would argue it seems a tragic review of the sad state of unaccountable/unscientific mass-media’s role in research. The debate should never have arrived to the point it did in the public eye (before peer-review). I believe most published scientists would agree.
The Mars One acknowledgement that it should have engaged earlier with the MIT students was politely received, and yet one could wonder how a start-up like Mars One could engage all schools without such esteemed pedigree in the thousands email’s “TO:” line that they receive weekly. To provide exclusive access to data for one entity otherwise seems a violation of principle until Mars One’s data are ready for publication from aerospace contractors to all. Of course, Mars One was also busy actively funding the ECLSS study through Paragon and finalizing publishing through ITAR for it from before end of March through July, while continuing work with Paragon at even this very moment on a Mars One Space-suit Design Study which will be integrated into the ECLSS Phase II. The Phase II study will hopefully address the real power and weight challenges needing optimization in the habitat.
And yet again it should be mentioned, if Mars One could manage it from a personnel perspective, could university students partnering with the Dutch company research as MIT has (with appropriate levels of review) for mere pennies on the dollar? Follow-through may be problematic with students, but we expect professors to guide and mentor aspiring scientists, and provide initial sanity checks on research. More on this later. We press on!
It seemed clear during the debate, that common language used was starkly different in each side’s response. It may be instructive to align expected common language for a debate’s technical review.
While keeping this in mind, we have to understand risk and modeling and simulation (M&S) basic concepts both to understand the report MIT produced, and also how MIT utterly failed to prove their premise in the debate by failing to address risk.
Let’s have that short discussion to frame the concept of risk and plans (otherwise known as schedules) in engineering, with additional notes on the not-yet-famously-known or even dreaded “Tin Triangle of Cost, Schedule, and Performance!” (choose two…or so the joke goes)
Wait, but what is RISK? When is a “plan”…a real “PLAN”?
Risk and Modeling and Simulation 101 (M&S) basic concepts
RISK (from the Defense Acquisition Guidebook [DAG])
Now that we have common language, let’s take a trip down memory lane to bygone eras of exploration:
Vikings! -> Britannia
Columbus! -> American West Indies
Jamestown! -> America
Apollo! -> Superpower flex
In each of these cases, risk was accepted at somewhat progressively lower relative levels as a consequence of an increase in both information and technical feasibility, combined with a strong human desire to explore. Information was, and still is, a commodity, which means it can be increased with resources and time and devotion to the Scientific Method. Information was (and is) power and it became valuable to the early explorers — in terms of finding arable lands and/or plunder in the case of the Vikings — in terms of knowledge concerning the true estimate of Earth’s diameter and the existence of an entire continent in the case of devout Columbus — or in terms of appropriate ways to grow native American crops and secure clean drinking water in the case of Jamestown (to feed colonists and prevent dysentery); of course, it was the culmination of all these foundational explorations that led today’s Informational Superpower, the United States, to risk national treasure and pride against the Soviets of old to recover a lost lead in space and explore the very nearest of neighbors, our Moon.
Era definition and risk acceptance
Let’s take another look at our bygone eras of exploration in terms of historical context and the particular era’s risk level acceptance.
Vikings, pretty much raided and pillaged to their heart’s content from Denmark, Norway and Sweden from AD 700 until about AD 1100 when they were incorporated into England and Scotland. As the raiding and pillaging type, one can predict quite correctly, Vikings had a large appetite for risk. One can imagine the adventure and thrill of crossing expansive waters in open-air longboats, exploring vast stretches of our planet. It was this risk acceptance that enabled Vikings to establish a presence in North America, Iceland, Greenland, Scotland, England, France, rounding Hispania and proceeding to Turkey via the Mediterranean Sea and on to the distant Arab empires of Mesopotamia. Many Vikings were lost in these early adventures and yet they have secured for themselves an enduring legacy of brute-force exploration.
The Spanish Empire of 1492 harbored a far different type of ruthless zeal for exploration than the Vikings. Queen Isabella sold her jewels against the (accurate) advice of her court Astronomers and Naval Navigators to fund Columbus’ risky journey westward. It was the unified nature of Spain’s religious homogeneity that permitted, in large part, that such a faith to be placed in such a small group of humans, and yet — the rewards for such a small wager were incalculable. Its effects on Spanish dominance and historical significance lasted centuries, with cultural effects continuing well to this day.
Jamestown showed us an analogue for private expansion and exploration funded by private and individual investors through The London Company and the Royal Virginia Charter Companies. Plans were laid and executed without requisite understanding of local weather conditions, without known sources and/or treatment of water — without adequate management of disease; the Jamestown settlers lacked suitable crops for the soil and climate — and yet they went. Nearly all died in the first few years, but a hardy few persevered. It is for this reason, primarily, you are now reading this article in English, and not for example, Queen Isabella’s castellano-accented Spanish.
Finally, we come to the pinnacle of human exploration: the Apollo Moon Program. A distance record for human exploration that has remained unchallenged since 1972, despite its beginning in a US space program that began in a frightful mess. As Vanguard and Redstone hardware was exploding on the launchpad, the first astronaut-candidates watched with (I can only guess) bemused skepticism as they strove to be selected into Project Mercury, Project Gemini and eventually Project Apollo.
Each project married human and hardware elements in the National Strategic Space Plan to win the Space Race from the Soviets. Shuttle astronauts came soon thereafter. Not one program was ever stopped in the face of disasters. Apollo 1, Challenger in 1986, Columbia in 2003 (though it was retired early)…countless “near-misses” and/or non-astronaut deaths in support of the space programs…none of these stopped the indomitable spirit of NASA. Risk was, and is, clearly part of the business.
PREVIOUS POSTS BY MIT STUDENTS
- Open Letter
MIT Strategic Engineering
An Open Letter on the Mars One Analysis by MIT Researchers 10/10/2014 Last week, at the International Astronautical…
- MIT students Reddit AMA:
MIT Students REDDIT AMA:
10/10/14 We are the authors of the MIT Mars One feasibility analysis — AMA!
“We would like to maintain an open line of communication with the Mars One organization to further develop our analysis. The purpose of our work was to better understand how to make Mars exploration feasible, not provide a list of problems and walk away. We look forward to refining our assumptions and examining alternate mission plans that may make colonizing Mars more sustainable. As detailed in the Open Letter linked to in the description of this thread, it is our hope that our analysis can provide guidance for future technology development and analysis. For example, we are investigating a study into what potential benefits could be gained at a systems level from development of new technologies such as in-situ spares manufacturing. This is very much an ongoing research topic, and we are excited to see where it goes.”
TL;DR DEBATE SUMMARY
BEGINNING NEGATIVE STATEMENT (MIT STUDENTS)
- Andrew: statement — claims their group is strategic level working group
- Andrew: What does it take to get us from here to Mars?
- Andrew: (assuming plan as a static component) “Is the Mars One mission plan feasible?”
- Andrew: “What major challenges do we need to solve, what technologies do we need to develop ?— that is what we will discuss tonight”
- Sydney: The Mars One plan is $6 billion and then $4 billion afterward every 26 months. (as opposed to the $4.5 billion estimated for the first crew/launch of 15 FH from the MIT paper…though increasing over time).
- Sydney: Feasibility — is scope attainable within Cost and Schedule?
- Sydney: Scope argument: allows for existing technology. EDL like Curiosity and ISS similar ECLSS.
- Sydney: 1960 — $102 billion in 8 years versus $6 billion 12 years versus Virgin Galactic (…why not versus SpaceX? Valued at $10 billion and has capability.)
- Sydney: /sarcastic Rover and autonomous robot — for 26 months, no maintenance. (…We have cars driving themselves and computers passing 3 wise men tests.)
- Note: Mars2020 robot is not “less capable” than the currently unknown Mars One robots — that is an opinion.
- Sydney: Far right Landed Mass (low-blow and I’ll explain why. Either Sydney doesn’t understand that Paragon’s heavy ECLSS result is a product of unoptimized weight/complexity and cost components, or he doesn’t care. Not sure which based on his facial expressions and tone.)
- Andrew: 3 months between resupply missions
- Andrew: Mars One resupply baseline spares reliability increase 15%, need testing time (…did those numbers come from their spares model and simulation? So they’re proving data from an untested M&S product known to be research-grade code?)
- Andrew: Fully manufacture using only local resources to enable sustainability — one of the major conclusions of their IAF report.
- Sydney: Time element — unprecedented launch cadence?
- Sydney: Biggest Concern is unsustainability over time. All these challenges — are they possible to solve? Sydney says: No.
BEGINNING POSITIVE STATEMENT (MARS ONE/PARAGON)
- Bas: Beginnings of NASA and Apollo mission being accomplished in 8 years.
- Bas: Both Apollo and Mars One in pre-Phase A development
- Bas: Goal of Mars One is to SEND HUMANS TO MARS
- Bas: Plan is not etched in stone — and flexibility is a healthy response to new data
- Bas: Mars One should have embraced MIT email solicitation
- Bas: shows Apollo design and changes
- Bas: Mars One has limited resources and would rather choose to organize a mission without money than with impossible tech development
- Bas: Paragon ECLSS aerospace suppliers etc and many more reports left to be done
- Chief Engineer Barry Finger goes on to list his extensive background and presents a series of points on his experience and historical basis for ECLSS designs
COMBINED NEGATIVE/POSITIVE DISCUSSION (MIT STUDENTS & MARS ONE/PARAGON)
- Sydney: Hard sell to public as a finalized plan, so how can you know feasibility?? Bas: Mars One is using aerospace companies and their timelines and estimates. Plan can be updated as new information comes in (like mass?) Need landed mass study next…needs more money to afford Lockheed Martin study. “Crazy” enough project that positive surprises like billionaire writing check might happen. Customer scenario then — commodity purchase afterwards
- Andrew: Given what you know now — what would you change in the plan, and what would you do with $6 billion? Bas: We need the landed mass study first. Andrew: Wrong landing altitude — Curiosity landed at -4 km (-4.4 km).
- Bas: Candidates followed 24/7? NO Candidates always in control of what they send to Earth. No Marines. Investors see the business case. Not worried about cost overruns. They worry about Tech risk (failures) — fortunately we can insure copy of InSIGHT/PHOENIX or that someone beats Mars One to it…Bas says $36 billion on return on investment. Space funeral company. Other new business cases.
- Sydney: the study did not find it feasible is fundamentally unsustainable because costs grow over time. When you launch people, if you do, you have to support them for their whole life?
- Barry: I agreed with the students for a while, unless we change the resupply/re-crew rate. Then I understood. That doesn’t change the business case.
- Bas: right we can change all that. ROI can be much higher than 8 if early and lower than 8 later. Share price lower then higher. Producing things are Mars locally. RFP’s schools, universities and how you build a habitat out of local materials, glass sintering, 3D printing in sand.
- Andrew: feasibility of ISRU in 20 years depends on funding and schedule. Lot of uncertainty and cost to developing capability. Tech versus new system. When Godard fired in 1926 why didn’t humans go to the moon?
- Barry: not equivalent comparison of Goddard to Apollo. That is not where we are. There is not fundamental physical process that we don’t understand for ECLSS. ISS is ISS-optimized. Completely different case in applied engineering. It’s what engineers are good at. When Bas gets the funding you get derived requirements. You can build upon what has already been done. But don’t wait for next new technology.
- Sydney: we presented a set of slides, it’s a house of cards, if any of them don’t work, the whole things fails. Can’t do feasibility of program if concept is still changing. (So why did you do feasibility?) Andrew: a lot of excellent arguments for going to Mars, what do we need to solve? We all agree Mars is a valuable goal…sooner rather than later? Why Mars One plans (as currently presented) are infeasible…what changes in cost, scope, schedule can be done to make it feasible? Concepts, are not a plan, WBS, funding…etc
- Bas: did Kennedy have a plan (meaning detailed plan)? No. (MIT nod heads yes). Bas agrees that timeline might slip…is it a failure if you land in 2032? He doesn’t think so. Mars systems can be tested on Earth to test crews and systems. Suits biggest challenges. Rovers may have seperate tasks…basic things are possible, but tactical decesions have not been made.
- Dr. Z: transition and question from German ESA engineer Herman Berkoltz (Hermes-ISS). When does Mars One intend to propose a plan in credible detail? (Bas: Does he have $50 million? Raising money is difficult and unpredictable. 6 weeks- end of year). Needed on Mars in addition to mission return, is construction equipment, repair, food and ISRU resources and testing on Earth and Space. Plans to test infrastructure on Mars in advance to Earth? Outpost Alpha testing for technical solutions.
- Andrew: technical analysis in June Bloomberg statement (referring Paragon report in pre-ITAR). Our initial study has been a good study (not NASA) but investors do. Timeline is not what investor use. Know your investors/audience on BloombergTV.
- Sydney: 1962 and before Apollo had lots of technical studies and you still don’t have a technical plan. Bas: better than NASA (applause). Andrew: Bashing NASA does not prove feasibility of Mars One (longer applause).
- Sydney: unsustainable resupply growth. Slowing doesn’t stop over time. How willing are you to descope to return or to orbital?
- Bas: NASA has done great things (light applause)…funding limits, not feasibility. Funding is difficult. Build on capabilities and produce things locally over time. Look at 3D printing over past 30 years…can we produce feedstock for 3D printer? It’s an energy question — plastics metal etc.
Questions from floor:
- To Bas: First few crews and sustainability. After that, couldn’t govt and others pay crews to do work on surface. Yes, but it’s too long-term. Columbus business case was to find route to India. Instead, he founded, the richest country in the world…with humans can be less-complex…if people get excited again about humans on Mars. Mars artifacts will be invaluable. Thousands of business cases.
- From Andrew: Sustained interest — Apollo interest peaked and ISS interest is flat. How do you deal with it?
- From Bas: we become co-owners of IP and sustain through new invention. ie. self-sustaining through industry, not media. Although media experts confirm that there will be interest for decades.
- Woman: crop production between O2 and CO2 for crops and fire risk.
- Sydney: testing claim for 50 m2 and found needed 4 times for 1/3 of the people under optimistic conditions. Admitted it’s been known in the ECLSS community for decades you need to decouple atmospheres and it was intermediate and not a final conclusion. Study found that growing crops was more expensive than just bringing food. Andrew says you are CO2 short if you are O2 long. Issues known for a long time in biological community. Sydney — you need more crop area…
- Barry: to the question — media picked up on it as major result— simple assumptions that were wrong in running the model. Tremendous knowledge base out there — over 680 published papers — don’t plant all at once, combust the inedible parts and those things were not included in your model. You made assumptions, and maybe you didn’t get feedback from Mars One, but you make those assumptions and you get nonsensical results. I know you guys know that it is nonsensical and it’s unfortunate because it takes away from the message.
- Dr. Z…so you are CO2 short if you grow enough food — how can you be CO2 short on Mars?? (laughter) Andrew— it’s a balance — it’s a —they’re all connected. Dr Z.: so just open a window? (more laughter)
- Man: Spare accumulating over time — how do you refurbish the existing spare parts? Are you planning on using TRL for each subsytem and letting people know where they are and testing in vacuum chamber? Bas: yes absolutely, lessons learned and TRL will be used/incorporated. We are not there yet, but of course that needs to be done. Can hire a tech team from suppliers or hiring a 3rd party for program management. Barry you have to rely on that. Systems are unpredictable — saline solution example. Human ingenuity supreme.
- Andrew: my dissertation topic is refurbishment level — details etc. Some critical piece may close loop. It costs money and takes time. Limited life components like filters…range of things. Expendables.
- Dr. Z: humans there — does that affect risk? Andrew — we assumed the mechanics were perfect mechanics after 10 years of training.
- Dr. Z: Sharp annoying question? — Man: in business model what are the hurdles in dealing with media versus sponsors? Bas: waiting to later time for media but we have some cultural investors.
- Sydney: descope to make it feasible — maybe an orbital mission?
- Bas: Crews could enjoy company of 8 weighed against possibility of not creating ISRU printing capability. Orbital mission and Moon mission is harder to fund.
- Man: crew deaths with investors? How do you recover from crew deaths? Bas: Part of the risks. Lay it all the table. Complete transparency and unmanned mission insured — first manned mission risk reduced through previous unmanned cargo missions. Apollo missions landed without testing — any residual risk has to be factored into investment. Chance of failure for two crews simultaneously is very small.
- Andrew: ISRU mission fails to produce resources, unmanned mission fails, when would you delay human mission? Bas: ISRU will be tested on first unmanned mission. Test on Earth. Rover will have capabilities as well, not sure on details yet, but will test soil for water.
- Is it feasible? No.
- Excellent case on why it’s possible, but the debate question is: was it feasible?
- You have a concrete plan presented on website. What needs to change to make it feasible?
- See no evidence from the other side.
MARS ONE SUMMARY
- (Is it feasible? Yes.)
- Mars One will not execute the website plan verbatim —
- Plan is to get humans on Mars…it is executable — no new inventions — leading aerospace companies agree with Mars One.
- Confident we can execute the plan in terms of goals, schedule and funding to what is presented.
END OF DEBATE.
MY RESPONSE TO THE MIT RESULTS (INCLUDING THE DEBATE):
First of all, I’d like to thank the interest of my peers at MIT for having attempted an initial Space Logistics simulation of the Mars One mission plan. I know Mars One invites and seeks more than just one institution to accomplish analyses of its strategic plan as more details can be ascertained at appropriate programmatic milestones to enable robust optimization of solutions and correction of any major errors before they affect program schedule.
Unfortunately for the MIT students, their study did not produce any novel information in the field of space logistics, except perhaps debugging data to help improve their research-grade MATLAB code, so that one day it can (we should laud this) be published open-source to the world.
Just to be clear, I am a huge fan of MIT. Though I never felt inclined to apply in the past, I am sure I would be just as proud of having been from MIT as they clearly are, had I attended. One of my friends at SpaceX graduated from MIT, and several others I know (either through HiSEAS crew activities or through other Mars One candidates who work there as researchers) really love the institution. This commentary is of course not about MIT. This commentary is not even about the MIT students. This commentary is intended to be a somewhat formal review, a peer review, that is based partly on currently-known science for modeling and simulations and risk management in systems engineering and based partly on my analysis of the quality of the MIT student’s methods.
The opinions of the author are his alone and do not constitute formal endorsement of any position by either the US Navy or Mars One, but Andrew and Sydney should take note if they intend to accomplish more research on this topic in the future.
FACTORS ADDRESSED BY MIT DURING DEBATE & IN REPORT THAT REMAIN ACTIVELY RESEARCHED
- 3D ISRU
- Plant growth
- Sustainment and Closure of open systems
Luckily, plant growth is assumed away easily enough by MIT in their report, so they wisely chose not to even mention the biological aspect in their formal debate presentation until an audience member asked about the pressurant gas problem leading to crew death on day 68 (well, day something…45? Listening to the FISO telecon, it’s pretty revealing there is no confidence in any of the gas-transfer code even after abstract presentation in Canada at IAF).
It’s always cringe-worthy when a Chief Engineer of a major corporation (leading ECLSS design) calls your report results nonsensical, due to nonsensical input conditions. The simple reason the results were completely misleading to the general public is because the model was neither appropriately Verified nor Validated. Had the MIT students followed NASA’s own Modeling and Simulation standard, NASA-STD-7009, their model would have either been fixed, or the abstract would have been reserved from conference proceedings until the model could be proven against a standard result.
Smartly, the duo decided to change the main focus of their conclusion to sparing resupply.
MIT FACTORS THAT WERE ERRORS AND/OR UNADDRESSED AND/OR RUIN MIT’s POSITION IN DEBATE
Because the debate performance was based on the MIT report, linked just about everywhere naysayers and skeptics lurk on the internet, it is fair (in my opinion) to begin to dismantle the debate points by dismantling the report points. It will take me less time to address the major flaws than it took Dr. Z to introduce the panel. Here we go!
http://www.journals.elsevier.com/acta-astronautica/editorial-board/, please take note. I hope these issues can be addressed/resolved in any candidate publication prior to the MIT Report passing peer-review.
- MIT Unaddressed: NASA-STD-7009, STANDARD FOR MODELS AND SIMULATIONS (https://standards.nasa.gov/documents/detail/3315599)
.. NASA Technical Standards Program ..
NASA Technical Standards Program
NASA Technical Standards Programstandards.nasa.gov
NASA is a government agency. As a systems engineer in the Navy (another government agency…and one that has produced a significant fraction of astronauts, from Pax River alone, I might add), I wondered: “surely NASA has a risk assessment standard for modeling and simulation?” One quick Google search later I had before me the glorious NASA-STD-7009. It is a beauty to behold and to read. Clear…concise…happily unlike most government guides.
In it, MIT students Andrew and Sydney would have found clear requirements for the exact thing they are trying to convince us of. Not the debate question, Is the Mars One Mission Plan Feasible?, but the implied debate premise…can Sydney Do and Andrew Owens use their report model and conclusions to construct and defend a debate position against (or even for) feasibility?
The answer is: No, not yet.
The reason for this is, the MATLAB research-grade code forming the basis of their results, which not many (if any) outside their research group have seen — this code which has been heavily modified already from when the initial results were produced until the Feb FISO telecon and doubtless again before final peer review for Acta Astronautica— has not been mentioned in the report based on its results. Not anywhere in the report do the MIT students tell us what specific code they used…not anywhere do they speak about the assumed initial conditions…not anywhere in the report is the necessary discussion on sensitivity of the code to small changes in initial conditions. If all this was accomplished, well and good. Where is the discussion? Where is the discussion on model Validation and Verification?
NASA-STD-7009, page 19 “Verification, Validation, and Uncertainty Quantification”
NASA-STD-7009, page 23 “Assessing the Credibility of M&S Results”
If basic M&S 101 best practices were not accomplished…well then…why should anyone trust the MIT report model or deliciously fancy debate slides based on that model? If they were, why weren’t they included in the report and/or debate?
Not only is the model likely to give us nonsensical outputs with known data, but the MIT students admitted in their report that they tried their best to assume gaps in the mission plan from Mars One’s high-level strategy on the website. So with nonsensical inputs of unknown data, nonsensical outputs are essentially guaranteed. More insidious yet, the outputs of these unverified models may even have the semblance of veracity. This is what Paragon Chief Engineer Barry Finger was trying to illustrate here:
Why not withhold a report like this until after June’s internal release of the Paragon pre-Phase A ECLSS report to Mars One? It took until the end of July for ITAR release and internal review, but that one report could have prevented most of the heartache felt keenly by Mars One on the fledgling commercial space side, due to fundamental negligence on the part of the methods of the MIT students. Why not do a SpaceX analysis…they’ve been in business far longer than Mars One, since they began in 2002, versus 2012. The MIT Report (and Debate) is really a learning opportunity to highlight a Modeling and Simulation case study in what not to do.
Where was Dr. de Weck during this critical juncture? I expect my advisor at ODU, Dr. Ash, to rein me in and mentor me when I venture too far in a particular direction.
- MIT Error: Future Operations in Space Telecon in Feb 2015 and bulkhead solving the atmospheres from report page 11 (decoupling atmospheres)
Now, in the NASA FISO telecon linked below, one can clearly hear the students being informed that their assumption of a single atmosphere model was wrong and that established human habitat design science had settled the separation of atmospheres question decades ago. Decoupling the atmospheres, which the students eventually did on page 11 of their report with a wall, ie. bulkhead, means humidity and gas pressures and compositions can be positively controlled. It’s not rocket science.
NASA FISO Presentation: An Independent Assessment of the Technical Feasibility of the Mars One…
©NASA/Sydney Do/MIT Now available is this weeks NASA Future In-Space Operations (FISO) telecon material from the…
- MIT Error: software correction and new results >40% error
Unperturbed, the MIT students continue their FISO presentation and claim a surprising ~40% error in crew death upon subsequent model runs (45 day versus 68 day deaths). There was no audible reaction from the NASA advisors and funders, but clearly MIT cannot be the sole source of Modeling and Simulation expertise in the United States, can it? Who can independently verify MIT to prevent such gross errors from being reported as fact? APL? ODU? There are many organizations with robust M&S expertise that can provide help to organizations like MIT, still apparently struggling with defining and executing their own M&S research. It should be a collaborative environment, but MIT cannot be allowed to dominate the community solely on the basis of massive street cred and a renowned acronym alone. There has to be sizzle to their M&S steak!
- MIT Logical Flaw: Definition of “plan” and the Scientific Method (or Engineering Method) as it relates to M&S:
The last point of contention (from the debate only) was the MIT student’s attempt to define unilaterally what a plan means.
Webster’s dictionary defines a “plan” as:
: a set of actions that have been thought of as a way to do or achieve something
: something that a person intends to do
: a detailed agreement for telephone service, medical care, insurance, etc.
Now, something you “intend to do,” does not make it set in stone. It is famously said in the military that no plan survives contact with the enemy (in this case, funding) but in essence the definition of a plan implies that the plan or ideas formulating the plan could change and/or improve.
The Scientific Method is a formal system for updating planned investigations of formal thought into the world around us. The Scientific Method is not static, because humans and our world are not static, and so it is graphically represented as a connected and/or circular process. Systems engineering, and all engineering generally, relies on the Scientific Method to iteratively improve how we incorporate new data and discoveries. This was a major point of contention during the debate and it beggars belief that MIT would reduce themselves to assuming a strategic plan on a website must only be interpreted as a tactical, detailed, and static entity. The whole ordeal was difficult to parse in person and recollections flitted through my mind of when President Clinton tried redefining the verb “is” during impeachment proceedings against him. It began to mar a reasonable MIT debate performance, despite the flawed report conclusions their position was based on.
ADMITTED MISTAKES IN THE REPORT THAT FORMS BASIS OF DEBATE
The first main problem I had with the MIT report (and the following debate) is that one cannot invent data one does not have access to. The fallacy of unwarranted assumption is committed when the conclusion of an argument is based on a premise (implicit or explicit) that is false or unwarranted. An assumption is also unwarranted when it is true but does not apply in the given context. The students chose to bridge gaps in their knowledge from the Mars One website based on nothing more than their (considered) opinions.
Design of Experiments would have had a field-day with MIT’s experimental setup. Presenting results of a sample size of one…it begs the question: Why should we trust your model, MIT? MIT seems to believe it has the advantage of credibility, but I insist. Why should we trust your model? You have presented no evidence to support the implied claim that your model is infallible. This however, despite Atmospheric gas control code that was heavily corrected after publication and before FISO telecon, leading to 40% change in results for crew death. You may as well have published an article titled, “Astronaut Crew Holds Breath — All Hands Die Aboard ISS.” The methods in the report from top to bottom were nonsensical and to obtusely disregard local and accesible resources, even as a test case, is absurd.
But (after having talked to Paragon for an hour or so one day before the debate, to privately address bioregeneratiion flaws in the report) MIT’s main result is now Unconstrained Sparing Requirements! This result only occurs if Mars One continues to send crews or those humans never develop ISRU 3D printing. Even if there were such ISRU systems (there are some working on Earth), the claim is they would have to be TRL 5+ and tested in an operationally relevant environment (Mars) for MIT to deem them feasible.
Apart from the illogical position of claiming feasibility only from landed components, as if that was the standard of feasibility, the MIT students fail to realize that local ISRU reliance emerges in human systems. Below is a tongue-in-cheek graphic I created showing notional Jamestown Colony spares deliveries until present day in terms of Axe-head imports (full disclosure: I estimated amounts and shamelessly copied the MIT slide):
If I’ve done it correctly, it should be humorously apparent that America is not breathlessly waiting for it’s next import of Axe Heads from the long-defunct London Company sub-contractors (oh, yes, that London Company was nothing more than a Project Management Co. and fundraising venture)!
ELEMENTS THAT MUST BE ADDRESSED, CHANGED, OR INCLUDED IN ORIGINAL PAPER TO PASS (MY) PEER-REVIEW & CONCLUSION
- Validation and Verification of both atmospheric module, but more importantly (knowing the state of the art), of the sparing module used in the model.
- Use of standardized results to establish M&S Credibility Analysis for MIT position.
4.4 Verification, Validation, and Uncertainty Quantification
The definitions of verification and validation used in this standard are those of the M&S community.
For verification, the responsible party performs the following:
Req. 4.4.1 — Shall document any verification techniques used and any domain of verification (e.g., the conditions under which verification was conducted).
Req. 4.4.2 — Shall document any numerical error estimates (e.g., numerical approximations, insufficient discretization, insufficient iterative convergence, finite-precision arithmetic) for the results of the computational model.
Req. 4.4.3 — Shall document the verification status of (conceptual, mathematical, and computational) models.
For validation, the responsible party performs the following:
Req. 4.4.4 — Shall document any techniques used to validate the M&S for its intended use, including the experimental design and analysis, and the domain of validation.
Req. 4.4.5 — Shall document any validation metrics, referents, and data sets used for model validation.
Req. 4.4.6 — Shall document any studies conducted and results of model validation.
For uncertainty quantification, the responsible party performs the following:
Req. 4.4.7 — Shall document any uncertainty quantification processes used for
a. The referent data.
b. The input data.
c. The M&S results.
d. The propagation of uncertainties.
e. The quantities derived from M&S results.
Req. 4.4.8 — Shall document any quantified uncertainties, both physical and numerical, for
a. The referent data.
b. The input data.
c. The M&S results.
d. The propagation of uncertainties.
e. The quantities derived from M&S results.
Req. 4.4.9 — Shall document the extent and results of any sensitivity analyses performed with the M&S.
Recommendations for verification, validation, and uncertainty quantification of M&S are that the responsible party should document the following:
a. Any aspects of M&S that have not been verified.
b. Any aspects of M&S that have not been validated.
c. If any significant physical processes, effects, scenarios, or environments have not been considered in the uncertainty quantification analysis.
4.5 Identification and Use of Recommended Practices
Recommended Practices for the following should be identified:
a. Input data verification and validation.
b. A quantified method of tracking adherence to Recommended Practices.
c. The purposes and objectives for the M&S and their pedigree.
d. Verification and validation processes for the M&S.
e. Uncertainty quantification methods for the M&S
f. Understanding of the disciplines incorporated in the M&S.
g. Analyzing and interpreting the M&S results including documentation of inference guidelines and statistical processes used.
h. Recognizing and capturing the need for any changes or improvements in the M&S.
i. Reporting procedures for results.
j. Identify best practices for user interface design to constrain the operation of the simulation to within its limits of operations.
4.7 Assessing the Credibility of M&S Results
In order to ensure that NASA decision makers are informed about the credibility of M&S results in terms of a common process and a common language, M&S results used in critical decisions will be assessed using the CAS in accordance with the requirements in this section.
The operational concept of the credibility assessment scale is that the presentation of any results from M&S to a decision maker include (1) the best estimate of the results, (2) a statement on the uncertainty in the results, (3) the evaluation of the results on the credibility assessment scale, and (4) any explicit caveats that accompany the results. (An example of such a caveat would be use of the model in violation of its assumptions.) The decision maker then makes his/her own assessment of credibility based upon all four pieces of information in the context of the decision at hand. Just to emphasize this fundamental point, the credibility assessment scale does not purport to measure credibility; rather, it assesses the M&S results, and the rigor of the processes used to produce them, against key factors that affect the credibility judgment. The fundamental premise of this approach is that as a general rule, the more rigorous the key processes used for generating the M&S results, the greater the credibility of the M&S results, all else (including the estimated uncertainty) being equal.
The reporting requirements in the following section include the results of the CAS along with several other contributing factors that are either standalone data (the uncertainty statement) or significant caveats.
The details of the CAS are provided in Appendix B. In summary, the scale is comprised of eight factors. Each factor is divided into levels ranging from 0 to 4, with level definitions describing the evidence necessary for achieving that particular level. Level 0 corresponds to insufficient or no evidence, i.e., the M&S produces results, but there is insufficient evidence to warrant even level 1 assessment on that factor. A single, summary score on the CAS is determined by the minimum of the eight factor scores, which produces a single number between 0 and 4.
This standard itself levies no requirements with respect to what levels must be achieved (the sufficiency threshold levels — see Appendix B.5), merely that the achieved levels must be determined and reported.
The requirements for the assessment of the credibility of the M&S results that support critical decisions are that the responsible party performs the following:
Req. 4.7.1 — Shall assess the credibility of M&S results for each of the eight factors in the CAS described in Appendices B.2 and B.3.
Req. 4.7.2 — Shall justify and document the credibility assessment for each of the eight factors referenced in Req. 4.7.1.
Req. 4.7.3 — Shall perform the roll-up to an overall score according to the process described in Appendix B.4.
A recommendation is to gain additional insight into the credibility of M&S results by applying the process in Appendix B.5 to calculate and report any gaps between the achieved scores and the program/project-defined threshold scores for each of the factors.
That is what MIT has to do to pass my peer-review. It is what I would expect for my own work if I was a NASA Fellow and nothing less. It is what I would have expected before public presentation of an idea or model, to ensure the continued good name of the institutions we represent (and our own reputations).
MIT and the world need to be reminded that Mars One is an audacious Project Management Start-up/Non-profit organization.
Below one can see where Mars One is progressing in terms of schedule. Notice how much of the Mars One Plan remains yet to be accomplished on the right hand side.
My final position —
An attempt to answer feasibility without risk is like never leaving the cave of Plato’s Allegory of the Cave. Is the world outside so scary some would not risk it all for the dream of exploring the unknown?
The risks of Mars
No one would argue that our culture hasn't become extremely, obsessively risk averse. Advancing technologies or space…
I’ll finish by sharing a few thoughtful quotes:
“We must dissent from the indifference. We must dissent from the apathy. We must dissent from the fear” — Thurgood Marshall
“It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, because there is no effort without error and shortcoming; but who does actually strive to do the deeds; who knows great enthusiasms, the great devotions; who spends himself in a worthy cause; who at the best knows in the end the triumph of high achievement, and who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who neither know victory nor defeat.”― Theodore Roosevelt
People seem to forget feasibility and risk are intimately connected concepts. One cannot speak of one without the other.
“Spaceflight is a dangerous endeavor, and we hope that any entity (public or private) that sends people into space would take the necessary precautions to ensure that their crews stay safe. However, risks must be taken in order to explore the solar system, and perhaps something that we need to do as a civilization is have a discussion on how much risk we are willing to accept to become an interplanetary species. It’s a difficult balance to strike.”
-MIT MLOG program REDDIT AMA
Tortoise versus Hare — is it winning the race, or crossing the finish line?
Mars One Candidate