As of January 12, 2017, there are almost 3,440 confirmed exoplanets, with 576 planets orbiting in multi-planet systems.[i] These numbers will continue to grow as more Kepler planet candidates are officially confirmed. The confirmed planets have been found using five main methods: radial velocity, transit, direct imaging, gravitational microlensing, and astrometry. Radial velocity uses the Doppler effect to find exoplanets. When a planet orbits a star, it is not only the planet that is moving around the star, but the star is also circling around the center of mass of the planetary system. Viewed from Earth, as the star circles the center of mass it moves closer and farther away, changing the color of light of that star as observed from Earth. The color shifts between blue and red, blue indicating that it is moving closer (more compact wavelength of visible light), and red that it is moving away from us (more stretched out waves of light). This blue-ing and reddening of the wavelengths is called Doppler shift. (It is also responsible for the changing frequency of sound, when an ambulance siren is coming towards you and moving away from you). The transit method of finding exoplanets is by far the most successful of the methods, finding most of the total number of confirmed exoplanets. If a solar system happens to be lined up with Earth, we can see the planet “eclipse” their parent star. These eclipses or transits cause a dip in the star’s observed brightness. This phenomenon is what the Kepler Space Observatory looks for. Direct imaging is a challenging method; it’s exactly what it sounds like, a direct image of the stars. Scientists do this by removing the glare from the planet’s host stars. Gravitational micro-lensing occurs when light from a distant star is focused by a moving star or planet. Particularly massive stars and planets gravity can focus the light of other stars, causing a temporary increase in brightness. The last method, astrometry, searches for hard to find minute changes. When planets orbit a star, the star is not stationary; it gets tugged around its center of mass. Without being able to see the planet itself the star will appear to move extremely tiny amounts relative to other stars as well as having periodic Doppler shift. [ii]
While all these methods are remarkable achievements in finding exoplanets, they have their limitations, the most obvious one being that these methods are not very sensitive to smaller planets. We’ve had the most success finding planets that are large and close to their stars. These “hot Jupiters” would never be considered Earth-like under our current definition. The technology is improving, but we still have more progress to make before Earth-sized [maybe even any Earth-like] planets around Sun-like stars are detectable. Transits, and lensing have the additional disadvantage of being localized occurrences. They aren’t predictable so we have to look are large areas of the sky over a long period to catch one as it is happening. The final limitation is that our region of search is only in a small region in our home galaxy, the Milky Way. We are far from getting a representative sampling of exoplanets.
Overfitting is when we take noise for signal, something that is easy to do in our unfortunate Earth-bound situation. Without a clear idea of what is meaningful, things that are don’t reflect the reality of the situation can creep into our conclusions. Earth, right now, encompasses our signal and our noise. We are in the thick of it, not seeing the forest for the trees. Mainstream science could have under fitted the habitable zone, choosing only the size of the planet and the liquid water as the necessary components. Planets must be terrestrial and have water; we will not be looking for gaseous aliens on Jupiter or Neptune.[iii] The problem is that there is no verification. We sit, in front of a board with a scatterplot filled with numerous disorganized points, hopelessly tasked with discovering the connections, if any, there are between the Earth’s many qualities and the formation of life.
We have so much information about Earth, but the questions of habitability from a scientific perspective are recent. Many fields study the Earth, geology, biology, astronomy, and earth and atmospheric science, habitability is an intersection of these disciplines. It makes sense that there are respective beliefs about exactly which dots should connect. The general trend seems to be that scientists whose specialties are the intricacies of the Earth think that earth-like conditions are rare, while astronomers and physicists, who have more of an outward focus, believe that Earths are common.[iv] Each field will have their own well of information to draw from affecting how they approach the massive number of dots on the board.
Mathematicians bypass the messy business of picking the important bits of Earth’s history, and rely on the numbers. The logic behind it is that the sheer size of the universe would allow for an Earth-like planet to be out there. No matter the calculated prior beliefs or the assigned probability given to Earth-like planets, the vastness gives many trials for a success. Even if there is a one-in-a-billion chance of an Earth-like planet occurring, we would be guaranteed by the size of the universe for many Earth-like planets.[v] While this is a hopeful calculation for those who wish for some company on the cosmic scale, it sidesteps the problems of making connections and developing the relationships between the data points.
The nature of truth in astronomical observational data is a sticky concept itself. Is the truth reflected in our understanding or is it a goal out there, the universe as it exists, waiting for us to observe enough data to really “get” it? In astrophysicist, Ray P Norris’s paper, Discovering the Unexpected in Astronomical Survey Data, he admits that the traditional Popperian scientific method of hypothesis testing doesn’t apply to many astronomical surveys.[vi] Astronomy relies on an alternative discovery-based scientific method, because many hypotheses and theories are unfalsifiable. Their scientific process values exploration and data collection over specific hypothesis testing. Norris encourages future astronomy surveys and data collection to account for both known unknowns, and the unknown unknowns. In other words, discovering of a new type of object (quasars), and discovering entirely new kinds of phenomenon (dark energy, the expanding universe).[vii] While we would like to think each new discovery and even discrepancies between our theoretical and our sample data strides us towards a complete picture of the universe, I’m not sure if they are strides, or just dainty steps. If we are consistently discovering new objects, and if the field’s majority of the knowledge is unexpected it seems we take for granted that we really are close to the truth.
Our approximation of the truth is of course limited by our understanding of the universe as is and our data collection methods. The information we collect from the universe is our most important tool in framing the truth. Despite possible errors, and compounded by the fact that we are always finding something new, the data is the best reflection of the universe as it is (whether it behaves as we expect it to or not). When the data aligns with our simulations then we will have arrived at the truth. According to philosopher Alex Schmid, the truth of simulation comes from three distinct types of truth theory: coherence, correspondence, and consensus. He explains:
- A simulation model is true if and only if it corresponds to a matter of fact in reality.
- A simulation model is true if and only if it is rationally acceptable under ideal or optimal conditions.
- A simulation model is true if and only if it is a member of a coherent system of believes[sic].
Schmid goes on to unpack these terms in his own paper, but even just listing them we can see how they occur in our current scientific climate. These formulations go hand in hand with the popular conception of the Popperian scientific method. In addition to being “true,” simulations must also meet two more criteria: validity and accuracy. Validity is based on how well a model jibes with our expectations and fulfillment of its purpose, and accuracy refers to how it well it reflects the collected data.
In our discussion of math and approaches to the probability problem, we have become separated from what the probability and our estimation means. Sometimes when we focus on the numbers and the equations the truth of the data can be lost in the shuffle of us trying to numerically capture reality. Our understanding of the universe doesn’t affect how the universe itself operates. Statistician William Briggs says on his blog,
“What is happening now is either being influenced by what has not yet occurred, or probability is physically real in the same way that mass or charge is. Yet there is, of course, zero evidence, and anyway its absurd, to think probability is material.” (cite)
Probability is not physical force, determining the data we receive. We create probability by measuring and counting. Predictive power is not something that we have mastered. While meteorological forecasts have gotten significantly better over the years, due to advances in computing power, there are still areas (both man-made and in nature) where we are hopeless at making accurate prediction: the stock market, earthquakes, pandemics, and the economy.[viii] These are phenomena that are affected by an unknown number of factors. And if these undeniably earthly activities are beyond our reach now, then how can we even begin to both identify and mathematically represent the factors involved in our whole planet?
In exoplanet research, our predictions usually are about the composition of the planets. Before we found our first exoplanets, we theorized the exo-solar systems would look like ours: terrestrial planets clustered towards the Sun and the gaseous larger planets farther out. Our solar system was the only “model” we had to base our theory on. However, once we found the first planet, we had to update our model. We had only theorized about massive Jupiter-like planets extremely close to their stars.[ix] Now since these planets show up so often in our exoplanet surveys, we readily accept them as a normal part of solar system development. A recent exoplanet prediction comes from a statistical survey of 6 years’ worth of data from a ground based telescope. [x]Daisuke Suzuki, one of the researchers, claims that using previous results that “We conclude that Neptune-mass planets in these outer orbits are about 10 times more common than Jupiter-mass planets in Jupiter-like orbits.” This statement needs qualifying. Few planets have been found using the microlensing technique, and the paper only has a sample size of 30 planets. In addition, microlensing has the capability to survey a larger area of our galaxy. Does “10 times more common” mean in this larger search area or our entire galaxy?
Ward and Brownlee, authors of Rare Earth: Why Intelligent Life is Uncommon in the Universe, created their own formula that reflects their belief of the rarity of earth life. Instead of assigning lower probabilities to get a low number of intelligent life, they increase the number of factors that they deem necessary. The formula comes at the end of the book, the result of them building a case against intelligent life’s commonness. Looking at their Rare-Earth Equation gives a hint of how the authors developed their prior probabilities.
N= N* x fp x fpm x ne x ng x fi x fc x fl x fm x fj x fme
Where:
N* = stars in the Milky Way galaxy
Fp= fraction of stars with planets
Fpm=fraction of metal rich planets
Ne=planets in a star’s habitable zone
Ng = starts in a galactic habitable zone
Fi-fraction of habitable planets where life does arise
Fc= fraction of planets where complex metazoans arise
Fl= percentage of lifetime of a planet that is marked by presence of complex metazoans
Fm = fraction of planets with a large moon
fj= fraction of solar systems with Jupiter size planets
Fme = fraction of planets with critically low number of mass extinction events
Compared to the Drake Equation, the Rare-Earth equation has already set itself to reflect their beliefs. Looking at the factors that they deem most important, we are invited to see how they develop the argument of their book. Without reading the entire case, we can understand what conditions (to them) make a planet Earth-like for the authors factors that Frank Drake didn’t put into his own equation. When we lay out equations out like this, we peek into a Bayesian world.
Frank Drake, Sara Seager, Peter Ward and Donald Brownlee have all developed equations to find the number of alien worlds out there. Drake was concerned with those who could contact us, while Seager, Ward, and Brownlee focused more on life in any form (intelligent or not). Although the Copernican Principle forms the foundation of their accepted factors, it allows for personal probabilities. The Drake equation makes use of our Copernican principle to establish estimates for the 2 of the 7 factors: the average rate of star formation per year in the Milky Way, and the fraction of starts that have planets. The rest of the factors, instead of turning outwards for physical evidence, we turn inward and assess our own probabilities. These personal probabilities can skew the result, in this case the number gets closer to 0 or more than 1. The number reflects our probabilities.
If I’m a skeptic of all this alien business, and I chose .001 for fl, fi, and fc, with 100 years as the lifetime of a situation and .125 as fraction of earthlike planets per solar system, I get .00000625 as the number of communicative civilizations within the Milky Way. Conversely, taking the same for value of L, and Ne, and increasing the fl, fi, and fc to 1, we get 6.25, a comparatively large number. Bayesian analysis is the reason why we can have a book that assures us intelligent life is an uncommon occurrence in the universe, and another that swears that the probability is 100 percent!
In a frequentist world, on the other hand, the truth can be found through perfect sampling. Our errors in estimating probabilities result from errors in our sampling methods. I do not believe that the frequentist approach works when it comes to probabilities of the existence of celestial objects (in our case habitable “earth-like worlds”). Measurements of exoplanets taken from our small blue orb can hardly represent planets in every corner of the universe. In fact, there’s a limit to how much of the universe we can observe. In the real world, more measurements and sampling would not lead to the objective truth.[xi] The useful data, the information needed to arrive at an accurate understanding could be different from the data being collected.
Astronomy is not a field of probabilities. Studies of our region result in updates to our understanding of the universe. If there is a possibility that we are wrong, we don’t concern ourselves with the hypothetically correct model. We operate in a permanent world that is subject to change. The frequentist school of thought is suitable to this oxymoronic world.
The beautiful thing about probabilities is that we entertain wrongness. We flirt with the idea of instability— a disconcerting thought if the instability is affecting the foundation of our knowledge. Unexpected results and wrong hypotheses drive the process of discovery. With our current understanding, we don’t allow for the possibility that we can be wrong at the most basic level. In admitting our unsureness of the phenomenon that we are trying to predict, we make ourselves less wrong. If we fully committed to always or never (100% or 0%) lines of thinking, we leave out the things that we consider impossible that are fully possible.[xii] I do not like being wrong, but I understand it as a necessary component of learning. Instead of thinking about elsewhere Earths as Probability 1 or near zero, we can assign a range of probabilities, based upon constant updates to our knowledge. Space is an area where we can’t be certain. There are few direct measurements and everything comes with a degree of uncertainty. In the absence of direct measurements, we check statistically for false positives and aberrant data. The continuous learning and updating of our beliefs of is the hallmark of Bayesian statistics. Complacency is the enemy of prediction. Professional forecaster Nate Silver hails the benefits of the Bayesian process.[xiii]This attitude would freshen up the unchanging models of rare Earth discussions. It’s not of matter if the factors undoubtedly contribute (or not) to what makes the Earth Earth-like, but reflected in how much we think these factors are important to the development of our planet.
These formulas provide a streamlined map of Ward and Brownlee’s train of thought, while also forcing us to reflect upon our own beliefs. If Drake looked at the Rare Earth equation, he would either think of his own comparatively sparse equations or question why Ward and Brownlee believe that so many factors are crucial to the development of life on Earth. Either way, the result is the same: both examples encourage research into the information that is different than the author’s own which the author can either pass up or incorporate into a new and hopefully improved formula.
Sara Seager looked at Drake’s whiteboard with his 8-term formula, and created her own, which of course, picks out factors that are most important to her. She determined that the number of planets with detectable life forms was more important than the intelligence of the life forms. Seager’s “revised” Drake Equation showed the strength of her beliefs by creating her own equation. [xiv]She lamented that it would be a shame to miss out on extraterrestrial life because we weren’t looking for the right signs. She recognizes the Bayesian aspects to her own equation:
I carefully crafted the last term of this equation so one could actually add more information in. Does life produce detectable signature? Are there systematic effects that rule out some bio signature gases being detected in some planets? Can we not find the signature for technical reasons?[xv]
In this mishmash of personal beliefs, it becomes hard to pick out the factors that contribute to my own beliefs. Being in a field where we have a lot information confuses the process of determining relevant information. It also brings the issue of testability. Bayesian hypothesis testing can be fine-tuned with experiments, but without hard proof, the pertinent factors are difficult to verify. Do Ward and Brownlee justify including Jupiter, extinction events, and large moons, in their equation? Do the any of the above scientists’ factors reflect reality? We cannot be sure. This field is particularly sticky because of the dearth of information. All formulas are valid until proven otherwise. Of course, ridiculous claims that the number of planets is somehow related to the current number of rhinoceroses still alive, we can be skeptical of. There’s no clear signal and a lot of noise, in Silver’s words.
As an aspiring statistician, I think data overload seems like the best problem to have. A small sample will not hold up to rigorous statistical testing. From either approach, frequentist or Bayesian, more data is better. However, the amount of data we must work through hampers us from making meaningful connections. Earth’s history stretches back to the formation of our solar system, and maybe even our universe. I am looking a timeline that starts at the Big Bang and goes through everything, even the minutiae of Earth’s history. This includes the Big Bang, the formation of early stars, the planetary disk that formed the Sun and our solar system, and then every single event on Earth. While the details of early Earth history seem hazy and far off, there is a possibility that they affected the conditions on Earth. Earth is more than just what happens on its surface. At this stage, decoding the Earth-like qualities from the extensive history of the planet is nearly impossible. Once we find another planet that is truly Earth-like we can cross- reference that planet’s history to find out what qualities we share.
Taking this attitude towards our understanding of the universe we still make guesses about the distributions of the subjects we are trying to study to accommodate for unknown unknowns. In probability and statistics, phenomena are thought to follow either a known or unknown probability distribution. This means that the probabilities of certain assigned values look like functions that change with certain parameters. Visualizing the data, can help clarify if the data aligns with one of the known distributions.[xvi] If it doesn’t we have to question “the sum of our understanding of the universe.” Usually astronomers develop characteristic distributions based on the data collected and compare it to distribution based on simulated models. While models, by definition, attempt to capture something accurately, they cannot be perfect, and are vulnerable to inaccuracies in reflecting truth. In this case, our simulation tests our theoretical understanding. It’s fascinating because here we have a direct comparison between the “truth” in the data and our truth approximate (the “truth” as we expect it to be).
WM Briggs, a statistician and professor at Columbia University believes that traditional frequentists and Bayesians are trapped in parameter estimation and over-certainty. Probability is based on data and nothing else. The universe is not beholden to us to behave according to the models. When our model coincides generally with the universe (meaning the data that we have measured), it doesn’t mean that we’ve captured reality, but only that we’ve captured the data. If our models fit the data, it is not a predictive tool of what else it out there. It only is based on the data that we have acquired, not that data-in-waiting. Like a financial firm on Wall-Street, a model’s past performance is not indicative of future success.
Briggs encourages a predictive approach to probability, but leaves us on our own to figure out how to apply it to a field such as astronomy. His finds issue with over certainty and unnecessary quantification. But in astronomy, aren’t we trying to “quantify the unquantifiable?” Understanding for us comes in the form of mathematically consistent equations, and anything less feels a little too intangible. Once we have created a formula, we feel like we truly “know” something. Probability, for Briggs, is a “soft” form of prediction. Good predictions require many things but a solid understanding of the phenomena in question is essential
“Knowledge of cause and essence is at the base of every probability.” “Probability is always a measure of our understanding — not of what exists what we can know of what exists.” These quotes represent astronomy’s fight to convert the field to Briggs’ preferred brand of predicative statistics. Probability does not spring from the head of the data fully formed like Athena from Zeus. Even simple probabilities, according to Briggs, are conditional and based on a multitude of factors. But predictive statistics rely on conditionals, even when we are not sure what the conditionals are. What would the conditionals be earth-like planets? What are our qualifications for our astronomical predictions? It is not enough to state the probability, without fully grounding it in the reality of the data. We collected the data from the real world, so our interpretation must be returned to that context.
If we are consistently discovering new factors, and not testing hypotheses, then how much knowledge of “cause and essence” do we really have? We find probabilities in the hopes that they predict the next data sets that we collect. When we say that 1.08% (percentage derived by dividing 1 for Earth/ 8 the number of planets in the solar system) of exoplanets are habitable, it is not a statement about the population of planets outside our solar system, but rather stating a fact about the data. I think that this separation is crucial. It’s the same sort of qualification that we see in poll results. If a poll reports that 17% of people believed that Movie A was better than Movie B, that it doesn’t tell me that 17% of everybody shared the same opinion. Of course, our hope is that our conclusions about the data are representative of whatever our greater population is. But there is a lot to unpack in a simple statement. Who are the people polled? Where were they polled? How were the questions asked? These are all factors that affect how representative of the results of the poll are. In this same line, any statements we make about exoplanets, only reflects the small area of the of the Milky Way we can observe. We don’t need random sampling, we need representative sampling. Unlike polling, we don’t have the luxury of being able to model our population accurately. Since we don’t know all the conditionals that affect the development of planets, we cannot develop a representative sample. In addition, we can only collect data as far as the technology will allow. Not only that are we limited by that, we are limited by uncertainty in our measurements themselves. Not all the “planets” we find are actual planets, there is rigorous testing to determine false positives.[xvii]
Subjective beliefs have found their place into science. Although Bayesian statistical techniques acknowledge the subjective determination of a prior, we have the goal of objective knowledge in science. When we say something is scientifically proven, we want undeniably accepted fact. When we look out into space, it’s almost like we are trolling the darkest parts of the ocean. We kind of know what sorts of creatures live down there, but many of them are weirder than we’ve ever expected and often we find new ones before we even know what we are looking for. The establishment of a prior probability requires knowledge, in our case it is ignorance.[xviii] This is not to say that we know nothing, rather that there is a lot out there that we don’t know or can’t predict. This lack of knowledge is enough of a problem where we need to look at the data being produced in studies for unexpected results.
The process of confirmation/consensus/observation as a pathway to objective truth becomes obsolete in astronomy. We are literally talking about everything that there is, and we only know a small fraction of it. There is no consensus/confirmation truth of the whole universe simply through the fact that we can’t observe the whole universe. It seems to me that for this Frequentist statistics and data reduction methods have been applied to astronomy without considering the philosophical implications. We’ve already made several assumptions about how to interpret our observations. And while there are error bounds that exist in our modeling, there are no error bounds for our base assumptions. How comfortably can make claims about the world around us, when our understanding isn’t falsifiable?
[i] Exoplanets Data Explorer | Exoplanets – Form Search | DataIR.” Accessed September 18, 2016. http://datair.soe.ucsc.edu/experiment/exoplanet/search/exact.
[ii] “5 Ways to Find a Planet.” Accessed September 9, 2016 https://exoplanets.nasa.gov/interactable/11/.
“________641639209_orig.jpg (2239×1476).” Accessed February 17, 2017.
[iii] Dartnell, Lewis. Astrobiology: Exploring Life in the Universe. 1st ed. New York, New York: Rosen Publishing Group, 2011.
[iv] Scharf, Caleb. The Copernicus Complex. First. New York, New York: Scientific American, 2014.
[v] Aczel, Amir. Probability One: Why There Must Be Intelligent Life in the Universe. Harcourt, Brace, & Company, 1998.
[vi] Norris, Ray P. “Discovering the Unexpected in Astronomical Survey Data.” Publications of the Astronomical Society of Australia 34 (2017). doi:10.1017/pasa.2016.63.
[vii] Norris, Ray P. “Discovering the Unexpected in Astronomical Survey Data.” Publications of the Astronomical Society of Australia 34 (2017). doi:10.1017/pasa.2016.63.
[viii] Silver, Nate. The Signal and the Noise
[ix] Mirror Earth
[x] Suzuki, D., D. P. Bennett, T. Sumi, I. A. Bond, L. A. Rogers, F. Abe, Y. Asakura, et al. “The Exoplanet Mass-Ratio Function from the MOA-II Survey: Discovery of a Break and Likely Peak at a Neptune Mass.” The Astrophysical Journal 833, no. 2 (2016): 145. doi:10.3847/1538-4357/833/2/145.
[xi] Silver Nate
[xii] Silver, Nate
[xiii] Silver,Nate
[xv] Powell, Devin, Astrobiology Magazine | September 4, and 2013 06:18pm ET. “The Drake Equation Revisited: Interview with Planet Hunter Sara Seager.” Space.com. Accessed February 24, 2017. http://www.space.com/22648-drake-equation-alien-life-seager.html.
[xvi] “Probability Distribution Functions.” Accessed March 18, 2017. http://astrostatistics.psu.edu/STScI/prob_distributions2011STScI_4.pdf.
[xvii] “Statistical Analysis of Viable Exoplanet Candidates.” Accessed September 9, 2016. http://arxiv.org/pdf/1608.00620v2.pdf.
[xviii] Romeijn, Jan-Willem. “Philosophy of Statistics.” In The Stanford Encyclopedia of Philosophy, edited by Edward N. Zalta, Spring 2017. Metaphysics Research Lab, Stanford University, 2017. https://plato.stanford.edu/archives/spr2017/entries/statistics/.