Tag: Samuel Arbesman

Half Life: The Decay of Knowledge and What to Do About It

Reading Time: 15 minutes

Understanding the concept of a half-life will change what you read and how you invest your time. It will explain why our careers are increasingly specialized and offer a look into how we can compete more effectively in a very crowded world.

The Basics

A half-life is the time taken for something to halve its quantity. The term is most often used in the context of radioactive decay, which occurs when unstable atomic particles lose energy. Twenty-nine elements are known to be capable of undergoing this process. Information also has a half-life, as do drugs, marketing campaigns, and all sorts of other things. We see the concept in any area where the quantity or strength of something decreases over time.

Radioactive decay is random, and measured half-lives are based on the most probable rate. We know that a nucleus will decay at some point; we just cannot predict when. It could be anywhere between instantaneous and the total age of the universe. Although scientists have defined half-lives for different elements, the exact rate is completely random.

Half-lives of elements vary tremendously. For example, carbon takes millions of years to decay; that’s why it is stable enough to be a component of the bodies of living organisms. Different isotopes of the same element can also have different half-lives.

Three main types of nuclear decay have been identified: alpha, beta, and gamma. Alpha decay occurs when a nucleus splits into two parts: a helium nucleus and the remainder of the original nucleus. Beta decay occurs when a neutron in the nucleus of an element changes into a proton. The result is that it turns into a different element, such as when potassium decays into calcium. Beta decay also releases a neutrino — a particle with virtually no mass. If a nucleus emits radiation without experiencing a change in its composition, it is subject to gamma decay. Gamma radiation contains an enormous amount of energy.

The Discovery of Half-Lives

The discovery of half-lives (and alpha and beta radiation) is credited to Ernest Rutherford, one of the most influential physicists of his time. Rutherford was at the forefront of this major discovery when he worked with physicist Joseph John Thompson on complementary experiments leading to the discovery of electrons. Rutherford recognized the potential of what he was observing and began researching radioactivity. Two years later, he identified the distinction between alpha and beta rays. This led to his discovery of half-lives, when he noticed that samples of radioactive materials took the same amount of time to decay by half. By 1902, Rutherford and his collaborators had a coherent theory of radioactive decay (which they called “atomic disintegration”). They demonstrated that radioactive decay enabled one element to turn into another — research which would earn Rutherford a Nobel Prize. A year later, he spotted the missing piece in the work of the chemist Paul Villard and named the third type of radiation gamma.

Half-lives are based on probabilistic thinking. If the half-life of an element is seven days, it is most probable that half of the atoms will have decayed in that time. For a large number of atoms, we can expect half-lives to be fairly consistent. It’s important to note that radioactive decay is based on the element itself, not the quantity of it. By contrast, in other situations, the half-life may vary depending on the amount of material. For example, the half-life of a chemical someone ingests might depend on the quantity.

In biology, a half-life is the time taken for a substance to lose half its effects. The most obvious instance is drugs; the half-life is the time it takes for their effect to halve, or for half of the substance to leave the body. The half-life of caffeine is around 6 hours, but (as with most biological half-lives) numerous factors can alter that number. People with compromised liver function or certain genes will take longer to metabolize caffeine. Consumption of grapefruit juice has been shown in some studies to slow caffeine metabolism. It takes around 24 hours for a dose of caffeine to fully leave the body.

The half-lives of drugs vary from a few seconds to several weeks. To complicate matters, biological half-lives vary for different parts of the body. Lead has a half-life of around a month in the blood, but a decade in bone. Plutonium in bone has a half-life of a century — more than double the time for the liver.

Marketers refer to the half-life of a campaign — the time taken to receive half the total responses. Unsurprisingly, this time varies among media. A paper catalog may have a half-life of about three weeks, whereas a tweet might have a half-life of a few minutes. Calculating this time is important for establishing how frequently a message should be sent.

“Every day that we read the news we have the possibility of being confronted with a fact about our world that is wildly different from what we thought we knew.”

— Samuel Arbesman

The Half-Life of Facts

In The Half-Life of Facts: Why Everything We Know Has an Expiration Date, Samuel Arbesman (see our Knowledge Project interview) posits that facts decay over time until they are no longer facts or perhaps no longer complete. According to Arbesman, information has a predictable half-life: the time taken for half of it to be replaced or disproved. Over time, one group of facts replaces another. As our tools and knowledge become more advanced, we can discover more — sometimes new things that contradict what we thought we knew, sometimes nuances about old things. Sometimes we discover a whole area that we didn’t know about.

The rate of these discoveries varies. Our body of engineering knowledge changes more slowly, for example, than does our body of psychological knowledge.

Arbesman studied the nature of facts. The field was born in 1947, when mathematician Derek J. de Solla Price was arranging a set of philosophical books on his shelf. Price noted something surprising: the sizes of the books fit an exponential curve. His curiosity piqued, he began to see whether the same curve applied to science as a whole. Price established that the quantity of scientific data available was doubling every 15 years. This meant that some of the information had to be rendered obsolete with time.

Scientometrics shows us that facts are always changing, and much of what we know is (or soon will be) incorrect. Indeed, much of the available published research, however often it is cited, has never been reproduced and cannot be considered true. In a controversial paper entitled “Why Most Published Research Findings Are False,” John Ioannides covers the rampant nature of poor science. Many researchers are incentivized to find results that will please those giving them funding. Intense competition makes it essential to find new information, even if it is found in a dubious manner. Yet we all have a tendency to turn a blind eye when beliefs we hold dear are disproved and to pay attention only to information confirming our existing opinions.

As an example, Arbesman points to the number of chromosomes in a human cell. Up until 1965, 48 was the accepted number that medical students were taught. (In 1953, it had been declared an established fact by a leading cytologist). Yet in 1956, two researchers, Joe Hin Tjio and Albert Levan, made a bold assertion. They declared the true number to be 46. During their research, Tjio and Levan could never find the number of chromosomes they expected. Discussing the problem with their peers, they discovered they were not alone. Plenty of other researchers found themselves two chromosomes short of the expected 48. Many researchers even abandoned their work because of this perceived error. But Tjio and Levan were right (for now, anyway). Although an extra two chromosomes seems like a minor mistake, we don’t know the opportunity costs of the time researchers invested in faulty hypotheses or the value of the work that was abandoned. It was an emperor’s-new-clothes situation, and anyone counting 46 chromosomes assumed they were the ones making the error.

As Arbesman puts it, facts change incessantly. Many of us have seen the ironic (in hindsight) doctor-endorsed cigarette ads from the past. A glance at a newspaper will doubtless reveal that meat or butter or sugar has gone from deadly to saintly, or vice versa. We forget that laughable, erroneous beliefs people once held are not necessarily any different from those we now hold. The people who believed that the earth was the center of the universe, or that some animals appeared out of nowhere or that the earth was flat, were not stupid. They just believed facts that have since decayed. Arbesman gives the example of a dermatology test that had the same question two years running, with a different answer each time. This is unsurprising considering the speed at which our world is changing.

As Arbesman points out, in the last century the world’s population has swelled from 2 billion to 7 billion, we have taken on space travel, and we have altered the very definition of science.

Our world seems to be in constant flux. With our knowledge changing all the time, even the most informed people can barely keep up. All this change may seem random and overwhelming (Dinosaurs have feathers? When did that happen?), but it turns out there is actually order within the shifting noise. This order is regular and systematic and is one that can be described by science and mathematics.

The order Arbesman describes mimics the decay of radioactive elements. Whenever new information is discovered, we can be sure it will break down and be proved wrong at some point. As with a radioactive atom, we don’t know precisely when that will happen, but we know it will occur at some point.

If we zoom out and look at a particular body of knowledge, the random decay becomes orderly. Through probabilistic thinking, we can predict the half-life of a group of facts with the same certainty with which we can predict the half-life of a radioactive atom. The problem is that we rarely consider the half-life of information. Many people assume that whatever they learned in school remains true years or decades later. Medical students who learned in university that cells have 48 chromosomes would not learn later in life that this is wrong unless they made an effort to do so.

OK, so we know that our knowledge will decay. What do we do with this information? Arbesman says,

… simply knowing that knowledge changes like this isn’t enough. We would end up going a little crazy as we frantically tried to keep up with the ever changing facts around us, forever living on some sort of informational treadmill. But it doesn’t have to be this way because there are patterns. Facts change in regular and mathematically understandable ways. And only by knowing the pattern of our knowledge evolution can we be better prepared for its change.

Recent initiatives have sought to calculate the half-life of an academic paper. Ironically, academic journals have largely neglected research into how people use them and how best to fund the efforts of researchers. Research by Philip Davis shows the time taken for a paper to receive half of its total downloads. Davis’s results are compelling. While most forms of media have a half-life measured in days or even hours, 97 percent of academic papers have a half-life longer than a year. Engineering papers have a slightly shorter half-life than other fields of research, with double the average (6 percent) having a half-life of under a year. This makes sense considering what we looked at earlier in this post. Health and medical publications have the shortest overall half-life: two to three years. Physics, mathematics, and humanities publications have the longest half-lives: two to four years.

The Half-Life of Secrets

According to Peter Swire, writing in “The Declining Half-Life of Secrets,” the half-life of secrets (by which Swire generally means classified information) is shrinking. In the past, a government secret could be kept for over 25 years. Nowadays, hacks and leaks have shrunk that time considerably. Swire writes:

During the Cold War, the United States developed the basic classification system that exists today. Under Executive Order 13526, an executive agency must declassify its documents after 25 years unless an exception applies, with stricter rules if documents stay classified for 50 years or longer. These time frames are significant, showing a basic mind-set of keeping secrets for a time measured in decades.

Swire notes that there are three main causes: “the continuing effects of Moore’s Law — or the idea that computing power doubles every two years, the sociology of information technologists, and the different source and methods for signals intelligence today compared with the Cold War.” One factor is that spreading leaked information is easier than ever. In the past, it was often difficult to get information published. Newspapers feared legal repercussions if they shared classified information. Anyone can now release secret information, often anonymously, as with WikiLeaks. Governments cannot as easily rely on media gatekeepers to cover up leaks.

Rapid changes in technology or geopolitics often reduce the value of classified information, so the value of some, but not all, classified information also has a half-life. Sometimes it’s days or weeks, and sometimes it’s years. For some secrets, it’s not worth investing the massive amount of computer time that would be needed to break them because by the time you crack the code, the information you wanted to know might have expired.

(As an aside, if you were to invert the problem of all these credit card and SSN leaks, you might conclude that reducing the value of possessing this information would be more effective than spending money to secure it.)

“Our policy (at Facebook) is literally to hire as many talented engineers as we can find. The whole limit in the system is that there are not enough people who are trained and have these skills today.”

— Mark Zuckerberg

The Half-Lives of Careers and Business Models

The issue with information having a half-life should be obvious. Many fields depend on individuals with specialized knowledge, learned through study or experience or both. But what if those individuals are failing to keep up with changes and clinging to outdated facts? What if your doctor is offering advice that has been rendered obsolete since they finished medical school? What if your own degree or qualifications are actually useless? These are real problems, and knowing about half-lives will help you make yourself more adaptable.

While figures for the half-lives of most knowledge-based careers are hard to find, we do know the half-life of an engineering career. A century ago, it would take 35 years for half of what an engineer learned when earning their degree to be disproved or replaced. By the 1960s, that time span shrank to a mere decade. Today that figure is probably even lower.

In 1966 paper entitled “The Dollars and Sense of Continuing Education,” Thomas Jones calculated the effort that would be required for an engineer to stay up to date, assuming a 10-year half-life. According to Jones, an engineer would need to devote at least five hours per week, 48 weeks a year, to stay up to date with new advancements. A typical degree requires about 4800 hours of work. Within 10 years, the information learned during 2400 of those hours would be obsolete. The five-hour figure does not include the time necessary to revise forgotten information that is still relevant. A 40-year career as an engineer would require 9600 hours of independent study.

Keep in mind that Jones made his calculations in the 1960s. Modern estimates place the half-life of an engineering degree at between 2.5 and 5 years, requiring between 10 and 20 hours of study per week. Welcome to the treadmill, where you have to run faster and faster so that you don’t fall behind.

Unsurprisingly, putting in this kind of time is simply impossible for most people. The result is an ever-shrinking length of a typical engineer’s career and a bias towards hiring recent graduates. A partial escape from this time-consuming treadmill that offers little progress is to recognize the continuous need for learning. If you agree with that, it becomes easier to place time and emphasis on developing heuristics and systems to foster learning. The faster the pace of knowledge change, the more valuable the skill of learning becomes.

A study by PayScale found that the median age of workers in most successful technology companies is substantially lower than that of other industries. Of 32 companies, just six had a median worker age above 35, despite the average across all workers being just over 42. Eight of the top companies had a median worker age of 30 or below — 28 for Facebook, 29 for Google, and 26 for Epic Games. The upshot is that salaries are high for those who can stay current while gaining years of experience.

In a similar vein, business models have ever shrinking half-lives. The nature of capitalism is that you have to be better last year than you were this year — not to gain market share but to maintain what you already have. If you want to get ahead, you need asymmetry; otherwise, you get lost in trench warfare. How long would it take for half of Uber or Facebook’s business models to be irrelevant? It’s hard to imagine it being more than a couple of years or even months.

In The Business Model Innovation Factory: How to Stay Relevant When the World Is Changing, Saul Kaplan highlights the changing half-lives of business models. In the past, models could last for generations. The majority of CEOs oversaw a single business for their entire careers. Business schools taught little about agility or pivoting. Kaplan writes:

During the industrial era once the basic rules for how a company creates, delivers, and captures value were established[,] they became etched in stone, fortified by functional silos, and sustained by reinforcing company cultures. All of a company’s DNA, energy, and resources were focused on scaling the business model and beating back competition attempting to do a better job executing the same business model. Companies with nearly identical business models slugged it out for market share within well-defined industry sectors.

[…]

Those days are over. The industrial era is not coming back. The half-life of a business model is declining. Business models just don’t last as long as they used to. In the twenty-first century business leaders are unlikely to manage a single business for an entire career. Business leaders are unlikely to hand down their businesses to the next generation of leaders with the same business model they inherited from the generation before.

The Burden of Knowledge

The flip side of a half-life is the time it takes to double something. A useful guideline to calculate the time it takes for something to double is to divide 70 by the rate of growth. This formula isn’t perfect, but it gives a good indication. Known as the Rule of 70, it applies only to exponential growth when the relative growth rate remains consistent, such as with compound interest.

The higher the rate of growth, the shorter the doubling time. For example, if the population of a city is increasing by 2 percent per year, we divide 70 by 2 to get a doubling time of 35 years. The rule of 70 is a useful heuristic; population growth of 2 percent might seem low, but your perspective might change when you consider that the city’s population could double in just 35 years. The Rule of 70 can also be used to calculate the time for an investment to double in value; for example, $100 at 7 percent compound interest will double in just a decade and quadruple in 20 years. The average newborn baby doubles its birth weight in under four months. The average doubling time for a tumor is also four months.

We can see how information changes in the figures for how long it takes for a body of knowledge to double in size. The figures quoted by Arbesman (drawn from Little Science, Big Science … and Beyond by Derek J. de Solla Price) are compelling, including:

  • Time for the number of entries in a dictionary of national biographies to double: 100 years
  • Time for the number of universities to double: 50 years
  • Time for the number of known chemical compounds to double: 15 years
  • Time for the number of known asteroids to double: 10 years

Arbesman also gives figures for the time taken for the available knowledge in a particular field to double, including:

  • Medicine: 87 years
  • Mathematics: 63 years
  • Chemistry: 35 years
  • Genetics: 32 years

The doubling of knowledge increases the learning load over time. As a body of knowledge doubles so does the cost of wrapping your head around what we already know. This cost is the burden of knowledge. To be the best in a general field today requires that you know more than the person who was the best only 20 years ago. Not only do you have to be better to be the best, but you also have to be better just to stay in the game.

The corollary is that because there is so much to know, we specialize in very niche areas. This makes it easier to grasp the existing body of facts, keep up to date on changes, and rise to the level of expert. The problem is that specializing also makes it easier to see the world through the narrow focus of your specialty, makes it harder to work with other people (as niches are often dominated by jargon), and makes you prone to overvalue the new and novel.

Conclusion

As we have seen, understanding how half-lives work has numerous practical applications, from determining when radioactive materials will become safe to figuring out effective drug dosages. Half-lives also show us that if we spend time learning something that changes quickly, we might be wasting our time. Like Alice in Wonderland — and a perfect example of the Red Queen Effect — we have to run faster and faster just to keep up with where we are. So if we want our knowledge to compound, we’ll need to focus on the invariant general principles.

***

Members can discuss this post on the Learning Community Forum.

Tagged: , , , , , ,

The Need for Biological Thinking to Solve Complex Problems

Reading Time: 6 minutes

“Biological thinking and physics thinking are distinct, and often complementary, approaches to the world, and ones that are appropriate for different kinds of systems.”

***

How should we think about complexity? Should we use a biological or physics system? The answer, of course, is that it depends. It’s important to have both tools available at your disposal.

These are the questions that Samuel Arbesman explores in his fascinating book Overcomplicated: Technology at the Limits of Comprehension.

[B]iological systems are generally more complicated than those in physics. In physics, the components are often identical—think of a system of nothing but gas particles, for example, or a single monolithic material, like a diamond. Beyond that, the types of interactions can often be uniform throughout an entire system, such as satellites orbiting a planet.

Biology is different and there is something meaningful to be learned from a biological approach to thinking.

In biology, there are a huge number of types of components, such as the diversity of proteins in a cell or the distinct types of tissues within a single creature; when studying, say, the mating behavior of blue whales, marine biologists may have to consider everything from their DNA to the temperature of the oceans. Not only is each component in a biological system distinctive, but it is also a lot harder to disentangle from the whole. For example, you can look at the nucleus of an amoeba and try to understand it on its own, but you generally need the rest of the organism to have a sense of how the nucleus fits into the operation of the amoeba, how it provides the core genetic information involved in the many functions of the entire cell.

Arbesman makes an interesting point here when it comes to how we should look at technology. As the interconnections and complexity of technology increases, it increasingly resembles a biological system rather than a physics one. There is another difference.

[B]iological systems are distinct from many physical systems in that they have a history. Living things evolve over time. While the objects of physics clearly do not emerge from thin air—astrophysicists even talk about the evolution of stars—biological systems are especially subject to evolutionary pressures; in fact, that is one of their defining features. The complicated structures of biology have the forms they do because of these complex historical paths, ones that have been affected by numerous factors over huge amounts of time. And often, because of the complex forms of living things, where any small change can create unexpected effects, the changes that have happened over time have been through tinkering: modifying a system in small ways to adapt to a new environment.

Biological systems are generally hacks that evolved to be good enough for a certain environment. They are far from pretty top-down designed systems. And to accommodate an ever-changing environment they are rarely the most optimal system on a mico-level, preferring to optimize for survival over any one particular attribute. And it’s not the survival of the individual that’s optimized, it’s the survival of the species.

Technologies can appear robust until they are confronted with some minor disturbance, causing a catastrophe. The same thing can happen to living things. For example, humans can adapt incredibly well to a large array of environments, but a tiny change in a person’s genome can cause dwarfism, and two copies of that mutation invariably cause death. We are of a different scale and material from a particle accelerator or a computer network, and yet these systems have profound similarities in their complexity and fragility.

Biological thinking, with a focus on details and diversity, is a necessary tool to deal with complexity.

The way biologists, particularly field biologists, study the massively complex diversity of organisms, taking into account their evolutionary trajectories, is therefore particularly appropriate for understanding our technologies. Field biologists often act as naturalists— collecting, recording, and cataloging what they find around them—but even more than that, when confronted with an enormously complex ecosystem, they don’t immediately try to understand it all in its totality. Instead, they recognize that they can study only a tiny part of such a system at a time, even if imperfectly. They’ll look at the interactions of a handful of species, for example, rather than examine the complete web of species within a single region. Field biologists are supremely aware of the assumptions they are making, and know they are looking at only a sliver of the complexity around them at any one moment.

[…]

When we’re dealing with different interacting levels of a system, seemingly minor details can rise to the top and become important to the system as a whole. We need “Field biologists” to catalog and study details and portions of our complex systems, including their failures and bugs. This kind of biological thinking not only leads to new insights, but might also be the primary way forward in a world of increasingly interconnected and incomprehensible technologies.

Waiting and observing isn’t enough.

Biologists will often be proactive, and inject the unexpected into a system to see how it reacts. For example, when biologists are trying to grow a specific type of bacteria, such as a variant that might produce a particular chemical, they will resort to a process known as mutagenesis. Mutagenesis is what it sounds like: actively trying to generate mutations, for example by irradiating the organisms or exposing them to toxic chemicals.

When systems are too complex for human understanding, often we need to insert randomness to discover the tolerances and limits of the system. One plus one doesn’t always equal two when you’re dealing with non-linear systems. For biologists, tinkering is the way to go.

As Stewart Brand noted about legacy systems, “Teasing a new function out of a legacy system is not done by command but by conducting a series of cautious experiments that with luck might converge toward the desired outcome.”

When Physics and Biology Meet

This doesn’t mean we should abandon the physics approach, searching for underlying regularities in complexity. The two systems complement one another rather than compete.

Arbesman recommends asking the following questions:

When attempting to understand a complex system, we must determine the proper resolution, or level of detail, at which to look at it. How fine-grained a level of detail are we focusing on? Do we focus on the individual enzyme molecules in a cell of a large organism, or do we focus on the organs and blood vessels? Do we focus on the binary signals winding their way through circuitry, or do we examine the overall shape and function of a computer program? At a larger scale, do we look at the general properties of a computer network, and ignore the individual machines and decisions that make up this structure?

When we need to abstract away a lot of the details we lean on physics thinking more. Think about it from an organizational perspective. The new employee at the lowest level is focused on the specific details of their job whereas the executive is focused on systems, strategy, culture, and flow — how things interact and reinforce one another. The details of the new employee’s job are lost on them.

We can’t use one system, whether biological or physics, exclusively. That’s a sure way to fragile thinking. Rather, we need to combine them.

In Cryptonomicon, a novel by Neal Stephenson, he makes exactly this point talking about the structure of the pantheon of Greek gods:

And yet there is something about the motley asymmetry of this pantheon that makes it more credible. Like the Periodic Table of the Elements or the family tree of the elementary particles, or just about any anatomical structure that you might pull up out of a cadaver, it has enough of a pattern to give our minds something to work on and yet an irregularity that indicates some kind of organic provenance—you have a sun god and a moon goddess, for example, which is all clean and symmetrical, and yet over here is Hera, who has no role whatsoever except to be a literal bitch goddess, and then there is Dionysus who isn’t even fully a god—he’s half human—but gets to be in the Pantheon anyway and sit on Olympus with the Gods, as if you went to the Supreme Court and found Bozo the Clown planted among the justices.

There is a balance and we need to find it.

Tagged: , , , , , ,

Samuel Arbesman, Interview No. 1

Reading Time: 9 minutes

Samuel Arbesman is an applied mathematician and network scientist. His recent book, The Half-life of Facts, explores how much of what we think we know has an expiry date.

Samuel, who was happy to be the first in an ongoing series of interviews, talks about his book, science, knowledge, and society.

A friend of mine, Neil Cruickshank, helped come up with some of the questions.

* * *

INTERVIEWER

Can you tell me a little bit about your background?

ARBESMAN

I began my training in evolutionary biology and I received a PhD in computational biology from Cornell University. However, even during graduate school I began to think about how to use the computational and mathematical models I had been learning about to help understand society. This transition continued when I did a postdoctoral fellowship at Harvard under Nicholas Christakis, where I explored social networks, cooperation, and scientific discovery. About two years ago I moved to Kansas City to be a Senior Scholar at the Ewing Marion Kauffman Foundation, where I study and write about a lot of different topics, ranging from the future of science to how cities grow and develop.

INTERVIEWER

What was the motivation behind writing The Half-life of facts?

ARBESMAN

I’ve always been aware of the huge amount of information that we learn that becomes out-of-date rather quickly. But as I moved into the field of quantitative social science, and explored topics from network science to scientometrics, I realized that there is a deep order to how knowledge grows and changes over time, and even how it spreads from person to person. I wanted to tell this story in the hope that a reader will find it as fascinating as I do, but more importantly, would come away with a deeper appreciation and understanding of the underlying regularities behind all of the knowledge change we see on a daily basis.

INTERVIEWER

… When I read about cognitive biases and also the research that suggests for some areas of expertise – such as medical surgery – at a certain point of time the accumulation of experience does not equate to better performance results, I think about how we often defend our opinions and decisions on the basis of our experience, but in fact that experience may just be a reinforcement of error or bias.

In your book, you supply additional reason for me to doubt even those things I may be very sure of. How do you see the connection between the half-life of facts and what this means to the idea of wisdom and the respect we offer to an individual’s experience?

ARBESMAN

It’s certainly true that many of the bits of information we learn over the years become outdated and are overturned, and so we have to make sure that what we are working with is not obsolete. And on the basis of this, experience might be a hindrance. But I think a lifetime of experience and wisdom, rather than simply an accumulation of facts, can often leave someone better prepared for dealing with change. Because they’ve had to deal with so much change throughout their lives, people often have a better sense of the shape and impact of change. While it is certainly true that someone with more experience might also be less likely to change their ways, and adhere to outdated information, understanding the regularities behind change—even if only known in an intuitive qualitative sense due to experience—can provide a mechanism for adaptation.

INTERVIEWER

When I read your book I often reflected on moments when I have been sure, and confident. And I also thought about how I have managed being in time of change. In psychology, one of the “big five” personality traits is Openness. Some of us seem to be much more comfortable with flux, or change, and readily able to respond to and even gain energy from change. Others seem to have a greater need for anchors and continuity with the past, and as the degree of change increases we focus more on and more on the things that remain unchanged, and change itself is fatiguing and depressing. Is there a fundamental disadvantage for those of us who are less open and more at ease with stability?

ARBESMAN

In a word, yes. People who cannot deal with change are going to be at a huge disadvantage in the world. These type of people might not have been disadvantaged in previous generations, where change proceeded rather more slowly, but as the many fundamental changes around us—in what we know and in what the world likes—continue to accumulate, we often have to deal with large numbers of these changes in a single lifetime. In the book, I chronicled the large number of computational information storage technologies (ranging from floppy disks to the Cloud) that I have used over the course of three decades, which is a far cry from the one or two that people of the Middle Ages might have used for storing information (books and scrolls). Those who can’t adapt will have a great deal of trouble in this world.

INTERVIEWER

You quote John H Jackson: “It takes 50 years to get a wrong idea out of medicine, and 100 years a right one into medicine.” I’ve heard teachers say similar things about changes to curriculum. Do you have any thoughts on how education and educators, particularly in schools, should incorporate your ideas into teaching and curriculum design?

ARBESMAN

This is a really important question. I think that we need to move from an educational system that is focused on memorizing facts to one that is focused on how to learn. Of course you need a fundamental background and familiarity with certain information in order to have a basic understanding of the world, so I wouldn’t throw out memorization entirely. But so much of what we know is going to change and we need to have an educational system that recognizes this. In medicine, there is continuing medical education—constantly learning what is new in one’s field—and I think this kind of attitude needs to be universalized for knowledge in general. Specifically, students need to be taught how to continue to learn new facts, and embrace the changing knowledge around them. If that is the focus, rather than the facts themselves, education will be more durable, but will also create graduates that can continue to learn on their own and adapt the world around them

INTERVIEWER

What about organizations… in a world of constant change how can understanding how facts change better prepare us for dealing with uncertainty?

ARBESMAN

Organizations often adapt slowly, just like many of us, sometimes even maintaining a mission after it has outlived its usefulness. A willingness to confront these changes must be deeply embedded within the leadership of the organization, which hopefully will be easier when people are educated to understand changing knowledge. Otherwise, the organization will slowly fail, hemorrhaging the more adaptable people—who are frustrated by the lack of change—along the way.

INTERVIEWER

I was struck by your statement: “ONE of the most fundamental rules of hidden knowledge is the lesson learned from InnoCentive: a long tail of expertise— everyday people in large numbers—has a greater chance of solving a problem than do the experts.”

I imagine trying to promote this idea in an organization, such as an IT firm or a Government Department – where there is a strong culture of respect for expertise – and I think it would be an extremely hard sell. If I can be extreme, this idea argues that credentials or other normally recognized markers of individual status are maybe not worth as much, or perhaps overvalued. Do you have any comments on how the idea of “the long tail of expertise” can actually function in a domain where expertise is part of the status and hierarchy?

ARBESMAN

I think expertise is still important for many questions, especially ones that can be solved in a relatively straightforward manner. But as we move into an increasingly complicated and interdisciplinary world, the expertise we value with likely shift: we will move from valuing those who can solve problems, to those who know different ways to solve problems, or at least those who know how to ask the right questions of a large crowd. Using InnoCentive, or other way to crowdsource expertise is by no means trivial, and understanding the ways that they succeed, as well as the many ways that they can fail, is going to become more important.

INTERVIEWER

How important of a role does diversity play in all of this? If we all goto the same training and schools, and we’re all taught to look at the problem the same way, to what extent would this impact the long tail?

ARBESMAN

Diversity is critical. We don’t want everyone to have the same background and information. At the same time, making sure that we have people who can bring together diverse backgrounds, translating from one field to another—even at the level of jargon—is also crucial, and something that we often neglect in our excitement of the power of intellectual diversity.

INTERVIEWER

For the most part, I find the tone of your book to be very positive and optimistic – a message of affirmation of the value of trying to understand and learn. But I also note your observation in the Chapter “The Human Side of Facts,” where you describe how we seem to come to a point, often quite early in our lives, where we cease to learn. I observe this often, how we feel there is a sense of having learned and after that learning, life- professional life – is really just the application of learned knowledge. I don’t see a great commitment to “lifelong learning” in North American society, certainly not between the ages of, say, forty and retirement. But professional people often have their greatest influence on the rest of society at this age. Do you have any thoughts on what your ideas imply about lifelong learning and personal development, particularly for those of us who are well-established in our professional careers?

ARBESMAN

As I mentioned earlier, I think we need to take a page from medicine and its devotion to continuing medical education. Of course, there is a clear incentive in this field, as lives are on the line. But If we can find ways to better incentivize continuing education for everyone, we’ll be a better society. Frankly, this is a hard thing to do. If we can teach students at an early age about the obsolescence of their knowledge, this task will be easier. But for now, it’s quite daunting.

INTERVIEWER

Changing gears a bit… What authors have your learned the most from and why?

ARBESMAN

I’ve learned a great deal from the novelist Neal Stephenson. His books are generally a set of fascinating ideas wrapped around an engaging plot. The plots pull you along, and in the process I’ve learned about—and been forced to think deeply about—the Scientific Revolution, the invention of the modern monetary system, mathematical platonism, the relationship between Greek mythology and the history of technology, and much more. If you need your mind expanded, Stephenson will deliver.

I’ve also gained a lot from Steven Johnson, who has written many fascinating “idea books” (this term doesn’t quite satisfy me but it’s hard to think of a better description). His ability to weave together numerous concepts that often seem unrelated on the surface and then convey them in a coherent and exciting way is something that is incredibly rare and wonderful to experience.

INTERVIEWER

Do you have daily writing ritual?

ARBESMAN

I unfortunately don’t have much in the way of rituals. Essentially, I set myself a low word count goal for the day (the amount varies based on how much writing I need to achieve). And then I exceed it. That way, I always overachieve and feel good about my writing for the day. And once I’ve gotten a whole lot of quantity, I then pare it down and do my best to turn it into quality.

INTERVIEWER

Say I’ve anointed you as dictator. What five books would you make every adult read?

ARBESMAN

This certainly sounds like an intriguing dictatorship. Rather than focusing on my favorite books, I’ll try to limit this to five books that I think are important for thinking about science, knowledge, and society:

Little Science, Big Science by Derek J. de Solla Price — the foundation for a rigorous and quantitative approach for thinking about how science works.

Collected Fictions of Jorge Luis Borges — Interested in thinking about knowledge and infinity? The stories of Borges are essential reading.

Gödel, Escher, Bach: An Eternal Golden Braid by Douglas Hofstadter — from computer science to how the mind works, this book will change how you think about the world of information.

Nonzero by Robert Wright — a wonderful exploration of how the world has become more complicated and better over time, improving each of our lives

The Varieties of Scientific Experience by Carl Sagan—Sagan’s examination of the complexity of the universe and his personal approach to religion as scientific awe

And an optional bonus book for my dictatorship:

How I Killed Pluto and Why It Had It Coming by Mike Brown — captures the excitement and process of science. It’s also a great story.

* * *

Tagged: , ,

The Half-life of Facts

Reading Time: 8 minutes

Facts change all the time. Smoking has gone from doctor recommended to deadly. We used to think the Earth was the center of the universe and that Pluto was a planet. For decades we were convinced that the brontosaurus was a real dinosaur.

Knowledge, like milk, has an expiry date. That’s the key message behind Samuel Arbesman’s excellent new book The Half-life of Facts: Why Everything We Know Has an Expiration Date.

We’re bombarded with studies that seemingly prove this or that. Caffeine is good for you one day and bad for you the next. What we think we know and understand about the world is constantly changing. Nothing is immune. While big ideas are overturned infrequently, little ideas churn regularly.

As scientific knowledge grows, we end up rethinking old knowledge. Abresman calls this “a churning of knowledge.” But understanding that facts change (and how they change) helps us cope in a world of constant uncertainty. We can never be too sure of what we know.

In introducing this idea, Abresam writes:

Knowledge is like radioactivity. If you look at a single atom of uranium, whether it’s going to decay — breaking down and unleashing its energy — is highly unpredictable. It might decay in the next second, or you might have to sit and stare at it for thousands, or perhaps even millions, of years before it breaks apart.

But when you take a chunk of uranium, itself made up of trillions upon trillions of atoms, suddenly the unpredictable becomes predictable. We know how uranium atoms work in the aggregate. As a group of atoms, uranium is highly regular. When we combine particles together, a rule of probability known as the law of large numbers takes over, and even the behavior of a tiny piece of uranium becomes understandable. If we are patient enough, half of a chunk of uranium will break down in 704 million years, like clock-work. This number — 704 million years — is a measurable amount of time, and it is known as the half-life of uranium.

It turns out that facts, when viewed as a large body of knowledge, are just as predictable. Facts, in the aggregate, have half-lives: We can measure the amount of time for half of a subject’s knowledge to be overturned. There is science that explores the rates at which new facts are created, new technologies developed, and even how facts spread. How knowledge changes can be understood scientifically.

This is a powerful idea. We don’t have to be at sea in a world of changing knowledge. Instead, we can understand how facts grow and change in the aggregate, just like radioactive materials. This book is a guide to the startling notion that our knowledge — even what each of us has in our head — changes in understandable and systematic ways.

Why does this happen? Why does knowledge churn? In Zen and the Art of Motocycle Maintenance, Robert Pirsig writes:

If all hypotheses cannot be tested, then the result of any experiment are inconclusive and the entire scientific method falls short of its goal of establishing proven knowledge.

About this Einstein had said, “Evolution has shown that at any given moment out of all conceivable constructions a single one has always proved itself absolutely superior to the rest,” and let it go at that.

… But there it was, the whole history of science, a clear story of continuously new and changing explanations of old facts. The time spans of permanence seemed completely random, he could see no order in them. Some scientific truths seemed to last for centuries, others for less than a year. Scientific truth was not dogma, good for eternity, but a temporal quantitative entity that could be studied like anything else.

A few pages later, Pirsig continues:

The purpose of scientific method is to select a single truth from among many hypothetical truths. That, more than anything else, is what science is all about. But historically science has done exactly the opposite. Through multiplication upon multiplication of facts, information, theories and hypotheses, it is science itself that is leading mankind from single absolute truths to multiple, indeterminate, relative ones.

With that, lets dig into how this looks. Arbesman offers a example:

A few years ago a team of scientists at a hospital in Paris decided to actually measure this (churning of knowledge). They decided to look at fields that they specialized in: cirrhosis and hepatitis, two areas that focus on liver diseases. They took nearly five hundred articles in these fields from more than fifty years and gave them to a battery of experts to examine.

Each expert was charged with saying whether the paper was factual, out-of-date, or disproved, according to more recent findings. Through doing this they were able to create a simple chart (see below) that showed the amount of factual content that had persisted over the previous decades. They found something striking: a clear decay in the number of papers that were still valid.

Furthermore, they got a clear measurement of the half-life of facts in these fields by looking at where the curve crosses 50 percent on this chart: 45 years. Essentially, information is like radioactive material: Medical knowledge about cirrhosis or hepatitis takes about forty-five years for half of it to be disproven or become out-of-date.

half-life of facts, decay in the truth of knowledge

Old knowledge, however, isn’t a waste. It’s not like we have to start from scratch. “Rather,” writes Arbesman, “the accumulation of knowledge can then lead us to a fuller and more accurate picture of the world around us.”

Isaac Asimov, in a wonderful essay, uses the Earth’s curvature to help explain this:

When people thought the earth was flat, they were wrong. When people thought the earth was spherical, they were wrong. But if you think that that thinking the earth is spherical is just as wrong as thinking the earth is flat, then your view is wronger than both of them put together.

When our knowledge in a field is immature, discoveries come easily and often explain the main ideas. “But there are uncountably more discoveries, although far rarer, in the tail of this distribution of discovery. As we delve deeper, whether it’s intro discovering the diversity of life in the oceans or the shape of the earth, we begin to truly understand the world around us.”

So what we’re really dealing with the long tail of discovery. Our search for what’s way out at the end of that tail, while it might not be as important or as Earth-shattering as the blockbuster discoveries, can be just as exciting and surprising. Each new little piece can teach us something about what we thought was possible in the world and help us to asymptotically approach a more complete understanding of our surroundings.

In an interview with the Economist, Arbesman was asked which scientific fields decay the slowest-and fastest-and what causes that difference.

Well it depends, because these rates tend to change over time. For example, when medicine transitioned from an art to a science, its half-life was much more rapid than it is now. That said, medicine still has a very short half-life; in fact it is one of the areas where knowledge changes the fastest. One of the slowest is mathematics, because when you prove something in mathematics it is pretty much a settled matter unless someone finds an error in one of your proofs.

One thing we have seen is that the social sciences have a much faster rate of decay than the physical sciences, because in the social sciences there is a lot more “noise” at the experimental level. For instance, in physics, if you want to understand the arc of a parabola, you shoot a cannon 100 times and see where the cannonballs land. And when you do that, you are likely to find a really nice cluster around a single location. But if you are making measurements that have to do with people, things are a lot messier, because people respond to a lot of different things, and that means the effect sizes are going to be smaller.

Arbesman concludes his economist interview:

I want to show people how knowledge changes. But at the same time I want to say, now that you know how knowledge changes, you have to be on guard, so you are not shocked when your children (are) coming home to tell you that dinosaurs have feathers. You have to look things up more often and recognise that most of the stuff you learned when you were younger is not at the cutting edge. We are coming a lot closer to a true understanding of the world; we know a lot more about the universe than we did even just a few decades ago. It is not the case that just because knowledge is constantly being overturned we do not know anything. But too often, we fail to acknowledge change.

Some fields are starting to recognise this. Medicine, for example, has got really good at encouraging its practitioners to stay current. A lot of medical students are taught that everything they learn is going to be obsolete soon after they graduate. There is even a website called “up to date” that constantly updates medical textbooks. In that sense we could all stand to learn from medicine; we constantly have to make an effort to explore the world anew—even if that means just looking at Wikipedia more often. And I am not just talking about dinosaurs and outer space. You see this same phenomenon with knowledge about nutrition or childcare—the stuff that has to do with how we live our lives.

Even when we find new information that contradicts what we thought we knew, we’re likely to be slow to change our minds. “A prevailing theory or paradigm is not overthrown by the accumulation of contrary evidence,” writes Richard Zeckhauser, “but rather by a new paradigm that, for whatever reasons, begins to be accepted by scientists.”

In this view, scientific scholars are subject to status quo persistence. Far from being objective decoders of the empirical evidence, scientists have decided preferences about the scientific beliefs they hold. From a psychological perspective, this preference for beliefs can be seen as a reaction to the tensions caused by cognitive dissonance.

A lot of scientific advancement happens only when the old guard dies off. Many years ago Max Planck offered this insight: “A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.”

While we have the best intentions and our minds change slowly, a lot of what we think we know is actually just a temporary knowledge to be updated in the future by more complete knowledge. I think this is why Nassim Taleb argues that we should read Seneca and not worry about someone like Jonah Lehrer bringing us sexy narratives of the latest discoveries. It turns out most of these discoveries are based on very little data and, while they may add to our cumulative knowledge, they are not likely to be around in 10 years.

The Half-life of Facts is a good read that help puts what we think we understand about the world into perspective.

Follow your curiosity and read my interview with the author. Knowing that knowledge has a half-life isn’t enough, we can use this to help us determine what to read.

Tagged: , , , , , , , , , , ,

Are Cities More Innovative?

Reading Time: 1 minute

Jane Jacobs in The Death and Life of Great American Cities: “The larger a city, the greater the variety of its manufacturing, and also the greater both the number and the proportion of its small manufacturers.”

The benefits that cities offer to smallness are just as marked in retail trade, cultural facilities and entertainment. This is because city populations are large enough to support wide ranges of variety and choices in these things. And again we find that bigness has all the advantage in smaller settlements. Towns and suburbs for instance are natural homes for huge supermarkets, and for little else in the way of groceries, for standard movie houses or drive ins for little else in the way of theatre.

There are simply not enough people to support further variety, although there may be people(too few of them) who would draw upon it were it there. Cities, however, are the natural homes of supermarkets, and standard movie houses, plus delicatessens, Viennese bakeries, foreign groceries, art movies, and so on, all of which can be found co-existing, the standard with the strange, the large with the small. Wherever lively and popular parts of the cities are found, the small much outnumber the large.

“Cities, then,” writes Steven Johnson in Where Good Ideas Come From: The Natural History of Innovation, “Cities, then, are environments that are ripe for exaptation, because they cultivate specialized skills and interests, and they create a liquid network where information can leak out of those subcultures, and influence their neighbors in surprising ways. This is one explanation for superlinear scaling in urban creativity. The cultural diversity those subcultures create is valuable not just because it makes urban life less boring. The value also lies in the unlikely migrations that happen between the different clusters.”

And Samuel Arbesman, in The Half-life of Facts, adds: “Larger groups of interacting people can maintain skills and innovations, and in turn develop new ones. A small group doesn’t have the benefit of specialization and idea exchange necessary for any of this to happen.”

***
Still curious? If you want a deeper understanding, read Growth in Cities.

Tagged: , , , ,

Explore

Browse