In this post I will go over some of these latter critiques, particularly the ones that focus on combatting “male bias” in science. Of course, there are also parts of science that will have female biases (and it goes without saying that feminism is about combatting those too), but given that women were, for the longest time, largely forbidden from even becoming scientists (and still are, in some parts of the world), and given that science as a whole is still male-dominated, it shouldn’t be too controversial to focus on the male bias in our scientific literature.

This bias comes in two forms: bias in data-collection, and bias in data-interpretation.
I don’t think the existence of such biases in data-collection is too controversial. I assume the scientists among us know of many stories where women were underrepresented —or even straight-up excluded— from scientific datasets. My personal favorite example is that time a feminist discovered that several evolutionary psychologists developed theories of the female orgasm, using only datasets of male orgasms. (Oh evo-psych, never change)

Bias in data-interpretation, on the other hand, might be a bit harder to visualize. So let me give you a famous example so you can see what the feminists are getting at:

Fertilization: A case of male bias in data interpretation

Traditionally, biologists adopted what is sometimes called the “Prince Charming model” of fertilization, wherein the process was treated as a hero story where a single, intrepid sperm battles through the hostile uterus, survives perilous challenges, defeats rival sperm, and finally claims the passive egg, which it penetrates and breathes life into.
This is similar to how men and women are often treated in literary works, with the men being active and heroic, while the women are passive trophies. Feminists argue that this imagery shaped descriptions of fertilization, even after it became clear this model is inaccurate since:

1. The egg is not passive; it actually actively selects a sperm and produces cell-surface projections that clasp the sperm and draw it inside.

2. Mammalian sperm cannot fertilize an egg immediately; they must undergo a process called “capacitation”. In this process the uterus is not an obstacle course for the sperm, but is actually an active participant that helps the sperm become ready for fertilization.

Photographs of 1 were published as early as 1895, yet their role was virtually ignored until the 1980s. Similarly, point 2 has been known since the 1950s, and yet the “Prince Charming model” was still the default way fertilization was presented. This suggests that gender stereotypes biased biologists’ interpretations; with “penetration” remaining the default verb, even though “engulfment” would be at least as accurate. The evidence was there, but male bias caused it to be ignored.

Combatting bias by increasing diversity

Feminists argue that to combat this bias, science should start incorporating feminist values. One way to do that, they argue, is by increasing diversity in research teams and peer review. This would help with the development of better scientific theories, because it increases the diversity of the theories we evaluate.
When a scientist generates a theory, they may do so for any reason (aesthetic reasons, political reasons, because it came to them in a dream…). Contemporary scientific methodology only concerns itself with testing theories, but what happens if there’s a systemic bias in which theories get generated?

Say a group of scientists have generated a couple theories which all have a male bias. When we start scientifically testing which one of these we should adopt, no matter how carefully we apply the scientific method, the one we will end up with is a theory that has a male bias. Science only helps us select the best theory amongst the available theories. To increase the quality of our theories we not only need to properly test our theories, we also need diversity in which theories get generated.

Feminist theory adoption

In science we often encounter the problem of underdetermination. This means that there are multiple theories that are equally supported by the available data. Which theory one should adopt in such a scenario is hotly contested amongst philosophers, but perhaps the most popular approach is to adopt the theory that has the most social utility.

For example, if one group of scientists present a theory that is totally inscrutable to almost everyone, while another group of scientists present an (equally empirical) theory that’s totally legible and allows engineers to start creating new inventions based on it, this approach champions adopting the second theory.

This is in line with the broader pragmatist school of epistemology, a school that says knowledge should be actually useful. If you (like me) also think we should adopt the theories that are the most socially useful, then adopting the more feminist theories (from the available evidence-based theories) would be a good idea in a world where we’re suffering from sexism. Do we live in such a world?

It sure seems that way…

Of course this doesn’t mean we should automatically pick the most feminist theory. There are other problems and biases too. You could, for example, copy-paste this entire blogpost and with minimal effort transform it into a blogpost about racial bias.
But it does suggest that we should overall err on the side of feminism.

This post was based on the lectures of Dr. Martin, Dr. Baker, and Dr. Okruhlik

People are often uncertain about which choices are morally right. Imagine you're standing in front of a food stall with €2 in your pocket. Do you get the tasty meat sandwich or settle for the less appetizing lettuce sandwich? You know everything there is to know about animal suffering and the environmental impact of meat consumption. But you’re still stuck because you don’t know to what degree you should care about animals. The empirical facts alone don’t tell you what you should do.

This is an example of moral uncertainty: uncertainty that does not arise from empirical factors, but from moral ones.

In this blog post I’ll present three arguments for why we should be morally uncertain. Then, I’ll discuss two arguments for why we should take moral uncertainty into account when making decisions. In a future post, I’ll examine some arguments for why we should not take moral uncertainty into account.

Unlike all my previous posts, this one won’t introduce any new ideas and will only present arguments found in the existing literature on moral uncertainty, particularly the works of MacAskill, Ord and Bykvist. If you’ve already read their works (and don’t need a refresher), you can skip this post.
Also, I will focus on the big-picture rationale for moral uncertainty, not specific case studies.

With that out of the way, let’s get into it:

1. Why Should We Be Morally Uncertain?

Examining moral uncertainty becomes relevant only if people actually experience doubts about their moral views. However, some individuals are firmly convinced of their moral beliefs.1 But this certainty is not always rational. Here are three arguments for why we should sometimes be morally uncertain.

1.1 Moral Disagreement

A good reason to not be too sure of yourself is that smart people disagree about morality… a lot.2 Although there are moral issues on which there is broad consensus (like that killing a random baby isn’t exactly great) many other issues remain the subject of significant debate. In many of these cases, intelligent and well-informed individuals disagree. This makes it less reasonable to adhere to a particular moral theory with absolute certainty. After all, it’s possible that others, who have thought about the issue just as deeply (or even more deeply), have reached a better-founded conclusion. They may have intuitions, experiences, or evidence that could influence your beliefs if you were aware of them. Therefore, it is rational to reconsider confidence in your own moral positions if reasonable and thoughtful individuals reach different conclusions.3

1.2 Moral Philosophy is Complex

The second reason, which follows from the previous one, is that moral philosophy is a very complex discipline. Small nuances between different moral theories can have major implications for our actions. Moreover, there’s a great diversity of moral theories, each with its own arguments for and against. Various factors can make a theory appealing, such as how simple or intuitive it is. Weighing all these factors against each other is not easy, and even small errors in this evaluation can have significant consequences for our moral judgment.

Additionally, moral judgments can be influenced by various forms of bias, e.g:

• Your culture, upbringing, and social environment shape what seems “obvious” to you.

• The status quo feels morally right just because it’s familiar.

• Your evolutionary instincts weren’t exactly fine-tuned for abstract ethical debates.

These factors introduce general irrational tendencies in our brains.4 Generally, we do not know whether we have these biases until they are pointed out to us.5 And even when they are pointed out, we often fail to eliminate them entirely (because, well, bias).6 It’s therefore better to assume that we are biased in many ways, even when we try our best not to be. Given all this, the odds that you’ve flawlessly navigated moral philosophy without error seem… let’s say, low-ish.

1.3 Overconfidence in Humans

A final argument is the general human tendency toward overconfidence.7 Research has consistently shown that people are often much more certain of their judgments than is actually justified. For example:

• Studies show that when people estimate that an event has a probability of more than 70%, it usually happens in less than 70% of cases.8

• Research has found that when people are 100% certain of something9, they are still wrong in about 20% of cases.10

• This overconfidence affects even experts.11

If we are overconfident in so many different domains, it’s likely that this also applies to moral philosophy. In fact, it even seems more likely that we are overconfident in moral matters because people are often strongly biased when it comes to moral issues.12 Therefore, when we feel great certainty about a particular moral belief, it’s very likely that we should actually be less certain.

In summary, these three arguments show that it is wise to assume that we should be morally uncertain. However, the question remains whether we should also incorporate this uncertainty into our decision-making.

2. Should We Act On Moral Uncertainty?

There are two main reasons to take moral uncertainty into account when making decisions. First, in some situations, it can be beneficial to consider moral uncertainty because we have nothing to lose by doing so. Second, in addition to objective moral obligations, there are also subjective moral obligations that can influence our decisions.13

2.1 Nothing to Lose

In certain cases, there is nothing to lose by considering moral uncertainty. Suppose Timmy enjoys both a lettuce sandwich and a meat sandwich equally. While he’s most confident in a moral theory that states that eating meat is neither better nor worse than not eating meat, he also finds another moral theory, which claims that eating meat is wrong, somewhat convincing. In such a case, it seems wiser to choose the lettuce sandwich just to be safe, even if it later turns out that eating meat was not unethical. This choice follows the logic of the dominating decision rule from decision theory (aka: if one choice is at least as good as another in all scenarios, but better in some, then you should pick that one).14 It’s therefore plausible that a similar rule applies in situations of moral uncertainty, implying that there are norms that can guide our behavior in such cases.

2.2 Subjective Norms

Philosopher Frank Jackson introduced a thought experiment to illustrate the existence of subjective norms:15

Kathandra is a doctor with a patient named Jimothy.16 Jimothy keeps sneezing out bubbles and she doesn’t know why. She’s unsure whether Jimothy has condition A or condition C; both possibilities seem equally likely. It’s impossible for Kathandra to obtain further diagnostic information to help her make a decision. She has three medicines available: A, B, and C. If Jimothy receives medicine A and he actually has condition A, he will fully recover; but if he has condition C, he will die. Conversely, if he receives medicine C and he has condition C, he will fully recover; but if he has condition A, he will die. If Kathandra chooses medicine B, Jimothy will almost fully recover regardless of the condition.

       Jimothy has condition A - 50%         Jimothy has condition C - 50%
A         Completely cured                                 Dead 😵
B         Almost completely cured                 Almost completely cured
C         Dead 😵                                                       Completely cured

Since Kathandra assigns equal probability to both conditions, it would be reckless for her to administer medicine A or C due to the significant risk of Jimothy’s death. Jackson's argument suggests that there are subjective norms that advocate choosing the safer medicine B, thereby reducing the risk of a fatal error.

2.3 Subjective Norms in Moral Uncertainty

The previous thought experiment showed that subjective norms exist. But what about subjective norms when it comes to moral uncertainty? MacAskill, Bykvist, and Ord explored this with a modified version of the thought experiment, specifically focusing on moral uncertainty:17

Kathandra is a doctor faced with two critically ill patients: Amandrew, a human, and Chrisabelle, a chimpanzee. Both suffer from the same fatal bubble sneeze disease and urgently need treatment. Kathandra has only one bottle of life-saving medicine.

• If she gives the entire bottle to Amandrew, Amandrew will survive but with a disability that halves their happiness.

• If she gives the entire bottle to Chrisabelle, Chrisabelle will fully recover.

• If she splits the medicine, both will survive, but their health and happiness will be slightly lower than if they had fully recovered.

Kathandra is a utilitarian, meaning she tends to add up happiness to determine the best outcome. However, she’s uncertain about the moral value of a chimpanzee’s happiness. She considers it equally likely that either:

1. A chimpanzee’s happiness has no moral value, or

2. A chimpanzee’s happiness is just as morally important as a human’s.

Since she cannot gather more information, she must decide between:

• A: Give the whole bottle to Amandrew.

• B: Split the bottle evenly.

• C: Give the whole bottle to Chrisabelle.

The outcomes in terms of happiness (higher is better):

      Amandrew’s happiness 🥳       Chrisabelle’s happiness 🐵🎉
A         50                                                      0
B         49                                                      49
C         0                                                        50

The moral theories disagree sharply on whether option A or C is best, but both agree that option B is only a small step away from the best choice. This can be summarized as:

Chimpanzee happiness does not matter 50%      Chimpanzee happiness does matter 50%
A         Morally right                                                             Very immoral 👎
B         Slightly immoral                                                      Slightly immoral
C         Very immoral 👎                                                       Morally right

Now, imagine that in the future, it turns out that chimpanzee happiness is indeed as morally valuable as human happiness, meaning that giving the whole bottle to Chrisabelle would have been the right choice. What should Kathandra do in this situation?

In the earlier thought experiment, we saw that it would be epistemically reckless for Kathandra to choose anything other than the safest option, medicine B. Similarly, in this moral scenario, it seems morally reckless to choose anything other than option B.18

This suggests that we should take both epistemic (knowledge-based) and moral uncertainty seriously and that there are norms that can guide our decisions in both cases.

I think this paints a pretty clear picture in favor of embracing moral uncertainty. For the sake of fairness I will also go over some objections to it in the future, but (spoiler alert) I mostly don’t find them too convincing. The only thing I haven’t touched on yet is how to incorporate moral uncertainty into your decisions/theories. If you don’t know where to start, then consider checking out my post Resolving moral uncertainty with randomization.

Here we run into an uncomfortable observation. Meat from larger animals tends to produce more CO₂, while meat from smaller animals tends to cause more animal suffering.
This makes intuitive sense: when converting animals into a set mass of meat, more individual animals must suffer if they are smaller. We can see this in the data, e.g., we slaughter 309 million cows per year, while the number of chickens slaughtered exceeds 75 Billion per year.

Let’s start ranking

Let's start by comparing greenhouse gas emissions. Which foods have the lowest CO₂ impact?
Here’s how many kilograms of CO₂-equivalized emissions are produced per kilogram of food (some create methane etc, but that’s accounted for by converting it into the same unit), ranked from worst to best:

Beef (beef herd)     99.48 kg
Lamb & Mutton    39.72 kg
Beef (dairy herd)   33.3 kg
Prawns (farmed)   26.87 kg
Cheese                       23.88 kg
Fish (farmed)          13.63 kg
Pig Meat                   12.31 kg
Poultry Meat           9.87 kg
Eggs                             4.67 kg
Rice                              4.45 kg
Tofu                             3.16 kg
Milk                             3.15 kg
Soy milk                     0.98 kg

As we can see, larger animals tend to create more CO₂.

Can we create a similar ranking for animal suffering? Yes we can, if we take the sentience figures from this report and apply it to the production process used to create the meat. Here’s the equivalized days of suffering per kg of food (some have bigger brains etc, but that’s accounted for by converting it into the same unit), ranked from worst to best:

Farmed catfish       290 d
Battery cage eggs  42 d
Farmed salmon      27 d
Chicken                      27 d
Turkey                         15 d
Pork                              3.9 d
Beef                              0.8 d
Milk                             0.049 d
Tofu                             0 d
Soy milk                    0 d

As we can see, the smaller animals tend to produce more suffering. The best products on these lists, both in terms of animal suffering and CO₂, are tofu and soy milk.
For those who don't want to go vegan, milk is still comparatively good in terms of both CO₂ and animal suffering, so you might want to go vegetarian then. But for those who want to keep eating meat, we have a dilemma between CO₂ and animal suffering. Compare beef and chicken again. Beef has ten times the CO₂ impact of chicken meat, but chicken meat has 34 times higher impact on suffering than beef.2

Now, don’t take these exact numbers as gospel. We’re still in the process of researching this so I expect the exact numbers to change over time. If you want to use different numbers you can play with the calculations using this interactive chart.3 The important thing is not the exact numbers but rather the orders of magnitude; which clearly point in the direction of meat production with smaller animals creating more suffering.

What to pick

Farmed fish is terrible in terms of animal welfare and pretty bad in terms of CO₂, so best avoid that. Beef is terrible in terms of CO₂, but comparatively good in terms of animal suffering (the word "comparative" is doing a lot of heavy lifting there). So if you don't care about animals but do care about CO₂, beef is the most important one to avoid. For those who only care about animals, farmed fish and chickens are the most important to avoid.

For those of you who are having trouble weighing the two against one another, consider this: When choosing between, e.g, chicken and beef, you're effectively choosing between creating the equivalent of 89 kg more CO₂ (with beef) or the equivalent of 26 days more torture (with chicken). To put this into context, a campfire produces about 10 kg of CO₂. Would you rather make 9 campfires or start torturing for 26 days?4 Are people who have a hearth morally equivalent to people who torture every day? Or what about a candle? It produces 0,01 kg of CO₂. Is lighting a candle worse than torturing an animal for a couple minutes? I would say: no, clearly not. The enormous quantity of suffering created in factory farms easily trumps the quantity of CO₂ that it creates.
We hear a lot about climate change in the news and online, while animal suffering gets little attention. But the scale of suffering that is produced in factory farms is unimaginably large. Please consider eating fewer small animals.

Keep in mind that this is only a heuristic and not an ironclad rule. E.g. a good case-control study is still preferable over a bad cohort study. However, when presented with studies of similar quality, or if you can’t assess which study is high quality, go for the one higher up the chain.

Also keep in mind that this is not a hierarchy of ‘usefulness’. I generally prefer listening to experts because they can explain vast amounts of things in “human” terms, inform me how different things interact, and subsequently answer my specific questions. It's just that for any single piece of information you'd rather have a meta-analysis backing you up than an expert opinion.

This is also not a hierarchy of ‘prowess as a scientist’ or ‘what we should fund’. Just because someone has worked on one of the top studies doesn’t mean they’re a better scientist than those who did studies near the bottom. The top also tends to be more expensive and labor intensive. Sometimes, especially with exploratory research, it’s a better use of time and money to go for a lower one.

Voting is at the heart of democracy—it's how we collectively decide who will lead us and shape public policy. Traditionally, most democratic elections rely on deterministic voting systems. These are the systems where the outcome is based on a clear majority, and the winner is the one who secures the most votes.
This approach has its objections, for example the problem of the "tyranny of the majority", where the minority's voice is often silenced, and the concentration of power is in the hands of just over half the voters.1

In response to these concerns, recent research has started exploring non-deterministic voting systems. Unlike deterministic systems, these introduce an element of chance, beyond just using randomness to break ties.
It might be best if I first give a simple example of a non-deterministic voting system so you can envision how this would work.

Example: The Random Ballot Method

One of the simplest non-deterministic voting systems is called the Random Ballot. Here’s how it works: voters cast their ballots like normal, but instead of counting the votes, one ballot is randomly selected to decide the winner. If you control 51% of the vote, you don’t automatically win—you just have a 51% chance of your candidate being selected. The same applies to any minority group; if they control 30% of the vote, they have a 30% chance of winning.

This might seem unfair at first glance, but it actually offers a more proportional distribution of power over time. In traditional systems, controlling a majority of votes means controlling all the power. In non-deterministic systems, power is spread according to the actual support each group has. The random ballot method, for instance, perfectly aligns power with voting percentage.

Non-Deterministic Systems

Interestingly, the idea of using randomness in decision-making isn’t new. In ancient Athenian democracy, officials were often chosen by sortition—essentially, by random lot. This was based on the belief that every citizen had an equal right and ability to contribute to governance, and it was a way to prevent a small elite from holding too much power.

Today, we see echoes of this practice in the use of citizens' assemblies, where a group of citizens is randomly selected to deliberate on policy issues. Non-deterministic voting systems share a similar ethos, aiming to distribute power more fairly and prevent the dominance of the majority.

Of course, not all non-deterministic systems are as simple or as proportional as the Random Ballot. Take for example the First-to-get-two system. It works like the random ballot, but instead of picking just one ballot at random we keep randomly drawing ballots until we have two that voted for the same candidate. This creates a power dynamic somewhere between purely random and majority-dominated. We can visualize how much power each voting system gives to a voting bloc with the following graph:

So over time most deterministic voting systems give 100% of the power to a candidate who can get >50% of the votes. We can visualize this as a step function (blue line). Of course some voting systems require a supermajority in which case it is still a step function but more to the right (purple line).
Now look at the random ballot (green line), see how the distribution of power over time is perfectly proportional to the amount of votes a candidate/party receives? But not all non-deterministic voting systems are like that, the “First-to-get-two” approach I explained earlier, is somewhere in between perfectly proportional and the step function (red line).

While I think that the random ballot and first-to-get-two are too simplistic of a voting system to be any good, I do hope that they serve as a way to show the features of non-deterministic voting systems in general, and how a more sophisticated non-deterministic system may positively influence elections.2

In this post, I’ll outline the reasons philosophers have given for why we should participate in democratic elections, and see whether these reasons hold up when it comes to non-deterministic voting systems. While I'll focus on elections where a single candidate is chosen, much of what I’ll discuss applies to other collective decisions, like referendums or parliamentary elections as well. I'll assume that these elections are free and fair, and that their outcomes are actually realized —since our goal is to focus on the inherent qualities of the voting system and not on outside factors like corruption.

2. Reasons to vote in non-deterministic elections

2.1. Prudentialist and Act-Consequentialist reasons to vote

Many people believe we have a duty to vote in a democracy. However, not everyone agrees, for instance, in their paper "Is there a duty to vote?", philosophers Lomasky and Brennan challenge two key arguments that support it: the prudentialist argument and the act-consequentialist argument.

1. Prudentialist Argument: This argument says that we should vote to advance our own interests. The logic goes like this:

   • You should work to promote your own interests.

   • Voting is a way to do that.

   • Therefore, you should vote.

2. Act-Consequentialist Argument: This is similar, but it shifts focus from self-interest to promoting the interests of others:

   • You should work to promote the interests of others.

   • Voting is a way to do that.

   • Therefore, you should vote.

The core idea behind both arguments is that election results affect people's well-being. However, critics like Lomasky and Brennan argue that the chance of an individual vote making a decisive impact on an election is so small that it hardly justifies the time and effort of voting. They point out that the probability of one vote swinging an election is nearly zero.

Philosopher Derek Parfit, however, presents a counter-argument. He agrees that the chances of one vote deciding an election are slim, but argues that small probabilities still matter when the stakes are high. For instance, if you were a nuclear engineer with a 1 in a million chance of causing a disaster, that tiny probability would still be worth worrying about. Similarly, voting is a small personal cost with a potential huge collective impact. According to Parfit, even though the chances of your vote being decisive are slim, the overall benefit of voting is still significant when considering the whole society.3

2.2. Prudentialist and Act-Consequentialist Arguments in Non-Deterministic Elections

One major assumption behind Lomasky and Brennan's critiques is that the voting system in question is deterministic. In non-deterministic systems, where each vote has an influence on the probability of a candidate winning, rather than directly determining the outcome, these two arguments for going to the polls become more compelling.

For instance, in a Random Ballot system, the influence of any voter is proportional to 1/N (where N is the number of voters). This influence is larger compared to deterministic systems, where the likelihood of any one vote being pivotal is much smaller. So, these two arguments become stronger for non-deterministic systems, since each vote carries a more consistent influence.

2.3. Fallibility

Lomasky and Brennan also argue that voters are often fallible—they make mistakes or misjudge candidates. Political outcomes are unpredictable, and campaign promises often don’t translate into actual policy changes. This uncertainty diminishes the expected benefits of voting. They suggest that the duty to vote is only valid if you’re well-informed and if the election is close enough that your vote could matter. The fallibility of voters applies to both deterministic and non-deterministic elections. The unpredictability of politics doesn’t disappear in non-deterministic systems. We can model this mathematically, so for the mathematically inclined: here’s my research paper (the basis for this blogpost) which includes more detailed and academic versions of all the arguments, and more formalisms.
TL;DR Unlike the prudentialist and act-consequentialist argument, this argument doesn’t make your reasons to vote in a non-deterministic election automatically stronger, but neither does it them automatically weaker. Instead it depends on the believed competency of the voter. For a voter who believes they are more competent than average, it strengthens the argument, for voters who believe they are less competent than average it weakens it.

2.4. Influence of Voting on the Winner’s Mandate

Another reason to vote is to influence the size of the winning candidate’s mandate—their authority to govern based on how many votes they receive. Even if your vote doesn’t decide the winner, it could help increase your preferred candidate's legitimacy or diminish the mandate of a losing candidate. A large mandate is often seen as giving a candidate more power to enact their policies.

However, research shows that a party’s effectiveness is mostly not tied to its mandate size. Also, an individual’s contribution to the mandate is almost negligible.

Suppose however that this research turns out to be incorrect, would the mandate argument be stronger in non-deterministic elections? After all, the argument works similarly; while randomness plays a role in determining the winner, vote counts still serve as a measure of public support. However, since non-deterministic systems can lead to winners with smaller mandates, supporters of minority candidates might have a stronger reason to vote to prevent an outcome where the winner lacks sufficient legitimacy. So I guess the strength of the argument depends on whether your candidate is a minority candidate or not.

2.5. The "Saving Democracy" Argument

The saving democracy argument says that the real power of voting isn’t just about influencing specific outcomes; it’s about keeping democracy healthy and stable by ensuring enough people participate. A high voter turnout signals that citizens are invested in their government. This engagement pressures elected leaders to be more responsive, knowing they are accountable to an active electorate. In short, even if your single vote doesn’t tip the scales, it still strengthens democracy by showing that people care and are watching.

But does it really? A single person's choice not to vote won’t collapse the system. Isn’t it more plausible that democracy’s health comes from knowing citizens could vote, not necessarily that they do. Many elections with low turnout still function well, so why is turnout considered essential to democracy’s quality?

In any case, the strength of this argument doesn’t seem to change depending on whether or not the election is deterministic.

3. Non-Instrumental reasons to vote

Most of us see voting as a way to influence election outcomes. But some philosophers argue there's also a broader, non-instrumental reasons to vote. Let’s look at some of them:

3.1. The Generalization Argument: A Kantian Perspective

What would happen if everyone abstained from voting? Wouldn’t democracy collapse? If democracy is inherently valuable, shouldn’t you vote to help maintain it?

This is a Kantian perspective on voting. Philosopher Immanuel Kant’s believed that one should only act on principles they’d be comfortable universalizing. In other words, an individual should only take actions they would be okay with everyone else taking. Applied to voting, if everyone adopted the principle “only vote when it’s in your interest,” democracy might cease to function as no one would feel obliged to participate regularly.

However, this argument faces some limitations. For instance, if we generalize that everyone must perform certain actions essential to society—like farming or building homes—it doesn’t logically mean everyone should be a farmer or a builder. Similarly, while democracy requires voters, it doesn’t require that every single person vote.

Furthermore, this argument may not apply to every reason for abstention. If someone genuinely believes that none of the parties represent their views, then abstention might seem principled rather than neglectful. For example, if a person adopts the principle “don’t vote if no candidate aligns with your political beliefs,” other people would vote and democracy could still function (assuming others do have candidates that align with their political beliefs).

3.2. The Free-Riding argument

Maybe a better way to frame this issue is through the lens of free-riding. Society generally frowns upon those who benefit from public goods, such as roads or clean air, without contributing to their maintenance, such as through taxes. Democracy can be seen as a public good that requires active participation to thrive. Voting, then, could be understood as a way of "paying your share" to sustain democracy, and abstaining from voting as a form of free-riding.

However, the free-riding perspective has its limitations when applied to voting. Voting, unlike taxes, doesn’t place an extra burden on others if someone abstains. In fact, abstention increases the influence of those who do vote. The less people vote, the more your vote influences the election.

In any case, the generalization argument doesn’t strengthen or weaken depending on whether or not the election is deterministic.

3.3. Voting as Self-Expression

Another argument for voting is that it’s valuable not just because of its instrumental effects, but because it is an expressive act—a way to affirm one’s identity, values, and support for democracy. This aligns with the ‘expressive theory of voting’, which holds that people vote not only to influence government but also as a means of self-expression. For example, even if sending flowers to a hospitalized friend doesn’t directly improve their health, it can still be seen as valuable because it serves as an expression of care and support.

Voting, in this sense, is viewed as a civic gesture, much like observing a two-minute silence on Remembrance Day. Citizens may feel a moral obligation to vote as an acknowledgment of democratic values and the historical sacrifices made to secure these rights. One could argue that by choosing not to vote a person is implicitly rejecting or downplaying the value of democratic participation.

But here too, questions arise. Unlike public gestures such as standing during a remembrance event, voting is anonymous and private, making it less visible as an expression of support for democracy. Furthermore, if voting were purely expressive, it wouldn’t explain the phenomenon of strategic voting, where individuals vote tactically rather than authentically, often supporting a candidate they dislike just to prevent a worse outcome.

In non-deterministic elections, like "Random Ballot," where the chance element reduces the incentive for strategic voting, it’s easier for people to vote authentically. In such cases, the expressive argument for voting could be seen as stronger, as individuals might feel freer to vote in alignment with their true preferences.

Conclusion

I made a little table to summarize the results:

The only argument that weakens our reason to vote in non-deterministic elections is the “polluting the polls argument”, which we looked at and subsequently dismantled in my last post. The rest of the arguments either vary in strength (depending on the circumstances), are equally strong, or are strengthened in non-deterministic elections.

So overall, it looks like the average voter has stronger reasons to vote in non-deterministic elections than in deterministic ones.

(If you want to see more detailed/academic versions of these arguments, check out the paper by me and Jobst Heitzig)

Philosopher Jason Brennan argues in his paper “Polluting The Polls: When Citizens Should Not Vote” that most people should abstain from voting. The argument boils down to the view that most people are not informed enough to vote well, and so should therefore not vote lest they “pollute the polls”.

The other political philosopher that writes about our duty to vote, who is also named Brennan, has argued that we don’t have a (positive) duty to vote, because the odds that our vote will make the difference in an election are so astronomically low. Which is true, the chance that your vote will be the one that tips the scale either one way or the other is negligible.
This can be used as a counter argument. If the odds of causing a positive outcome are low enough that it makes no practical difference, the same can be said about the odds of causing a negative outcome.

Nondeterministic elections

I’ve been thinking about this because I’ve been analyzing “nondeterministic” voting systems. These are voting systems that use an element of chance. So you might be familiar with some different voting systems, such as Plurality Voting, Approval Voting, Ranked Choice Voting…
Even though there are many different voting systems and some have much better theoretical properties, most systems used in practice are deterministic. This means the election result depends basically only on the votes cast (although in case of ties, a deterministic system might use a coin toss or similar random method just to break the tie).
In contrast, nondeterministic voting systems use chance for more than just breaking ties. The simplest example is the ‘Random Ballot’: each voter submits a ballot, and one ballot is randomly drawn to pick the winner.

One attractive feature of non-deterministic systems is their ability to reduce the ‘tyranny of the majority’. In most deterministic systems, controlling just 51% of the votes gives 100% power, leaving the other 49% with none. Some deterministic methods need even more than a simple majority to ensure a win: for instance, with the ‘Borda score’ system, you need two-thirds (a super-majority) of the votes.
With the ‘Random Ballot’ system, having 51% of the votes only gives you a 51% chance of winning. Any group’s share of the vote directly translates to their winning probability, making the distribution of power perfectly proportional. Historically, nondeterministic systems can be traced back to ancient Athenian democracy, where officials were chosen by lot to ensure equal participation and prevent the concentration of power.1

Not all non-deterministic systems are perfectly proportional. For example, in a method where ballots are drawn one by one until two ballots for the same candidate are drawn, the effective power of a group controlling X% of the votes would lie somewhere between the perfectly proportional 'Random Ballot' method and the step-function of most standard systems. This relationship can be visualized as a function:

For most winner-takes-all systems (conventional voting systems) it is a step function; if you have 51% of the vote you are in power 100% of the time (blue line). The random ballot (green line) is perfectly proportional (49% votes = 49% in power), and the method of drawing ballots until you’ve drawn the same candidate two times is somewhere in between (red line).

In a future blog post I will look at the (positive) arguments to vote in nondeterministic elections (which will be based on my paper on the subject), but for now I will tackle one concern with nondeterministic voting systems:

If they increase the influence of the average voter, doesn’t that increase the strength of the “Polluting the polls” argument from earlier?
Indeed it does, in an election where the average citizen has more influence, they also have more opportunities to create adverse outcomes. If we want to refute the “polluting the polls” argument in nondeterministic elections, we will need a different argument.

Share

Polluting the polls and minority groups

One issue with Brennan's argument is that it seems to suggest that minorities shouldn’t vote. Those who are least educated, and therefore (according to Brennan) should abstain from voting, often belong to disadvantaged groups, like the poor and minorities. Discrimination and lack of opportunities can limit their education, so they would be the ones who disproportionately have to refrain from voting.

Brennan acknowledges this but argues that while minorities have been poorly served, advocating for better education and opportunities doesn’t mean they should vote as much as other groups. He compares this to professions like surgery or law, where unfair advantages due to discrimination should be fixed by improving education and opportunities, not by allowing unqualified people to work in these fields. Brennan argues that allowing politically ignorant people to vote is similar. If minority groups are predominantly politically uninformed because of discrimination, efforts should focus on improving their situation rather than pushing them to vote, which might lead to bad decisions.

But is this such a good idea? If a minority group tends to abstain from voting, won’t the government simply ignore their needs? While Brennan suggests that educated experts could advocate for these communities, this is questionable. Many experts might prioritize other issues, and people usually understand problems that affect them directly better. So, if minorities disengage from voting, their concerns will probably be overlooked.

Is this level of knowledge feasible?

Another issue with Brennan’s argument is practicality. Getting the level of knowledge he thinks is necessary for voting seems impossible. General elections involve a wide range of topics like defense, taxes, healthcare, housing, crime, transportation, international relations, education, and a whole lot more. Even if you’re well-educated in some areas, you’ll be ignorant in others. It raises the question: Do you need to be an expert to vote, or is some level of knowledge enough? If so, where is the line?2

This becomes especially worrying if we remember the existence of the Dunning-Kruger effect, which shows that people who lack competence often overestimate their knowledge, and vice-versa. If people followed Brennan’s advice, those ignorant of their lack of knowledge would keep voting, while well-educated people might think they’re not competent enough and abstain. This could result in a less-informed electorate, with more ignorant voters continuing to vote due to their overconfidence, while the educated abstain.

Conclusion

Because of these issues with practicality and minority disenfranchisement, I don’t find the “polluting the polls” argument very convincing. Which means that if we want to undermine the reasons to vote (in nondeterministic voting systems) we will need other arguments. I will look at some of these arguments in a future blog post.

Crossposted from our post on the EA forum.
This post is about technical AI-safety and the latter half will contain much more formalisms than usual. Feel free to skip it if you’re not into that, I will return to more informal blogging in the next post. For some reason it only wants to show formal notation if I have a large LaTeX block in the beginning and no footnote links. If the notation doesn’t show up in your browser you can click on the EA forum link.

Summary: This post documents research by SatisfIA, an ongoing project on non-maximizing, "aspiration-based" designs for AI agents that fulfill goals specified by constraints ("aspirations") rather than maximizing an objective function​​. We aim to contribute to AI safety by exploring design approaches and their software implementations that we believe might be promising, but neglected or novel. Our approach is roughly related to, but largely complementary to, concepts like Decision Transformers, Active Inference, Quantilization and Satisficing (or soft optimization in general).

It's divided into two parts, the informal introduction and the formal introduction.

First, the informal introduction describes the motivation for the research, our working hypotheses, and our theoretical framework. It does not contain results and can safely be skipped if you want to get directly into the actual research.

Second, the formal introduction presents the simplest form of the type of aspiration-based algorithms we study. We do this for a simple form of aspiration-type goals: making the expectation of some variable equal to some given target value.¹

At the end of the post we give an overview of the projects status (we're looking for collaborators!) and have a small glossary of terms.

This post is the summary for an emerging sequence on LessWrong and the Alignment Forum, authored by us and other members of the SatisfIA project team. If you wish to learn more about our approach you can read the full sequence there.
 

🄸🄽🄵🄾🅁🄼🄰🄻 🄸🄽🅃🅁🄾🄳🅄🄲🅃🄸🄾🄽

Motivation

We share a general concern regarding the trajectory of Artificial General Intelligence (AGI) development, particularly the risks associated with creating AGI agents designed to maximize objective functions. We have two main concerns:

AGI development might be inevitable

The development of AGI seems inevitable due to its immense potential economic, military, and scientific value. This inevitability is driven by:

• Economic and military incentives: The prospect of gaining significant advantages in efficiency, capability, and strategic power makes the pursuit of AGI highly attractive to both private and state actors​​.

• Scientific curiosity: The inherent human drive to understand and replicate intelligence propels the advancement of AGI research.

It might be impossible to implement an objective function the maximization of which would be safe

The conventional view on A(G)I agents (see, e.g., Wikipedia) is that they should aim to maximize some function of the state or trajectory of the world, often called a "utility function", sometimes also called a "welfare function". It tacitly assumes that there is such an objective function that can adequately make the AGI behave in a moral way. However, this assumption faces several significant challenges:

• Moral ambiguity: The notion that a universally acceptable, safe utility function exists is highly speculative. Given the philosophical debates surrounding moral cognitivism and moral realism and similar debates in welfare economics, it is possible that there are no universally agreeable moral truths, casting doubt on the existence of a utility function that encapsulates all relevant ethical considerations​​.

• Historical track-record: Humanity's long-standing struggle to define and agree upon universal values or ethical standards raises skepticism about our capacity to discover or construct a comprehensive utility function that safely governs AGI behavior (Outer Alignment)​​ in time.

• Formal specification and Tractability: Even if a theoretically safe and comprehensive utility function could be conceptualized, the challenges of formalizing such a function into a computable and tractable form are immense. This includes the difficulty of accurately capturing complex human values and ethical considerations in a format that can be interpreted and tractably evaluated by an AGI​​.

• Inner Alignment and Verification: Even if such a tractable formal specification of the utility function has been written down, it might not be possible to make sure it is actually followed by the AGI because the specification might be extremely complicated and computationally intractable to verify that the agent's behavior complies with it. The latter is also related to Explainability.

Given these concerns, the implication is clear: AI safety research should spend considerably more effort on identifying and developing AGI designs that do not rely on the maximization of an objective function. Given our impression that currently not enough researchers pursue this, we chose to work on it, complementing existing work on what some people call "mild" or "soft optimization", such as Quantilization or Bayesian utility meta-modeling. In contrast to the latter approaches, which are still based on the notion of a "(proxy) utility function", we explore the apparently mostly neglected design alternative that avoids the very concept of a utility function altogether.2 The closest existing aspiration-based design we know of is Decision Transformer, which is inherently a learning approach. To complement this, we focus on a planning approach. 
 

Working hypotheses

We use the following working hypotheses during the project, which we ask the reader to adopt as hypothetical premises while reading our text (you don't need to believe them, just assume them to see what might follow from them).

1: We should not allow the maximization of any function of the state or trajectory of the world

Following the motivation described above, our primary hypothesis posits that it must not be allowed that an AGI aims to maximize any form of objective function that evaluates the state or trajectory of the world. (This does not necessarily rule out that the agent employs any type of optimization of any objective function as part of its decision making, as long as that function is not only a function of the state or trajectory of the world. For example, we might allow some form of constrained maximization of  entropy or minimization of free energy or the like, which are functions of a probabilistic policy rather than of the state of the world.)

2: The core decision algorithm must be hard-coded

The 2nd hypothesis is that to keep an AGI from aiming to maximize some utility function, the AGI agent must use a decision algorithm to pick actions or plans on the basis of the available information, and that decision algorithm must be hard-coded and cannot be allowed to emerge as a byproduct of some form of learning or training process. This premise seeks to ensure that the foundational principles guiding the agent's decision making are known in advance and have verifiable (ideally even provable) properties. This is similar to the "safe by design" paradigm and implies a modular design where decision making and knowledge representation are kept separate. In particular, it rules out monolithic architectures (like using only a single transformer or other large neural network that represents a policy, and a corresponding learning algorithm).

3: We should focus on model-based planning first and only consider learning later

Although in reality, an AGI agent will never have a perfect model of the world and hence also needs some learning algorithm(s) to improve its imperfect understanding of the world on the basis of incoming data, our 3rd hypothesis is that for the design of the decision algorithm, it is helpful to hypothetically assume in the beginning that the agent already possesses a fixed, sufficiently good probabilistic world model that can predict the consequences of possible courses of action, so that the decision algorithm can choose actions or plans on the basis of these predictions. The rationale is that this "model-based planning" framework is simpler and mathematically more convenient to work with and allows us to address the fundamental design issues that arise even without learning, before addressing additional issues related to learning. (see also Project history)

4: There are useful generic, abstract safety criteria largely unrelated to concrete human values

We hypothesize the existence of generic and abstract safety criteria that can enhance AGI safety in a broad range of scenarios. These criteria focus on structural and behavioral aspects of the agent's interaction with the world, largely independent of the specific semantics or contextual details of actions and states and mostly unrelated to concrete human values. Examples include the level of randomization in decision-making, the degree of change introduced into the environment, the novelty of behaviors, and the agent's capacity to drastically alter world states. Such criteria are envisaged as broadly enhancing safety, very roughly analogous to guidelines such as caution, modesty, patience, neutrality, awareness, and humility, without assuming a one-to-one correspondence to the latter.

5: One can and should provide certain guarantees on performance and behavior

We assume it is possible to offer concrete guarantees concerning certain aspects of the AGI's performance and behavior. By adhering to the outlined hypotheses and safety criteria, our aim is to develop AGI systems that exhibit behaviour that is in some essential respects predictable and controlled, reducing risk while fulfilling their intended functions within safety constraints.

6: Algorithms should be designed with tractability in mind, ideally employing Bellman-style recursive formulas

Because the number of possible policies grows exponentially fast with the number of possible states, any algorithm that requires scanning a considerable portion of the policy space (like, e.g., "naive" quantilization over full policies would) soon becomes intractable in complex environments. In optimal control theory, this problem is solved by exploiting certain mathematical features of expected values that allow to make decisions sequentially and compute the relevant quantities (V- and Q-values) in an efficient recursive way using the Hamilton–Jacobi–Bellman equation. Based on our preliminary results, we hypothesize that a very similar recursive approach is possible in our aspiration-based framework in order to keep our algorithms tractable in complex environments. 

Theoretical framework

Agent-environment interface. Our theoretical framework and methodological tools draw inspiration from established paradigms in artificial intelligence (AI), aiming to construct a robust understanding of how agents interact with their environment and make decisions. At the core of our approach is the standard agent–environment interface, a concept that we later elaborate through standard models like (Partially Observed) Markov Decision Processes. For now, we consider agents as entirely separate from the environment — an abstraction that simplifies early discussions, though we acknowledge that the study of agents which are a part of their environment is an essential extension for future exploration.

Simple modular design. To be able to analyse decision-making more clearly, we assume a modular architecture that divides the agent into two main components: the world model and the decision algorithm. The world model represents the agent's understanding and representation of its environment. It encompasses everything the agent knows or predicts about the environment, including how it believes the environment responds to its actions. The decision algorithm, on the other hand, is the mechanism through which the agent selects actions and plans based on its world model and set goals. It evaluates potential actions and their predicted outcomes to choose courses of action that appear safe and aligned with its aspirations.

Studying a modular setup can also be justified by the fact that some leading figures in AI push for modular designs even though current LLM-based systems are not yet modular in this way.

Information theory. For formalizing generic safety criteria, we mostly employ an information theoretic approach, using the mathematically very convenient central concept of Shannon entropy (unconditional, conditional, mutual, directed, etc.) and derived concepts (e.g., channel capacity). This also allows us to compare and combine our findings with approaches based on the free energy principle and active inference. where goals are formulated as desired probability distributions of observations.

High-level structure of studied decision algorithms

The decision algorithms we currently study have the following high-level structure:

1. The agent accepts tasks given as aspirations, i.e., as constraints on the probability distribution of certain task-relevant evaluation metrics. The aspiration will be updated over time in response to achievements, good or bad luck, and other observations.

2. Whenever the agent must decide what the next action is, it performs a number of steps.

   1. First, it uses its world model to estimate what evolutions of the evaluation metrics are still possible from the current state on, and after taking each possible action in the current state. This defines the state's and all possible actions' "feasibility sets".

   2. It then compares each possible action's feasibility set with its current aspiration and decides whether it would have to adjust its aspiration if it were to take that action, and how to adjust its aspiration in that case. This defines all possible actions' "action-aspirations".

   3. It also evaluates all possible actions using one or more safety and performance criteria and picks a combination of one or more candidate actions that appear rather safe.

   4. It then randomizes between these candidate actions in a certain way.

   5. Based on the next observation, it updates its beliefs about the current state.

   6. Finally, in particular when the environment is partially unpredictable and the agent has had some form of good or bad luck, it potentially adjusts its aspiration again ("aspiration propagation").

Depending on the type of aspiration, the details can be designed so that the algorithm comes with certain guarantees about the fulfillment of the goal. 
 

🄵🄾🅁🄼🄰🄻   🄸🄽🅃🅁🄾🄳🅄🄲🅃🄸🄾🄽

Assumptions

In line with the working hypotheses, we assume more specifically the following in this post:

• The agent is a general-purpose AI system that is given a potentially long sequence of tasks, one by one, which it does not know in advance. Most aspects of what we discuss focus on the current task only, but some aspects relate to the fact that there will be further, unknown tasks later (e.g., the question of how much power the agent shall aim to retain at the end of the task).

• It possesses an overall world model that represents a good enough general understanding of how the world works.

• Whenever the agent is given a task, an episode begins and its overall world model provides it with a (potentially much simpler) task-specific world model that represents everything that is relevant for the time period until the agent gets a different task or is deactivated, and that can be used to predict the potentially stochastic consequences of taking certain actions in certain world states.

• That task-specific world model has the form of a (fully observed) Markov Decision Process (MDP) that however does not contain a reward function R but instead contains what we call an evaluation function related to the task (see 2nd to next bullet point).

• As a consequence of a state transition, i.e., of taking a certain action a in a certain state s and finding itself in a certain successor state s′, a certain task-relevant evaluation metric changes by some amount. Importantly, we do not assume that the evaluation metric inherently encodes things of which more is better. E.g., the evaluation metric could be global mean temperature, client's body mass, x coordinate of the agent's right thumb, etc. We call the step-wise change in the evaluation metric the received Delta in that time step, denoted δ. We call its cumulative sum over all time steps of the episode the Total, denoted τ. Formally, Delta and Total play a similar role for our aspiration-based approach as the concepts of "reward" and "return" play for maximization-based approaches. The crucial difference is that our agent is not tasked to maximize Total (since the evaluation metric does not have the interpretation of "more is better") but to aim for some specific value of the Total.

• The evaluation function contained in the MDP specifies the expected value of δ for all possible transitions: Eδ(s,a,s′).

First challenge: guaranteeing the fulfillment of expectation-type goals

The challenge in this post is to design a decision algorithm for tasks where the agent's goal is to make the expected (!) Total equal (!) a certain value E∈R which we call the aspiration value.  This is a crucial difference from a "satisficing" approach that would aim to make expected Total at least as large as E and would thus still be happy to maximize Total. Later we consider other types of tasks, both less restrictive ones (including those related to satisficing) and more specific ones that also care about other aspects of the resulting distribution of Total or states.

It turns out that we can guarantee the fulfillment of this type of goal under some weak conditions!

Notice that a special case of such expectation-type goals is making sure that the probability of reaching a certain set of acceptable terminal states equals a certain value, because we can simply assume that each such acceptable terminal state gives 1 Delta and all others give zero Delta. We will come back to that special case later when discussing aspiration intervals.

Example: Shopping for apples

• The agent is a certain household's AI butler. Among all kinds of other tasks, roughly once a week it is tasked to go shopping for apples. When it gets this task in the morning, an apple shopping episode begins, which ends when the agent returns to the household to get new instructions or is called by a household member on its smartphone, or when its battery runs empty.

• The relevant evaluation metric for this task is the household's and agent's joint stock of apples. It changes by some Delta each time the agent gets handed some apples by a merchant or takes some from the supermarket's shelf or when some apples fall from its clumsy hands or get stolen by some robot hater (or when some member of the household eats an apple).

• As the household's demand for apples is 21 apples per week on average, the task in a single apple shopping episode is to buy a certain number E of apples in expectation. They also want to maintain some small stock for hard times, say about 5 apples. So the household's policy is to set the aspiration to E=21+(5−x), where x is its current stock of apples. As this is a recurrent task, it is perfectly fine if this aspiration is only fulfilled in expectation if the Total doesn't vary too much, since over many weeks (and assuming long-lived apples), the deviations will probably average out and the stock will vary around 26 apples right after a shopping mission and 5 apples just before the next shopping mission. In reality, the household would of course also want the variance to be small, but that is a performance criterion we will only add later.

Possible generalizations (can be skipped safely)

In later posts, we will generalize the above assumptions in the following ways:

• Instead of as a single value E∈R that expected Total shall equal, the task can be given as an aspiration interval E=[e―,e―] into which expected Total shall fall (e.g., "buy about 21+(5−x)±2 apples").

• Instead of a single evaluation metric (stock of apples), there can be d>1 many evaluation metrics (stock of apples, stock of pears, and money in pocket), and the task can be given as a convex aspiration set E⊆Rd (e.g., buy at least two apples and one pear but don't spend more than 1 Euro per item).

• Instead of in terms of evaluation metrics, the task could be given in terms of the terminal state of the episode, by specifying a particular "desired" state, a set of "acceptable" states, a desired probability distribution of terminal states, or a set of acceptable probability distribution of terminal states. For example, demanding that the expected number of stocked apples after the episode be 26 is the same as saying that all probability distributions of terminal states are acceptable that have the feature that the expected number of stocked apples is 26. A more demanding task would then be to say that only those probability distributions of terminal states are acceptable for which the expected number of stocked apples is 26, its standard deviation is at most 3, its 5 per cent quantile is at least 10, and the 5 per cent quantile of the number of surviving humans is at least 8 billion.

• The world model might have a more general form than an MDP: to represent different forms of uncertainty, it might be an only partially observed MDP (POMDP), or an ambiguous POMDP (APOMDP); to represent complex tasks, it might be a hierarchical MDP whose top-level actions (e.g., buy 6 apples from this merchant) are represented as lower-level MDPs with lower-level aspirations specified in terms of auxiliary evaluation metrics (e.g., don't spend more than 5 minutes waiting at this merchant), and its lowest level might also have continuous rather than discrete time (if it represents, e.g., the continuous control of a robot's motors).

Notation

We focus on a single episode for a specific task.

Environment. We assume the agent's interaction with the environment consists of an alternating sequence of discrete observations and actions. As usual, we formalize this by assuming that after each observation, the agent chooses an action a and then "calls" a (potentially stochastic) function step(a) provided by the environment that returns the next observation.

World model. The episode's world model, M, is a finite, acyclic MDP. The model's discrete time counter, t, advances whenever the agent makes an observation. From the sequence of observations made until time t, the world model constructs a representation of the state of the world, which we call the model state or simply the state and denote by st, and which also contains information about the time index t itself. The set of all possible (model) states is a finite set S. A subset S⊤ of states is considered terminal. If the state is terminal, the episode ends without further action or Delta, which represents the fact that the agent becomes inactive until given a new task. Otherwise, the agent chooses an action at∈A. The world model predicts the consequences of each possible action by providing a probability distribution PM(st+1|st,at) for the successor state st+1. It also predicts the expected Delta for the task-relevant evaluation metric as a function of the state, action, and successor state: Eδt+1=Eδ(st,at,st+1).

The following two graphs depict all this. Entities occurring in the world model are blue, those in the real world red, green is what we want to design, and dotted things are unknown to the agent:

We hide the process of constructing the next state from the previous state, action, and next observation by simply assuming that the agent can call a version of the function step that is given the current state and action and returns the successor state constructed from the next observation: st+1←step(st,at).

Goal. The goal is given by an aspiration value E(s0)∈R. The task is to choose actions so that the expected Total, 

Eτ=E(s0,a0,s1,a1,…)∑tEδ(st,at,st+1),

 equals E(s0).

Auxiliary notation for interval arithmetic. We will use the following abbreviations:

• x:λ:z=x(1−λ)+λz 
  (interpolation between x and z),

• x∖y∖z=y−xz−x 
  (relative position of y in interval [x,z], with the convention that 00=12),

• x[y]z=min{max{x,y},z} 
  ("clipping'' y to interval [x,z]).

Sequential decisions based on propagated aspirations

Main idea

 Our agent will achieve the goal by

• propagating the aspiration along the trajectory as we go from states st via actions at to successor states st+1, leading to an alternating sequence of state aspirations E(st) and action aspirations E(st,at).

• sequentially deciding on the next action at on the basis of the current state aspiration E(st) and suitably chosen action-aspirations E(st,a) for all possible actions a∈A.

For both aspiration propagation and decision making, the agent uses some auxiliary quantities that it computes upfront at the beginning of the episode from the world model as follows.

Feasibility intervals

Similar to what is done in optimal control theory, the agent computes the V- and Q-functions of the hypothetical policy that would maximize expected Total, here denoted V― and Q―, by solving the respective Bellman equations 

V―(s)=maxa∈AQ―(s,a),Q―(s,a)=Es′∼PM(⋅|s,a)(Eδ(s,a,s′)+V―(s′)),

with V―(s)=0 for terminal states s∈S⊤. It also computes the analogous quantities for the hypothetical policy that would minimize expected Total, denoted V― and Q―: 

V―(s)=mina∈AQ―(s,a),Q―(s,a)=Es′∼PM(⋅|s,a)(Eδ(s,a,s′)+V―(s′)),

with V―(s)=0 for terminal states s∈S⊤. These define the state's and action's feasibility intervals,

(F)V(s)=[V―(s),V―(s)],Q(s,a)=[Q―(s,a),Q―(s,a)].

The eventual use of these intervals will be to rescale aspirations from step to step. Before we come to that, however, we can already prove a first easy fact about goals of the type "make sure that expected Total equals a certain value":

Lemma: Trivial guarantee

If the world model predicts state transitions correctly, then there is a decision algorithm that fulfills the goal Eτ=E(s0) if and only if the episode's starting aspiration E(s0) is in the initial state's feasibility interval V(s0).

Proof. The values V―(s0) and V―(s0) are, by definition, the expected Total of the maximizing resp. minimizing policy, and hence it is clear that there cannot be a policy which attains E(s0) in expectation if E(s0) is larger than V―(s0) or smaller than V―(s0).

Conversely, assuming that E(s0) lies inside the interval V(s0), the following procedure fulfills the goal:

• We compute the relative position of E(s0) inside V(s0), p=E(s0)−V―(s0)V―(s0)−V―(s0)∈[0,1].

• With probability p, we use the maximizing policy π― throughout the episode, and with probability 1−p, we use the minimizing policy π― throughout the episode.

This fulfills the goal, since the correctness of the model implies that, when using π― or π―, we actually get an expected Total of V―(s0) resp. V―(s0). ◻

Of course, using this "either maximize or minimize the evaluation metric" approach would be catastrophic for safety. For example, if we tasked an agent with restoring Earth's climate to a pre-industrial state, using as our evaluation metric the global mean temperature, this decision algorithm might randomize, with carefully chosen probability, between causing an ice age and inducing a runaway greenhouse effect! This is very different from what we want, which is something roughly similar to pre-industrial climate.

Another trivial idea is to randomize in each time step t between the action with the largest Q―(st,a) and the one with the smallest Q―(st,a), using a fixed probability p′ resp.  1−p′. Since expected Total is a continuous function of p′ which varies between V―(s0)and V―(s0), by the Intermediate Value Theorem there exists some value of p′ for which this algorithm gives the correct expected Total; however, it is unclear how to compute the right p′ in practice.

If the episode consists of many time steps, this method might not lead to extreme values of the Total, but it would still make the agent take an extreme action in each time step. Intuition also suggests that the agent's behavior would be less predictable and fluctuate more than in the first version, where it consistently maximizes or minimizes after the initial randomization, and that this is undesirable.

So let us study more intelligent ways to guarantee that Eτ=E(s0).

Decision Algorithm 1: steadfast action-aspirations, rescaled state-aspirations

In order to avoid extreme actions, our actual decision algorithm chooses "suitable" intermediate actions which it expects to allow it to fulfill the goal in expectation. When in state s, it does so by

• setting action-aspirations E(s,a) for each possible action a∈A on the basis of the current state-aspiration E(s) and the action's feasibility interval Q(s,a), trying to keep E(s,a) close to E(s),

• choosing in some arbitrary way an "under-achieving" action a− and an "over-achieving" action a+ w.r.t. E(s) and these computed action-aspirations E(s,a),

• choosing probabilistically between a− and a+ with suitable probabilities,

• executing the chosen action a and observing the resulting successor state s′, and

• propagating the action-aspiration E(s,a) to the new state-aspiration E(s′) by rescaling between the feasibility intervals of a and s′.

More precisely: First compute (or learn) the functions V―,V―,Q―, and Q―. Then, given state s and a feasible state-aspiration E(s)∈V(s),

1. For all available actions a∈A, compute action-aspirations E(s,a)=Q―(s,a)[E(s)]Q―(s,a)∈Q(s,a).

2. Pick some actions a−,a+∈A with E(s,a−)≤E(s)≤E(s,a+); these necessarily exist because E(s)∈V(s).

3. Compute p=E(s,a−)∖E(s)∖E(s,a+)∈[0,1].

4. With probability p, let a=a+, otherwise let a=a−.

5. Execute action a in the environment and observe the successor state s′←step(s,a).

6. Compute λ=Q―(s,a)∖E(s,a)∖Q―(s,a)∈[0,1] 
   and the successor state's state-aspiration E(s′)=V―(s′):λ:V―(s′)∈V(s′).

If we add the state- and action aspirations as entities to the diagram of Fig. 1, we get this:

Example: Shopping for apples, revisited with math

We return to the apple-shopping scenario mentioned above, which we model by the following simple state-action diagram:

Our agent starts at home (state s) and wishes to obtain a certain number of apples, which are available at a market (state m). It can either walk to the market (action a), which will certainly succeed, or take public transportation (action b), which gives a 2/3 chance of arriving successfully at the market and a 1/3 chance of not reaching the market before it closes and returning home empty-handed. Of course, the agent can also decide to take the null action (action c) and simply stay home the entire day doing nothing.
Once it reaches the market m, the agent can buy either one or two packs of three apples (actions m1 and m2, respectively) before returning home at the end of the day (state t).

To apply Algorithm 1, we first compute the Q- and V-functions for the maximizing and minimizing policies. Since there are no possible cycles in this environment, straightforwardly unrolling the recursive definitions from the back ("backward induction") yields:

(The asterisks* mark which actions give the V values)

Suppose that the agent is in the initial state s and has the aspiration E(s)=2.5.

1.  First, we calculate the action-aspirations: if I were to take a certain action, what would I aspire to? Here, the state-aspiration E(s) lies within the action's feasibility set Q(s,b), so the action-aspiration E(s,b) is simply set equal to E(s). By contrast, the intervals Q(s,a) and Q(s,c) do not contain the point E(s), and so E(s) is clipped to the nearest admissible value:

1. Next, we choose an over-achieving action a+ and an under-achieving action a−.
   Suppose we arbitrarily choose actions c (do nothing) and a (walk to the market).

2. We calculate the relative position of the aspiration E(s) between E(s,c) and E(s,a):  p=E(s,c)∖E(s)∖E(s,a)=56.

3. We roll a die and choose our next action to be either c with probability 1−p=16 or a with probability p=56.

4. We take the chosen action, walking to the market or doing nothing depending on the result of the die roll, and observe the consequences of our actions! In this case, there are no surprises, as the transitions are deterministic.

5. We rescale the action-aspiration to determine our new state-aspiration. If we chose action a, we deterministically transitioned to state m, and so the feasibility interval Q(s,a) is equal to V(m) and no rescaling is necessary (in other words, the rescaling is simply the identity map): we simply set our new state-aspiration to be E(m)=E(s,a)=3. Likewise, if we took action c, we end up in state t with state-aspiration E(t)=0.

Suppose now that we started with the same initial aspiration E(s)=2.5, but instead chose action b as our over-achieving action in step 2. In this case, algorithm execution would go as follows:

1. Determine action-aspirations as before.

2. Choose a−=c,a+=b.

3. Since E(s,b) is exactly equal to our state-aspiration E(s) and E(s,a) is not, p is 1!

4. Hence, our next action is deterministically b.

5. We execute action b and observe whether public transportation is late today (ending up in state t) or not (which brings us to state m).

6. For the rescaling, we determine the relative position of our action-aspiration in the feasibility interval: λ=Q―(s,b)∖E(s,b)∖Q―(s,b)=1/4.
   If we ended up in state m, our new state-aspiration is then E(m)=V―(m):λ:V―(m)=3.75; if we ended up in state t, the state-aspiration is E(t)=0.

These examples demonstrate two cases:

• If neither of the action feasibility intervals Q(a+) nor Q(a−) contain the state aspiration E(s), we choose between taking action a+ and henceforth minimizing, or taking action a− and henceforth maximizing, with the probability p that makes the expected total match our original aspiration E(s).

• If exactly one of the feasibility intervals Q(s,a+) or Q(s,a−) contains E(s), then we choose the corresponding action with certainty, and propagate the aspiration by rescaling it.

There is one last case, where both feasibility intervals contain E(s); this is the case, for example, if we choose E(s)=3.5 in the above environment. Execution then proceeds as follows:

1. We determine action-aspirations, as shown here:

1. Suppose now that we choose actions a and b as a+ and a−. (If action c is chosen, then we are in the second case shown above.)

2. p is defined as the relative position of E(s) between E(s,a−) and E(s,a+), but in this case, these three values are equal! We have chosen above (see auxiliary notation) that p=12 in this case, but any other probability would also be acceptable.

3. We toss a coin to decide between actions a and b.

4. The chosen action is performed, either taking us deterministically to the market m if we chose action a or randomizing between m and t if we chose action b.

5. We propagate the action-aspiration to state-aspiration as before. If we chose action a, then we have Q(s,a)=V(m) and so the new state-aspiration is E(m)=E(s,a)=3.5. If we chose action b, then the new state-aspiration is E(m)=V―(s,m):34:V―(s,m)=5.25 if we reached the market and E(t)=0 otherwise.

Now that we have an idea of how the decision algorithm works, it is time to prove its correctness.

Theorem: Algorithm 1 fulfills the goal

If the world model predicts state transitions correctly and the episode-aspiration E(s0) is in the initial state's feasibility interval, E(s0)∈V(s0), then decision algorithm 1 fulfills the goal Eτ=E(s0).

Proof.

First, let us observe that algorithm 1 preserves feasibility: if we start from state s0 with state-aspiration E(s0)∈V(s0),  then for all states s and actions a visited, we will have E(s0)∈V(s0) and E(s,a)∈Q(s,a).
This statement is easily seen to be true for action-aspirations, as they are required to be feasible by definition in step 1, and correctness for state-aspirations follows from the definition of E(s′) in step 6.

Let us now denote by Vπ1(s,e) the expected Total obtained by algorithm 1 starting from state s with state-aspiration E(s)=e, and likewise by Qπ1(s,a,e) the expected Total obtained by starting at step 5 in algorithm 1 with action-aspiration E(s,a)=e.

Since the environment is assumed to be acyclic and finite, we can straightforwardly prove the following claims by backwards induction:

1. For any state s and any state-aspiration E(s) belonging to the feasibility interval V(s), we indeed have Vπ1(s,E(s))=E(s).

2. For any state-action pair (s,a) and any action-aspiration E(s,a) belonging to the feasibility interval Q(s,a), the expected future Total Qπ1(s,a,E(s,a)) is in fact equal to E(s,a).

We start with claim 1. The core reason why this is true is that, for non-terminal states s, we chose the right p in step 3 of the algorithm:Vπ1(s,E(s))=(1−p)⋅Qπ1(s,a−,E(s,a−))+p⋅Qπ1(s,a+,E(s,a+))=(1−p)⋅E(s,a−)+p⋅E(s,a+)by induction hypothesis=E(s,a−):p:E(s,a+)=E(s)because p=E(s,a−)∖E(s)∖E(s,a+).

Claim 1 also serves as the base case for our induction: if s is a terminal state, then Q(s) is an interval made up of a single point, and in this case claim 1 is trivially true.

Claim 2 requires that the translation between action-aspirations, chosen before the world's reaction is observed, and subsequent state-aspirations, preserves expected Total. The core reason why this works is the linearity of the rescaling operations in step 6:

Qπ1(s,a,E(s,a))=∑s′∈SPM(s′∣s,a)(Eδ(s,a,s′)+Vπ1(s′,E(s′)))assuming correct worldmodel=∑s′PM(s′∣s,a)⋅(Eδ(s,a,s′)+E(s′))by induction hypothesis=∑s′PM(s′∣s,a)⋅(Eδ(s,a,s′)+V―(s′):λ:V―(s′))by definition in step 6=∑s′PM(s′∣s,a)⋅((Eδ(s,a,s′)+V―(s′)):λ:(Eδ(s,a,s′)V―(s′)))=(∑s′PM(s′∣s,a)⋅(Eδ(s,a,s′)+V―(s′))):λ:(∑s′PM(s′∣s,a)⋅(Eδ(s,a,s′)V―(s′)))=Q―(s,a):λ:Q―(s,a)using the Bellman equation for Q=E(s,a)because λ=Q―(s,a)∖E(s,a)∖Q―(s,a) by definition

This concludes the correctness proof of algorithm 1. ◻

Notes

• It might seem counterintuitive that the received Delta (or at least the expected Delta) is never explicitly used in propagating the aspiration. The proof above however shows that it is implicitly used when rescaling from Q(s,a) (which contains Es′δ(s,a,s′)) to V(s′) (which does not contain it any longer). 

• We clip the state aspiration to the actions' feasibility intervals to keep the variability of the resulting realized Total low. If the state aspiration is already in the action's feasibility interval, this does not lead to extreme actions. However, if it is outside an action's feasibility interval, it will be mapped onto one endpoint of that interval, so if that action is actually chosen, the subsequent behavior will coincide with a maximizer's or minimizer's behavior from that point on.  

• The intervals V(s) and Q(s,a) used for clipping and rescaling can also be defined in various other ways than equation (F) to avoid taking extreme actions even more, e.g., by using two other, more moderate reference policies than the maximizer's and minimizer's policy.

• Rather than clipping the state aspiration to an action's feasibility interval, one could also rescale it from the state's to the action's feasibility interval, E′(s,a)←Q―(s,a):(V―(s)∖E(s)∖V―(s)):Q―(s,a). This would avoid extreme actions even more, but would overly increase the variability of received Total even in very simple environments.

• Whatever rule the agent uses to set action-aspirations and choose candidate actions, it should be for each action independently, because looking at all possible pairs or even all subsets or lotteries of actions would increase the algorithm's time complexity more than we think is acceptable if we want it to remain tractable. (We tackle the question of algorithm complexity in the main sequence).  

What can we do with this?

We can extend the basic algorithm by adding criteria for choosing the two candidate actions the algorithm mixes, and by generalizing the goal from making the expected Total equal a particular value to making it fall into a particular interval. 

The interesting question now is what criteria the agent should use to pick the two candidate actions a−,a+ in step 2. We might use this freedom to choose actions in a way that increases safety, e.g., by choosing randomly as a form a "soft optimization" or by incorporating safety criteria like limiting impact.

Let's look at some simple illustrative examples of performance and safety criteria (More useful criteria will be discussed in the full LW sequence).  

Using the gained freedom to increase safety

After having introduced the basic structure of our decision algorithms we will focus on the core question: How shall we make use of the freedom gained from having aspiration-type goals rather than maximization goals? 

After all, while there is typically only a single policy that maximize some objective function (or very few, more or less equivalent policies), there is typically a much larger set of policies that fulfill some constraints (such as the aspiration to make the expected Total equal some desired value). 

More formally: Let us think of the space of all (probabilistic) policies, Π, as a compact convex subset of a high-dimensional vector space with dimension d≫1 and Lebesgue measure μ. Let us call a policy π∈Π successful iff it fulfills the specified goal, G, and let ΠG⊆Π be the set of successful policies. Then this set has typically zero measure, μ(ΠG)=0, and low dimension, dim(ΠG)≪d, if the goal is a maximization goals, but it has large dimension, dim(ΠG)≈d, for most aspiration-type goals. 

E.g., if the goal is to make expected Total equal an aspiration value, Eτ=E, we typically have dim(ΠG)=d−1 but still μ(ΠG)=0. At the end of this post, we discuss how the set of successful policies can be further enlarged by switching from aspiration values to aspiration intervals to encode goals, which makes the set have full dimension, dim(ΠG)=d, and positive measure, μ(ΠG)>0.

What does that mean? It means we have a lot of freedom to choose the actual policy π∈ΠG that the agent should use to fulfill an aspiration-type goal. We can try to use this freedom to choose policies that promise to be rather safe than unsafe according to some generic safety metric, similar to the impact metrics used in reward function regularization for maximizers.

Depending on the type of goal, we might also want to use this freedom to choose policies that fulfill the goal in a rather desirable than undesirable way according to some goal-related performance metric. 

In this post, we will illustrate this with only very few, "toy" safety metrics, and one rather simple goal-related performance metric, to exemplify how such metrics might be used in our framework. In the LW sequence we discuss more sophisticated and hopefully more useful safety metrics. 

Let us begin with a simple goal-related performance metric since that is the most straightforward.

Simple example of a goal-related performance metric 

Recall that in step 2 of the basic algorithm, we could make the agent pick any action a− whose action-aspiration is at most as large as the current state-aspiration, E(s,a−)≤E(s), and it can also pick any other action, a+, whose action-aspiration is at least as large as the current state-aspiration, E(s,a+)≥E(s). This flexibility is because in steps 3 and 4 of the algorithm, the agent is still able to randomize between these two actions a−,a+ in a way that makes expected Total, Eτ, become exactly E(s).

If one had an optimization mindset, one might immediately get the idea to not only match the desired expectation for the Total, but also to minimize the variability of the Total, as measured by some suitable statistic such as its variance. In a sequential decision making situation like an MDP, estimating the variance of the Total requires a recursive calculation that anticipates future actions, which can be done but is not trivial (see the LW sequence).

Let us instead look at a simpler metric to illustrate the basic approach: the (relative, one-step) squared deviation of aspiration, which is very easy to compute:

SDA(s,a):=(E(s,a)−E(s))2(V―(s)−V―(s))2∈[0,1].

The rationale for this is that at any time, the action aspirations E(s,a±) are the expected values of Total-to-go conditional on taking action a±, so keeping them close to E(s) will tend to lead to a smaller variance. Indeed, that variance is lower bounded by (E(s,a−)−E(s))2:p:(E(s,a+)−E(s))2, where p is the probability for choosing a+ calculated in step 3 of Algorithm 1.  

If the action space is rich enough, there will often be at least one action the action-aspiration of which equals the state-aspiration, E(s,a)=E(s), because the state-aspiration is contained in that action's feasibility interval, E(s)∈Q(s,a). There might even be a large number of such actions. This will in particular be the case in the early steps of an episode. This is because often one can distribute the amount of effort spent on a task more or less flexibly over time; as long as there is enough time left, one might start by exerting little effort and then make up for this in later steps, or begin with a lot of effort and then relax later. 

If this is the case, then minimizing SDA(s,a) simply means choosing one of the actions for which E(s)∈Q(s,a) and thus E(s,a)=E(s) and thus SDA(s,a)=0, and then put a−=a+=a. When there are many such candidate actions a, that will still leave us with some remaining freedom for incorporating some other safety criteria to choose between them, maybe deterministically. One might thus think that this form of optimization is safe enough and should be performed because it gets rid of all variability and randomization in that step. 

For example: Assume the goal is to get back home with fourteen apples within a week, and the agent can harvest or eat at most six apples on each day (apples are really scarce). Then each day the agent might choose to harvest or eat any number of apples, as long as its current stock of apples does not deviate from fourteen by more than six times the remaining number of days. Only in the last several days it might then have to harvest or eat exactly six apples per day, to make up for earlier deviations and to land at fourteen apples sharp eventually.   

But there's at least two objections to minimizing SDA. First, generically, we can not be sure enough that there will indeed be many different actions for which E(s)∈Q(s,a), and so restricting our choice to the potentially only few actions that fulfill this might not allow us to incorporate safety criteria to a sufficient level. In particular, we should expect that the latter is often the case in the final steps of an episode, where there might at best be a single such action that perfectly makes up for the earlier under- or over-achievements, or even no such action at all. Second, getting rid of all randomization goes against the intuition of some many members of our project team that randomization is a desirable feature that tends to increase rather than decrease safety (this intuition also underlies the alternative approach of quantilization). 

We think that one should thus not minimize performance metrics such as SDA or any other of the later discussed metrics, but at best use them as soft criteria. Arguably the most standard way to do this is to use a softmin (Boltzmann) policy for drawing both candidate actions a− and a+ independently of each other, on the basis of their SDA (or another metric's) scores: 

a±∼exp⁡(−β×SDA(s,a±))

restricted to those a−,a+ with Q(s,a−)≤Q(s)≤Q(s,a+), and for some sufficiently small inverse temperature β>0 to ensure sufficient randomization. 

While the SDA criterion is about keeping the variance of the Total rather low and is thus about fulfilling the goal in a presumably more desirable way from the viewpoint of the goal-setting client, the more important type of action selection criterion is not goal- but safety-related. So let us now look at the latter type of criterion. 

Toy example of a generic safety metric

As the apple example above shows, the agent might easily get into a situation where it seems to be a good idea to take an extreme action. A very coarse but straightforward and cheap to compute heuristic for detecting at least some forms of extreme actions is when the corresponding aspiration is close to an extreme value of the feasibility interval: E(s,a)≈Q―(s,a) or E(s,a)≈Q―(s,a). 

This motivates the following generic safety criterion, the (relative, one-step) squared extremity of aspiration:

SEA(s,a):=4×((Q―(s,a)∖E(s,a)∖Q―(s,a))−1/2)2∈[0,1].

(The factor 4 is for normalization only) 

The rationale is that using actions a−,a+ with rather low values of SEA would tend to make the agent's aspirations stay close to the center of the relevant feasibility intervals, and thus hopefully also make its actions remain moderate in terms of the achieved Total. 

Let's see whether that hope is justified by studying what minimizing SEA would lead to in the above apple example:

• On day zero, E(s0)=14, V(s)=[−42,42], and Q(s0,a)=[a−36,a+36] for a∈{−6,…,6}. Since the current state-aspiration, 14, is in all actions' feasibility intervals, all action-aspirations are also 14. Both the under- and overachieving action with smallest SEA is thus a−=a+=6, since this action's Q-midpoint, 6, is closest to its action-aspiration, 14. The corresponding λ=−30∖14∖42=44/72. 

• On the next day, the new feasibility interval is V(s1)=[−36,36], so the new state-aspiration is E(s1)=−36:44/72:36=8. This is simply the difference between the previous state-aspiration, 14, and the received Delta, 6. (It is easy to see that the aspiration propagation mechanism used has this property whenever the environment is deterministic). Since Q(s1,a)=[a−30,a+30]∋8, we again have E(s1,a)=8 for all a, and thus again a−=a+=6 since 6 is closest to 8. 

• On the next day, E(s2)=8−6=2,  Q(s2,a)=[a−24,a+24]∋2, and now a−=a+=2 since 2 is closest to 2. Afterwards, the agent neither harvests not eats any apples but lies in the grass, relaxing.

The example shows that the consequence of minimizing SEA is not, as hoped, always an avoidance of extreme actions. Rather, the agent chooses the maximum action in the first two steps (which might or might not be what we eventually consider safe, but is certainly not what we hoped the criterion would prevent). 

Maybe we should compare Deltas rather than aspirations if we want to avoid this? So what about (relative, one-step) squared extremity of Delta,  

SED(s,a):=4×((mina′Eδ(s,a′)∖Eδ(s,a)∖maxa′Eδ(s,a′))−1/2)2∈[0,1],

If the agent minimizes this instead of SEA in the apple example, it will procrastinate by choosing a−=a+=0 for the first several days, as long as 14∈Q(st,0)=[6t−36,36−6t]. This happens on the first four days. On the fifth day (t=4), it will still have aspiration 14. It will then again put a−=0 with action-aspirationE(s4,0)=12=Q―(s4,0), but it will put a+=2 with action-aspiration E(s4,2)=14. Since the latter equals the state-aspiration, the calculated probability in step 3 of the algorithm turns out to be p=1, meaning the agent will use action a+ for sure after all, rather than randomizing it with a−. This leaves an aspiration of 14−2=12. On the remaining two days, the agent then has to use a=6 to fulfill the aspiration in expectation.

Huh? Even though both heuristics seem to be based on a similar idea, one of them leads to early extreme action and later relaxation, and the other to procrastination and later extreme action, just the opposite. Neither of them avoids extreme actions. 

One might think that one could fix the problem by simply adding the two metrics up into a combined safety loss, 

L(s,a):=SEA(s,a)+SED(s,a)∈[0,2].

But in the apple example, the agent would still harvest six apples on the first day, since (14−6)2+(6−0)2<(14−5)2+(5−0)2. Only on the second day, it would harvest just four instead of six apples, because (8−4)2+(4−0)2 is minimal. On the third day: two apples because (4−2)2−(2−0)2 is minimal. Then one apple, and then randomize between one or zero apples, until it has the 14 apples, or until the last day has come where it needs to harvest one last apple.

The main reason for all these heuristics failing in that example is their one-step nature which does not anticipate later actions. In the LW sequence we will talk about more complex, farsighted metrics that can still be computed efficiently in a recursive manner, such as disordering potential,

DP(s,a)=Es′|s,a(−log⁡P(s′|s,a)+log⁡∑aexp⁡(DP(s,a))),

which measures the largest Shannon entropy in the state trajectory that the agent could cause if it aimed to, or terminal state distance,

TSD(s,a,E(s,a))=Es′|s,a(1s′ is terminald(s′,s0)+1s′ is not terminalEa′∼π(s′,E(s′))TSD(s′,a′,E(s′,a′))),

which measures the expected distance between the terminal and initial state according to some metric d on state space and depends on the actual policy π and aspiration propagation scheme E(s)→E(s,a)→E(s′) that the agent uses. 

Let us finish this post by discussing other ways to combine several performance and/or safety metrics. 

Using several safety and performance metrics

Defining a combined loss function like L(s,a) above by adding up components is clearly not the only way to combine several safety and/or performance metrics, and is maybe not the best way to do that because it allows unlimited trade-offs between the components. 

It also requires them to be measured in the same units and to be of similar scale, or to make them so by multiplying them with suitable prefactors that would have to be justified. This at least can always be achieved by some form of normalization, like we did above to make SEA and SED dimensionless and bounded by [0,1].

Trade-offs between several normalized loss components Li(s,a)∈[0,1] can be limited in all kinds of ways, for example by using some form of economics-inspired "safety production function" such as the CES-type function 

L(s,a)=1−(∑ibi(1−Li(s,a))ρ)1/ρ

with suitable parameters ρ and bi>0 with ∑ibi=1. For ρ=1, we just have a linear combination that allows for unlimited trade-offs. At the other extreme, in the limit of ρ→−∞, we get L(s,a)=maxiLi(s,a), which does not allow for any trade-offs.

Such a combined loss function can then be used to determine the two actions a−,a+, e.g., using a softmin policy as suggested above. 

An alternative to this approach is to use some kind of filtering approach to prevent unwanted trade-offs. E.g., one could use one of the safety loss metrics, L1(s,a), to restrict the set of candidate actions to those with sufficiently small loss, say withL1(s,a)≤mina′L1(s,a′):0.1:maxa′L1(s,a′), then use a second metric, L2, to filter further, and so on, and finally use the last metric, Lk, in a softmin policy.

One can come up with many plausible generic safety criteria (see LW sequence), and one might be tempted to just combine them all in one of these ways, in the hope to thereby have found the ultimate safety loss function or filtering scheme. But it might also well be that one will have forgotten some safety aspects and end up with an imperfect combined safety metric. This would be just another example of a misspecified objective function. Hence we should again resist falling into the optimization trap of strictly minimizing that combined loss function, or the final loss function of the filtering scheme. Rather, the agent should probably still use a sufficient amount of randomization in the end, for example by using a softmin policy with sufficient temperature. 

If it is unclear what a sufficient temperature is, one can set the temperature automatically so that the odds ratio between the least and most likely actions equals a specified value: β:=η/(maxaL(s,a)−minaL(s,a)) for some fixed η>0.

More freedom via aspiration intervals

We saw earlier that if the goal is to make expected Total equal an aspiration value, Eτ=E∈R, the set of successful policies has large but not full dimension and thus still has measure zero. In other words, making expected Total exactly equal to some value still requires policies that are very "precise" and in this regard very "special" and potentially dangerous. So we should probably allow some leeway, which should not make any difference in almost all real-world tasks but increase safety further by avoiding solutions that are too special. 

Of course the simplest way to provide this leeway would be to just get rid of the hard constraint altogether and replace it by a soft incentive to make Eτ close to E∈R, for example by using a softmin policy based on the mean squared error. This might be acceptable for some tasks but less so for others. The situation appears to be somewhat similar to the question of choosing estimators in statistics (e.g., a suitable estimator of variance), where sometimes one only wants the estimator with the smallest standard error, not caring for bias (and thus not having any guarantee about the expected value), sometimes one wants an unbiased estimator (i.e., an estimator that comes with an exact guarantee about the expected value), and sometimes one wants at least a consistent estimator that is unbiased in the limit of large data and only approximately unbiased otherwise (i.e., have only an asymptotic rather than an exact guarantee about the expected value). 

For tasks where one wants at least some guarantee about the expected Total, one can replace the aspiration value by an aspiration interval E=[e―,e―]⊆R and require that Eτ∈E. 

The basic algorithm (algorithm 1) can easily be generalized to this case and only becomes a little bit more complicated due to the involved interval arithmetic:

Decision algorithm 2    

Similar to algorithm 1, we... 

• set action-aspiration intervals E(s,a)=[e―(s,a),e―(s,a)]⊆Q(s,a) for each possible action a∈A on the basis of the current state-aspiration interval E(s)=[e―(s),e―(s)] and the action's feasibility interval Q(s,a), trying to keep E(s,a) similar to and no wider than E(s),

• choose an "under-achieving" action a− and an "over-achieving" action a+ w.r.t. the midpoints of the intervals E(s) and E(s,a),

• choose probabilistically between a− and a+ with suitable probabilities, and

• propagate the action-aspiration interval E(s,a) to the new state-aspiration interval E(s′) by rescaling between the feasibility intervals of a and s′.

More precisely: Let m(I) denote the midpoint of interval I. Given state s and state-aspiration interval E(s),

1. For all a∈A, let E(s,a) be the closest interval to E(s) that lies within Q(s,a) and is as wide as the smaller of E(s) and Q(s,a).

2. Pick some a−,a+∈A with m(E(s,a−))≤m(E(s))≤m(E(s,a+))

3. Compute p←m(E(s,a−))∖m(E(s))∖m(E(s,a−))

4. With probability p, put a←a+, otherwise put a←a−

5. Implement action a in the environment and observe successor state s′←step(s,a)

6. Compute λ―←Q―(s,a)∖e―(s,a)∖Q―(s,a) and e―(s′)←V―(s′):λ―:V―(s′), and similarly for λ― and e―(s′)

Note that the condition that no E(s,a) must be wider than E(s) in step 1 ensures that, independently of what the value of p computed in step 3 will turn out to be, any convex combination q−:p:q+ of values q±∈E(s,a±) is an element of E(s). This is the crucial feature that ensures that aspirations will be met:

Theorem (Interval guarantee)

If the world model predicts state transition probabilities correctly and the episode-aspiration interval E(s0) is a subset of the initial state's feasibility interval, E(s0)⊆V(s0), then algorithm 2 fulfills the goal Eτ∈E(s0).

The Proof is completely analogous to the last one, except that each occurrence of V(s,e)=E(s)∈V(s) and Q(s,a,e)=E(s,a)∈Q(s,a) is replaced by V(s,e)∈E(s)⊆V(s) and Q(s,a,e)∈E(s,a)⊆Q(s,a), linear combination of sets is defined as cX+c′X′={cx+c′x′:x∈X,x′∈X′}, and we use the fact that m(E(s,a−)):p:m(E(s,a+))=m(E(s)) implies E(s,a−):p:E(s,a+)⊆E(s).

Special cases

Satisficing. The special case where the upper end of the aspiration interval coincides with the upper end of the feasibility interval leads to a form of satisficing guided by additional criteria.

Probability of ending in a desirable state. A subcase of satisficing is when all Deltas are 0 except for some terminal states where it is 1, indicating that a "desirable" terminal state has been reached. In that case, the lower bound of the aspiration interval is simply the minimum acceptable probability of ending in a desirable state.

Glossary

This is how we use certain potentially ambiguous terms in this post: 

Agent: We employ a very broad definition of "agent" here: The agent is a machine with perceptors that produce observations, and actuators that it can use to take actions, and that uses a decision algorithm to pick actions that it then takes. Think: a household's robot assistant, a firm's or government's strategic AI consultant. We do not assume that the agent has a goal of its own, let alone that it is a maximizer.

Aspiration: A goal formalized via a set of constraints on the state or trajectory or probability distribution of trajectories of the world. E.g., "The expected position of the spaceship in 1 week from now should be the L1 Lagrange point, its total standard deviation should be at most 100 km, and its velocity relative to the L1 point should be at most 1 km/h".

Decision algorithm: A (potentially stochastic) algorithm whose input is a sequence of observations (that the agent's perceptors have made) and whose output is an action (which the agent's actuators then take). A decision algorithm might use a fixed "policy" or a policy it adapts from time to time, e.g. via some form of learning, or some other way of deriving a decision.

Maximizer: An agent whose decision algorithm takes an objective function and uses an optimization algorithm to find the (typically unique) action at which the predicted expected value of the objective function (or some other summary statistic of the predicted probability distribution of the values of this function) is (at least approximately) maximal, and then outputs that action.

Optimization algorithm: An algorithm that takes some function f  (given as a formula or as another algorithm that can be used to compute values f(x) of that function), calculates or approximates the (typically unique) location x of the global maximum (or minimum) of f, and returns that location x. In other words: an algorithm O:f↦O(f)≈arg⁡maxxf(x).

Utility function: A function (only defined up to positive affine transformations) of the (predicted) state of the world or trajectory of states of the world that represents the preferences of the holder of the utility function over all possible probability distributions of states or trajectories of the world in a way conforming to the axioms of expected utility theory or some form of non-expected utility theory such as cumulative prospect theory.

Loss function: A non-negative function of potential outputs of an algorithm (actions, action sequences, policies, parameter values, changes in parameters, etc.) that is meant to represent something the algorithm shall rather keep small.

Goal: Any type of additional input (additional to the observations) to the decision algorithm that guides the direction of the decision algorithm, such as: 

• a desirable state of the world or trajectory of states of the world, or probability distribution over trajectories of states of the world, or expected value for some observable

• a (crisp or fuzzy) set of acceptable states, trajectories, distributions of such, or expected value of some observable

• a utility function for expected (or otherwise aggregated) utility maximization 

Planning: The activity of considering several action sequences or policies ("plans") for a significant planning time horizon, predicting their consequences using some world model, evaluating those consequences, and deciding on one plan the basis of these evaluations.

Learning: Improving one's beliefs/knowledge over time on the basis of some stream of incoming data, using some form of learning algorithm, and potentially using some data acquisition strategy such as some form of exploration. 

Satisficing: A decision-making strategy that entails selecting an option that meets a predefined level of acceptability or adequacy rather than optimizing for the best possible outcome. This approach emphasizes practicality and efficiency by prioritizing satisfactory solutions over the potentially exhaustive search for the optimal one. See also the satisficer tag.

Quantilization: A technique in decision theory, introduced in a 2015 paper by Jessica Taylor. Instead of selecting the action that maximizes expected utility, a quantilizing agent chooses randomly among the actions which rank higher than a predetermined quantile in some base distribution of actions. This method aims to balance between exploiting known strategies and exploring potentially better but riskier ones, thereby mitigating the impact of errors in the utility estimation. See also the quantilization tag.

Reward Function Regularization: A method in machine learning and AI design where the reward function is modified by adding penalty terms or constraints to discourage certain behaviors or encourage others, such as in Impact Regularization. This approach is often used to incorporate safety, ethical, or other secondary objectives into the optimization process. As such, it can make use of the same type of safety criteria as an aspiration-based algorithm might.

Safety Criterion: A (typically quantitative) formal criterion that assesses some potentially relevant safety aspect of a potential action or plan. E.g., how much change in the world trajectory could this plan maximally cause? Safety criteria can be used as the basis of loss functions or in reward function regularization to guide behavior.  

Using Aspirations as a Means to Maximize Reward: An approach in adaptive reinforcement learning where aspiration levels (desired outcomes or benchmarks) are adjusted over time based on performance feedback. Here, aspirations serve as dynamic targets to guide learning and action selection towards ultimately maximizing cumulative rewards. This method contrasts with using aspirations as fixed goals, emphasizing their role as flexible benchmarks in the pursuit of long-term utility optimization. This is not what we assume in this project, where we assume no such underlying utility function.

Project status

The SatisfIA project is a collaborative effort by volunteers from different programs: AI Safety Camp, SPAR, and interns from different ENS​​s, led by Jobst Heitzig at PIK's Complexity Science Dept. and the FutureLab on Game Theory and Networks of Interacting Agents that has started to work on AI safety in 2023.

Motivated by the above thoughts, two of us (Jobst and Clément) began to investigate the possibility of avoiding the notion of utility function altogether and develop decision algorithms based on goals specified through constraints called "aspirations", to avoid risks from Goodhart's law and extreme actions​​. We originally worked in a reinforcement learning framework, modifying existing temporal difference learning algorithms, but soon ran into issues more related to the learning algorithms than to the aspiration-based policies we actually wanted to study (see Clément's earlier post). 

Because of these issues, discussions at VAISU 2023 and FAR Labs, and comments from Stuart Russell, Jobst then switched from a learning framework to a model-based planning framework and adapted the optimal planning algorithms from agentmodels.org to work with aspirations instead, which worked much better and allowed us to focus on the decision making aspect.   

As of spring 2024, the project is participating in AI Safety Camp and SPAR with about a dozen people investing a few hours a week. We are looking for additional collaborators. Currently, our efforts focus on theoretical design, prototypical software implementation, and simulations in simple test environments. We will continue documenting our progress in the LW sequence. We also plan to submit a first academic conference paper soon.

Acknowledgements

In addition to AISC, SPAR, VAISU, FAR Labs, and GaNe, we'd like to thank everyone who contributed or continues to contribute to the SatisfIA project. 

1. ^{^}

   The algorithm is based on the idea of propagating aspirations along time, and we prove that the algorithm gives a performance guarantee if the goal is feasible. Then we extend the basic algorithm by 1) adding criteria for choosing the two candidate actions the algorithm mixes, and 2) by generalizing the goal from making the expected Total equal a particular value to making it fall into a particular interval.

2. ^{^}

   This might not be trivial of course if the agent might "outsource" or otherwise cause the maximization of some objective function by others, a question that is related to "reflective stability" in the sense of this post. An rule additional  "do not maximize" would hence be something like "do not cause maximization by other agents", but this is beyond the scope of the project at the moment.

3. ^{^}

   Information-theoretic, not thermodynamic.

4. ^{^}

   This is also why Sutton and Barto's leading textbook on RL starts with model-based algorithms and turns to learning only in chapter 6.

5. ^{^}

   When we extend our approach later to incorporate performance and safety criteria, we might also have to assume further functions, such as the expected squared Delta (to be able to estimate the variance of Total), or some transition-internal world trajectory entropy (to be able to estimate total world trajectory entropy), etc.

6. ^{^}

   Note that this type of goal is also implicitly assumed in the alternative Decision Transformer approach, where a transformer network is asked to predict an action leading to a prompted expected Total. 

7. ^{^}

   If the agent is a part of a more complex system of collaborating agents (e.g., a hierarchy of subsystems), the "action" might consist in specifying a subtask for another agent, that other agent would be modeled as part of the "environment" here, and the observation returned by step might be what that other agent reports back at the end of its performing that subtask.    

8. ^{^}

   It is important to note that st might be an incomplete description of the true environment state, which we denote xt but rarely refer to here. 

9. ^{^}

   Note that we're in a model-based planning context, so it directly computes the values recursively using the world model, rather than trying to learn it from acting in the real or simulated environment using some form of reinforcement learning.

10. ^{^}

    These assumptions may of course be generalized, at the cost of some hassle in verifying that expectations are well-defined, to allow cycles or infinite time horizons, but the general idea of the proof remains the same.

11. ^{^}

    E.g., assume the agent can buy zero or one apple per day, has two days time, and the aspiration is one apple. On the first day, the state's feasibility interval is [0,2], the action of buying zero apples has feasibility interval [0,1] and would thus get action-aspiration 0.5, and the action of buying one apple has feasibility interval [1,2] and would thus get action-aspiration 1.5. To get 1 on average, the agent would thus toss a coin. So far, this is fine, but on the next day the aspiration would not be 0 or 1 in order to land at 1 apple exactly, but would be 0.5. So on the second day, the agent would again toss a coin. Altogether, it would get 0 or 1 or 2 apples, with probabilities 1/4, 1/2, and 1/4, rather than getting 1 apple for sure.

12. ^{^}

    That the algorithm uses an optimization algorithm that it gives the function to maximize to is crucial in this definition. Without this condition, every agent would be a maximizer, since for whatever output a the decision algorithm produces, one can always find ex post some function f for which arg⁡maxxf(x)=a. 

Here’s a Bayes theorem cheat sheet if you need it:

The problem of confirmation
 

Nayesian: Remind me again: how does confirmation work in Bayesianism?

Bayesian: The evidence E confirms hypothesis H if and only if the posterior probability of H given E is greater than the prior probability of H.

N: So there is no difference between increasing the hypothesis’ probability and confirming a hypothesis?

B: Not really, no.

N: I would say that there are cases where something increases the probability of a hypothesis, but we would not say that it confirms the hypothesis. Let's say there is a recent hypothesis that has a strong theoretical foundation and some evidence from a few experiments. Scientists disagree about whether or not this hypothesis is correct. But, again, it's a novel proposal. Say that a recent article published in a highly regarded scientific journal supports this hypothesis. Appearing in the journal increases my degree of belief that the hypothesis is true. So it appears that the Bayesian must conclude that a publication in a reputable scientific journal confirms the hypothesis, but surely that is incorrect. Appearing in a reputable scientific journal isn't in and of itself evidence that a hypothesis is correct; it merely implies that there exists evidence supporting the hypothesis.

B: No, I would say that appearing in a scientific journal is evidence, although maybe a different type of evidence than we would normally associate with that word. Perhaps we should make a taxonomy of evidence so we don’t end up in information cascades.

N: Weird. What about the other way around? What if there’s a fact that confirms a hypothesis without increasing its probability?

B: Can you give an example?

N: Sure! Let’s say I throw a rock at someone and use Newtonian mechanics to calculate the trajectory of the rock. The predicted trajectory indeed happens, but that doesn’t increase my credence in Newtonian mechanics since I believe that general relativity has displaced it. If we know Newtonian mechanics is false, its probability is zero, and that probability will never increase regardless of how many correct predictions it makes.

B: So? That seems reasonable to me. A falsified theory remains falsified even if it makes a correct prediction.

N: But surely the theory of Newtonian mechanics is less wrong than, I don’t know, the existence of fairies that make square circles. We want to be able to say that theories that make loads of correct predictions are better than those that don’t, and that the people in the past who believed in Newtonian mechanics were reasonable while those who believed in square-circle-making fairies weren’t. The first group used evidence to support their hypothesis, while the second group didn’t.

B: I don’t think you can use Bayesianism retroactively like that.

N: It’s not just a problem for retroactive evaluations. Many modern scientific theories and models include idealizations, in which certain properties of a system are intentionally simplified. For example, in physics, we often use the ideal gas law. An ideal gas consists of dimensionless particles whose movements are completely random. But an ideal gas doesn’t exist; we invented the concept to decrease the complexity of our computations. We know that the actual probability of theories that use the ideal gas law is 0. Under Bayesianism, any and all theories that make use of the ideal gas law would have no way to increase their probability. Yet we continue to believe that new evidence confirms these models, and it seems rational to do so.

B: Okay, I guess I’ll have to actually make this taxonomy of evidence now. Let’s call the evidence provided by being published in a scientific journal "secondhand evidence". What we want is "firsthand evidence". Personal confirmation might come from secondhand evidence, but the only way to confirm a hypothesis in the conventional sense of the word is to do primary research. Do experiments, try to falsify it etc. When a hypothesis appears in a scientific journal, it is not a test of the hypothesis; rather, the paper in the journal simply reports on previous research. It’s secondhand evidence.

N: I mean, it’s a kind of "test" whether or not a theory can even make it into a journal.

B: But it’s not a scientific test. Similarly, we can obviously set up tests of hypotheses we know are false, including models with idealizations. We can, for example, use a false hypothesis to design an experiment and predict a specific set of outcomes.

N: Seems vague. You would need to find a way to differentiate secondhand evidence from firsthand evidence and then design different ways Bayesianism deals with both types of evidence.

B: I’ll get right on it!

The problem of old evidence
 

BaNo: I think Bayesianism struggles with retrodiction.1

BaYes: Why do you think so?

N: Well, consider the following scenario: scientists can’t explain why volcanoes erupted when they did. We’ll call this piece of evidence (the pattern of eruptions) E. Then a new volcanological theory comes out that retrodicts the timing of all eruptions to the second. It seems like the fact that this theory can perfectly explain the timing of the eruptions is evidence that said theory is correct.
However, Bayesianism says that E confirms H when H’s posterior given E is higher than H’s prior, and we work out the posterior by applying Bayes’ rule. At the time the new theory was proposed, the pattern of the eruptions was already known, so the probability of E equals 1.

Which means the probability of E given H is also 1. It then follows that the probability of H given E is equal to the probability of H, so the posterior is equal to the prior. In other words: E can't confirm H when E is already known.
Under Bayesianism, no matter how impressive of a retrodiction a theory makes, it can never strengthen that theory.

B: I mean, what if I just give theories that provide good retrodictions a higher prior?

N: That wouldn’t work in scenarios where we only discover the retrodiction after the theory has already been introduced. If I propose this new volcanological theory and we assign it a prior and only later we discover its perfect retrodiction, the prior has already been assigned.

B: What if we used a counterfactual? Instead of asking ourselves what the scientist’s actual degree of belief is in E we ask ourselves what her degree of belief would have been had she not known about E. In that case, the probability of E does not just equal 1. 

N: How do we know what her degree of belief would have been?

B: Well, say she forgets all the volcanic eruptions without it altering her other knowledge.

N: Impossible, knowledge is entangled with one another, especially something as drastic and traumatic as volcanic eruptions.

B: Okay okay, what about a counterfactual history instead, where no one knows about volcanic eruptions and we ask the scientific community in this timeline what they think.

N: And these scientists don’t know about volcanic eruptions? What, do they live on Mars or something? How are we supposed to know what alternate universe alien scientists believe?

B: Alright, alright, I’ll bite the bullet, retrodictions don’t strengthen a theory.

N: But this is not only a problem for retrodictions, but also for old predictions. Say a theory made a correct prediction. E.g. germ theory predicted that if we looked under a microscope we would see microbes. Then when the modern microscope was invented it turned out to be a correct prediction. But we live in the present, and for us the fact that looking into a microscope will show us microbes is not new evidence. For us its probability is one. So, according to Bayesianism, when we first learn of germ theory, the fact that we know that we can look into a microscope to see germs can’t confirm germ theory. That’s ridiculous!

B: I think I can combine a solution for the problem of retrodiction with the problem of confirmation. The problem of us wanting to update on the ‘secondhand evidence’ of appearing in a scientific journal seems analogous to germ theory correctly predicting microbes in the past, and us ‘wanting’ to update on that past successful prediction.
What if we considered a kind of ‘collective Bayesianism’ which describes what an interconnected collection of agents (ought to) update towards. A ‘Bayesian collective’ does update because of germ theory’s successful prediction, since it’s around for that. At this point it becomes easy to make that distinction between ‘firsthand evidence’ and ‘secondhand evidence’. ‘Firsthand evidence’ is that which makes the Bayesian collective and the Bayesian individual update, whereas ‘secondhand evidence’ only makes the individual Bayesian update.
For you as an individual it’s a surprise that something has appeared in a scientific journal and ‘confirms’ a theory, but it doesn’t for the collective. The goal of the Bayesian individual is not only to use ‘firsthand evidence’ to update the knowledgebase of themself (and the collective), but also to use ‘secondhand evidence’ to bring their own credences as much in line with the Bayesian collective as possible.

N: So would an alien scientist be part of our Bayesian collective?

B: It must be interconnected, so if it can’t communicate with us, no.

N: In this model, if a historian discovers a long lost text from ancient Greece they aren’t collecting firsthand evidence? The collective doesn’t update?

B: Bayesianism is an epistemic ideal to strive towards, not a description of how people actually work. An actual collective will not conform to how the ideal of a Bayesian collective operates. An ideal Bayesian collective doesn’t forget anything, just like an ideal Bayesian doesn’t forget anything, but obviously real people and groups do forget things. An ideal Bayesian collective wouldn’t need historians, the insights from ancient greek writers would continue to be in the network, and the collective thus wouldn’t update on the ancient greek text. But real collectives do need historians, and they do update on the ancient greek text, because mankind keeps forgetting its history.2

The problem of logical omniscience
 

Bay=cray: What are the axioms of probability theory again?

Bay=bae: They are:

• Axiom 1: The probability of an event is a real number greater than or equal to 0 and smaller than or equal to 1.

• Axiom 2: The probability that at least one of all the possible outcomes of a process (such as rolling a die) will occur is 1.

• Axiom 3: If two events A and B are mutually exclusive, then the probability of either A or B occurring is the probability of A occurring plus the probability of B occurring.

N: And Bayesianism treats the axioms of probability theory as constraints on one's degrees of belief. In other words, for the Bayesian, probabilities are the same as degrees of belief, right?

B: Correct.

N: How do we know what our degrees of beliefs are?

B: With bets. If you think the odds of Biden being reelected is one in three, the most risky odds you would take for a bet on that outcome is one in three.

N: I don’t know, it seems like degrees of belief and probability are dissimilar in many ways. We have all sorts of biases, like base rate neglect, that make our beliefs different from what a Bayesian would prescribe.

B: Yes, just like with the memory issue, Bayesianism is not a descriptive model of how humans form beliefs, it is a prescriptive model of what humans ought to believe. Treat it as a goal to strive towards.

N: Okay, but what about mathematical truths? The statement 4 = 4 is true by definition. So, according to the axioms of probability theory, it should have a probability of 1, since logical truths are necessarily true. But there are many logical truths about which we are uncertain. Just think of all of the currently unproven mathematical conjectures. Do you think P=NP is true? False? Are you unsure? I doubt most people would say they are 100% confident either way. But logically these conjectures are either necessarily true or necessarily false. So they should all have a probability of either 0 or 1.
This becomes especially problematic when you think about how Bayesianism tells me I should be willing to take bets based on a theory’s probability. The probability of Pythagoras’ theorem is 1, but I’m not willing to bet all my money on it without someone else putting money in too. I can believe that the probability of a mathematical theorem or conjecture is 1, without being certain that it is true.
Bayesianism seems to have trouble explaining doubts about logical and mathematical truths, which is a shame because those doubts are often reasonable, if not unavoidable.

B: I have the same response as before. Bayesianism is an ideal to strive towards. The platonic ideal of a scientist would be aware of all logical truths, but real world scientists obviously aren’t ideal.

N: Why is this ideal? Why should logically omniscient scientists be preferred over any other type of ideal? As an idealization, we could also assume that scientists already have access to all possible evidence. In that case, there would be no need to test theories because scientists would already know the outcome of every possible test.

B: This idealization would be unhelpful. It would not reveal much about how actual scientists behave or the methodologies they employ. Logical truths are a better idealization because logical truths aren't really relevant to scientific confirmation. Scientists don't deal with logical hypotheses; they deal with empirical hypotheses, and Bayesianism is great at dealing with those.

N: What about mathematicians? They do have to deal with mathematical/logical conjectures.

B: They can disregard Bayesianism and use conventional mathematical methods.

N: What about philosophers and computer scientists who need to combine logical conjectures with empirical evidence?

B: We might be able to tackle it with "logical uncertainty," but that’s still a developing field.
Alternatively we might give Bayesianism its own axioms that are similar, but not exactly the same axioms as probability theory. Maybe something like:

• Axiom 2: The credence that at least one of all the imagined outcomes of a process will occur is 1.

• Axiom 3: If two events A and B are imagined to be mutually exclusive, then the credence of either A or B occurring is the credence of A occurring plus the credence of B occurring.

P=NP is either necessarily true or necessarily false, but we can imagine untrue things. By allowing imagination to enter our axioms we can account for this discrepancy between our minds and the mathematical laws.

N: Interesting…

The problem of agnosticism
 

Bayliever: Does that answer all your questions?

Bagan: Nope! What are the Bayesian probabilities? The problem of logical omniscience suggests that we can't simply say they are degrees of belief, so what are they? Take a claim like "There are a billion planets outside the observable universe". How do you assign a probability to that? We can’t observe them, so we can’t rely on empiricism or mathematics, so... shouldn’t we be agnostic? How do we represent agnosticism in terms of probability assignments?

B: Prior probabilities can be anything you want. Just pick something at random between 0 and 1. It doesn’t really matter because our probabilities will converge over time given enough incoming data.

N: If I just pick a prior at random, that prior doesn’t represent my epistemic state. If I pick 0.7, I now have to pretend I’m 70% certain that there are a billion planets outside the observable universe, even though I feel totally agnostic. I’m not even sure we’ll ever find out whether there really are a billion planets outside the observable universe. Why can’t I just say that it’s somewhere between 0 and 1, but I don’t know where?

B: You need to be able to update. A rational thinker needs to have a definite value.

N: Why? There is no Dutch book argument against being agnostic. If someone offers me Dutch book bets based on the number of planets outside the observable universe, I can just decline.

B: What if you don’t have a choice? What if that person has a gun?

N: How would that person even resolve the bet? You’d have to know the amount of planets outside the observable universe.

B: It’s God, and God has a gun.

N: Okay, fine, but even in that absurd scenario I don’t have to have a definite value to take on bets. I can, for example, use a random procedure, like rolling a dice.

B: What if that procedure gives you a 0 or a 1? You would have a trapped prior, and you couldn’t update your beliefs no matter what evidence you observed.

N: I can’t update my beliefs if I follow Bayesianism. The axioms of probability theory allow me to assign a 0 or a 1 to a hypothesis. It’s Bayesianism that traps my priors.

B: You can’t assign a 0 or a 1 to an empirical hypothesis for that reason.

N: Isn’t that ad hoc? The probabilities were meant to represent an agent's degree of belief, and agents can certainly be certain about a belief. It seems the probabilities do not represent an agent's degree of belief after all. The Bayesian needs to add all sorts of extra rules, like that we can assign 0 and 1 to logical theorems but not empirical theories, which must actually be assigned a probability between 0 and 1. So... what are the probabilities exactly?

B: Hmmm… Let me get back to you on that one!

The problem of the foreacting agent
 

Doubting Thomas: Say there is an agent whose behavior I want to anticipate. However, I know that this agent is:

1. extremely good at predicting what I’m going to guess (maybe it’s an AI or a neuroscientist with a brain scanner) and…

2. this agent wants me to make a successful prediction.

If I guess the agent has a 90% chance of pushing a button they will have already predicted it, and will afterwards push the button with 90% probability. Same with any other probability, they will predict it and set their probability for acting accordingly. It’s forecasting my guess and reacting before I predict, hence foreacting. After learning this information what should my posterior be? What probability should I assign to them pushing the button?

Thomas Bayes: Whatever you want to.

Doubting Thomas: But ‘whatever you want to’ is not a number between 0 and 1.

B: Just pick a number at random then.

N: If I just pick a prior at random, that doesn’t represent my epistemic state.

B: Ah, this is the problem of agnosticism again. I think I’ve found a solution. Instead of Bayesianism being about discrete numbers, we make it about ranges of numbers. So instead of saying the probability is around 0.7 we say it’s 0.6–0.8. That way we can say in this scenario and in the case of agnosticism that the range is 0–1.3

N: This would be an adequate solution to one of the problems, but can’t be a solution for both agnosticism and foreacting predictors.

B: Why not?

N: Because they don’t depict the same epistemic state. In fact, they represent an almost opposite state. With agnosticism I have basically no confidence in any prediction, whereas with the foreacting predictor I have ultimate confidence in all predictions. Also, what if the agent is foreacting non-uniformly? Let’s say it makes its probability of acting 40% and 60% if I predict it will be 40% and 60% respectively, but makes its probability not conform with my prediction when I predict anything else. So if I predict, say, 51% it will act with a probability of, say, 30%. Let’s also assume I know this about the predictor. Now the range is not 0–1, it’s not even 0.4–0.6 since it will act with a different probability when I predict 51%.

B: Hmmm…

N: And what if I have non-epistemic reasons to prefer one credence over another. Let’s say I’m trying to predict whether the foreacting agent will kill babies. I have a prior probability of 99% that it will. The agent foreacts, and I observe that it does indeed kill a baby. Now I learn it’s a foreacting agent. With Bayesianism I keep my credence at 99%, but surely I ought to switch to 0%. 0% is the ‘moral credence’.

B: This is a farfetched scenario.

N: Similar things can happen in e.g. a prediction market. If the market participants think an agent has a 100% probability of killing a baby they will bet on 100%. But if they then learn that the agent will 100% kill the baby if they bet on 1%-100%, but will not kill the baby if the market is 0% they have a problem. Each individual participant might want to switch to 0%, but if they act first the other participants are financially incentivized to not switch. You have a coordination problem. The market causes the bad outcome. You don’t even need foreacting for this, a reacting market is enough. Also, there might be disagreement on what the ‘moral credence’ even is. In such a scenario the first buyers can set the equilibrium and thus cause an outcome that the majority might not want.4

B: This talk about ‘moral credences’ is besides the point. Epistemology is not about morality. Bayesianism picks an accurate credence and that’s all it needs to do.

N: But if two credences are equally good epistemically, but one is better morally, shouldn’t you have a system that picks the more moral one?

B: Alright, what if we make Bayesianism not about discrete numbers, nor about ranges, but instead about distributions? On the x-axis we put all the credences you could pick (so any number between 0 and 1) and on the y-axis what you think the probability will be based on which number you pick.
So when you encounter a phenomenon that you think has a 60% chance of occurring (no matter what you predict/which credence you pick) the graph looks like this:

And when you encounter a uniformly foreacting agent who (you believe) makes the odds of something occurring conform to what you predict (either in your head or out loud), you have a uniform distribution:

With this you can just pick any number and be correct. However if you encounter the non-uniformly foreacting agent of your example the graph could look something like this: (green line included for the sake of comparison) 

Picking 0.2 (B) will result in the predictor giving you a terrible track record (0.4). But picking 0.4 or 0.6 (C or E) will give you an incredible track record. Let’s call C and E ‘overlap points’. If this distribution is about whether the agent will kill a baby, C is the ‘moral credence’.

N: Wouldn’t A be the moral credence, since that has the lowest chance of killing a baby?

B: Humans can’t will themselves to believe A since they know that predicting a 0% chance will actually result in a 20% chance.

N: What about an agent that is especially good at self deception?

B: Right, so if you have e.g. an AI that can tamper with its own memories, it might have a moral duty to delete the memory that 0% will result in 20% and instead forge a memory that 0% will lead to 0%, just so the baby only has a 20% chance of dying.

N: What if you have a range? What if you don’t know what the probability of something is but you know it’s somewhere between 0.5 and 0.7?

B: Then it wouldn’t be thin line at 0.6, but a ‘thick’ line, a field:

N: What about total agnosticism?

B: Agnosticism would be a black box instead of a line:

The point could be anywhere between here, but you don’t know where.

N: What if you’re partially agnostic with regards to a foreacting agent?

B: This method allows for that too. If you know what the probabilities are for the foreacting agent from A to E, but are completely clueless about E to F it looks like this:

N: What if I don’t know the probabilities of the agent between E and F, but I do know it's somewhere between 0.2 and 0.6?

B: It would look something like this:

N: What if it doesn’t foreact to your credences, but the graph as a whole?

B: Then you add an axis, if it reacts to that you add another axis etc.

N: This is still rather abstract, can you be more mathematical?

B: Sure!

Applying Bayesian modeling and updating to the foreacting agent
 

To apply the Bayesian method, the main thing we need is a world model, which we can then use to calculate posterior probability distributions for things we are interested in. The world model is a Bayesian network that has… 

• one node for each relevant variable X

• one directed arrow Y→X for each direct dependency among such variables, leading from a parent node Y to a child node X

• and for each node X a formula that calculates the probability distribution for that variable from the values of all its parents: P(X | parents(X)) 

For nodes X without parents, the latter formula specifies an unconditional probability distribution: P(X | parents(X)) = P(X | empty set) = P(X).

In our case, I believe the proper model should be this:

• Variables:

  • B: whether the agent will press the button. This is a boolean variable with possible values True and False.

  • p: the credence you assign to the event B=True. This is a real-valued variable with possible values 0…1

  • q: the probability that the agent uses to decide whether to press the button or not. This is also a real-valued variable with possible values 0…1 

• Dependencies:

  • B depends only on q: parents(B) = {q}

  • q depends only on p: parents(q) = {p}

• Formulas for all variables’ (conditional) probability distributions:

  • P(B=True | q) = q, P(B=False | q) = 1 – q

  • P(q | p) is given by two functions flow, fhigh as follows:

    • If flow(p) = fhigh(p) = f(p), then q = f(p), in other words: P(q | p) = 1 iff q = f_low(p) and 0 otherwise

    • If flow(p) < fhigh(p), then P(q | p) has uniform density 1 / (f_high(p) – f_low(p) for f_low(p) < q < f_high(p) and 0 otherwise.

    • For the “uniformly foreacting agent” we have  flow(p) = fhigh(p) = f(p) = p

    • Note that we assume to know the response function upfront, so the functions flow, fhigh are not variables of the model but fixed parameters in this analysis. We might later study models in which you are told the nature of the agent only at some time point and where we therefore also model flow, fhigh as a variable, but that gets harder to denote then.

  • P(p) = whatever you initially believe about what credence you assign to the event B=True

At this point, we might be surprising necessity of the Bayesian method and get a little wary: because our model of the situation contains statements about how our credence in some variable influences that variable, we needed to include both that variable (B) and our credence (p) as nodes into the Bayesian network. Since we have to specify probability distributions for each parentless node in the network, we need to specify them about p, i.e., a probability distribution on all possible values of p, i.e., a credence about our credence in B being 0.3, a credence about our credence in B being 0.7, etc. This is the P(p) in the last line above. In other words, we need to specify 2nd-order credences! Let us for now assume that P(p) is given by a probability density g(p) for some given function g. 

The whole model thus have two parameters: 

• two function  flow, fhigh  encoding what you know about how the agent will choose q depending on p, 

• and a function  g  encoding your beliefs about your credence p.

The Bayesian network can directly be used to make predictions. Making a prediction here is nothing else than calculating the probability of an event.

• In our case, we can calculate 

P(B=True) = integral of  P(B=True | q) dP(q)  over all possible values of q
= integral of  P(B=True | q) dP(q | p) dP(p)  over all possible values of q and p
= integral of  f(p) g(p) dp  over p=0…1   (if flow=fhigh=f, otherwise a little more complicated)

• For example:

  • If we consider the uniformly foreacting agent with f(p) = p and believe that we will assign credence p = 0.3 for sure, then  P(B=True) = 0.3  and we are happy.

  • If we consider the uniformly foreacting agent with f(p) = p and believe that we will assign either credence p=0.3 or p=0.8, each with probability 50%, then  P(B=True) = 0.55  and we are unhappy. 

  • If we consider any f for which there is at least one possible value p* of p such that f(p*)=p*, and believe that we will assign credence p = p*, then  P(B=True) = f(p*) = p*  and we are happy.

  • If we consider an f for which there is no possible value p with f(p)=p, and believe that we will assign some particular credence p* for sure, then we get P(B=True) != p* and will be unhappy.

  • But: If we consider an f for which there is no possible value p with f(p)=p, and believe that we might assign any possible credence value p between 0 and 1 with some positive probability, then we indeed get some result P(B=True) between 0 and 1, and since we have attached positive probability to that value, we should be happy since the result does not contradict what we believed we would predict!

Let’s assume we interpret the node p as a control variable of a rational us with some utility function u(B), let’s say u(B=True) = 1 and u(B=False) = 0. Then we can use the Bayesian model to calculate the expected utility given all possible values of p: E(u(B) | p) = q = (flow(p) + fhigh(p)) / 2. So a rational agent would choose that p which maximizes (flow(p) + fhigh(p)) / 2. If this is all we want from the model, we don’t need g! So we only need an incomplete Bayesian network which does not specify the probability distributions of control variables, since we will choose them.

Things get more interesting if u depends on B but also on whether p = q, e.g. u(B,p,q) = 1_B=True – |p – q| . In that case, E(u | p) = f(p) – |p – f(p)|. If f(p) > p, this equals f(p) – |f(p) – p| = f(p) – (f(p) – p) = p. If f(p) < p, this equals 2f(p) – p.

Let’s assume the rational us cannot choose a p for which f(p) != p.

Excursion: If you are uncertain about whether your utility function equals u1 or u2 and give credence c1 to u1 and c2 to u2, then you can simply use the function u = c1*u1 + c2*u2.

Bayesian updating is the following process: 

• We keep track of what you know (rather than just believe!) about which combinations of variable values are still possible given the data you have. We model this knowledge via a set D: the set of all possible variable value combinations that are still possible according to your data (Formally, D is a subset of the probability space Omega). If at first you have no data at all, D simply contains all possible variable combinations, i.e., D=Omega.

  • In our case, D and Omega equal the set of all possible value triples (B,p,q), i.e., they are the Cartesian product of the sets {True,False}, the interval [0,1] and another copy of the interval [0,1]: 

    • D = Omega = {True,False} x [0,1] x [0,1]

• Whenever we get more data:

  • We reflect this by throwing out those elements of D that are ruled out by the incoming data and are thus no longer considered possible. In other words, we replace D by some subset D’ of D.

  • Then we calculate the conditional probability distribution of those events E we are interested in, given D, using Bayes’ formula:

    • P(E | D) = P(E and D) / P(D)

At this point, we might be tempted to treat the value we derived for  P(B=True)  on the basis of some choice of f and g as data about p. Let’s consider the consequences of that. Let’s assume we start with some fixed f, g and with no knowledge about the actual values of the three variables, i.e., with D₀ = Omega = {True,False} x [0,1] x [0,1]. We then calculate P(B=True) and get some value p₁ between 0 and 1. We treat this as evidence for the fact that p = p₁ update our cumulative data to D₁ = {True,False} x {p₁} x [0,1], and update our probabilities so that now P(B=True) = f(p₁). If the latter value, let’s call it p₂, equals p₁, we are happy. Otherwise, we wonder. We have then several alternative avenues to pursue:

• We can treat the result P(B=True) = p₂ as another incoming data about p, which needs to be combined with our earlier data. But our earlier data and this new data contradict each other. Not both can be true at the same time, so the statement S₁: p = p₁ , that was suggested by our earlier data is false, or the statement S₂: p = p₂ that was suggested by our earlier data is false. If we consider that S₁ is false, we must consider why it is false since that might enable us to draw valuable conclusions. S₁ was derived purely from our world model, parameterized by the functions f and g, so either at least one of those functions must have been incorrect or the whole model was incorrect. 

  • The shakiest part of the model is g, so we should probably conclude that our choice of g was incorrect. We should then try to find a specification of g that does not lead to such a contradiction. We can only succeed in doing so if there is a value p* for which f(p*) = p*. If such a value exists, we can put g(p*) = infinity (remember, g specifies probability densities rather than probabilities) and g(p) = 0 for all p != p*, i.e., assume from the beginning that we will predict p* for sure. But if such a value p* does not exist, we cannot choose g so that the contradiction is avoided. 

  • In that case, something else about the model must have been incorrect, and the next best candidate for what is wrong is the function f. Since no p with f(p)=p exists, f must be discontinuous. Does it make sense to assume a discontinuous f? Probably not. So we replace f by some continuous function. And et voila: now there is some value p* with f(p*)=p*, and we can now choose a suitable g and avoid the contradiction. 

  • If we desperately want to stick to a discontinuous f, then something else about the model must be wrong. I think it is the idea of the agent being able to know p with certainty, rather than just being able to measure p with some random measurement noise epsilon. I suggest adding two more variables, the noise epsilon and the measurement m, and modify the formulae as follows:

    • epsilon ~ N(0,1),  i.e., Gaussian noise

    • m = h(p, epsilon)  for some continuous function  h  that represents the influence of the random noise epsilon on the agent’s measurement m of p.

      • For example: h(p, epsilon) = expit(logit(p) + sigma epsilon)  for some magnitude parameter sigma > 0.

    • q = f(m)  rather than  q = f(p)

With this modified model, we will get

P(B=True) = integral of  P(B=True | q) dP(q)  over all possible values of q
= integral of  P(B=True | q) dP(q | m) dP(m | p, epsilon) dP(p) dP(epsilon) over all possible values of q, p and epsilon
= integral of  E_{epsilon~N(0,1)}f(h(p, epsilon)) g(p) dp  over p=0…1,  where E is the expectation operator w.r.t. epsilon

If our choice of g assigns 100% probability to a certain value p₁ of p, the calculation results in

p₂ := P(B=True) = E_{epsilon~N(0,1)}f(h(p₁, epsilon)),

which is a continuous function of p₁ even if f is discontinuous, due to the “smearing out” performed by h! So there is some choice of p₁ for which p₂ = p₁ without contradiction. This means that whatever continuous noise function h and possibly discontinuous reaction function f we assume, we can specify a function g encoding our certain belief that we will predict p₁, and the Bayesian network will spit out a prediction p₂ that exactly matches our assumption p₁.5

Acknowledgement

A huge thanks to Jobst Heitzig for checking my writing and for doing the heavy lifting of the “Applying Bayesian modeling and updating to the foreacting agent” section of the post. He finds it incomplete, but I appreciate it anyway. And special thanks to the countless people who provide free secondary literature on philosophy which makes me understand these problems better. You guys deserve my tuition money.

The objection can be introduced with the following thought experiment. Say you had to choose between:

Option 1: A 99% chance of causing the death of everyone and a 1% chance of bringing about nine trillion maximally happy people.

Option 2: A guaranteed outcome of creating nine billion maximally happy people.

According to the principles of classic utilitarianism, the expected utility from Option 1 (0.01 x 9 trillion = 90 billion) exceeds that of Option 2. Yet, it gives us a glaring 99% chance of a disastrous outcome.1

Negative Utility

The Wagering Calamity Objection becomes even more alarming when we frame it in terms of negative utility. Say you face the following decision:

Option 1: A 99% chance that everyone on earth gets tortured for all of time (-100 utils per person) and a 1% chance that a septillion happy people get created (+100 utils pp) for all of time

Option 2: A 100% chance that everyone on earth becomes maximally happy for all of time (+100 utils pp)

Let's assume the population in both these scenario's remain stable over time (or grow similarly), Expected Value Theory (and classic utilitarianism by extension) says we should choose option 1, even though this has a 99% chance of an s-risk, over a guaranteed everlasting utopia for everyone.

We can make it more pernicious by combining it with the repugnant conclusion to give option 1 a 1% chance of creating an enormous amount of people whose lives are barely worth living (+1 util pp), but are still in aggregate more utils.

Morality and the Role of Risk

Classic utilitarian calculations seem to disregard our innate moral preferences that lean towards risk aversion. Given the scenarios presented, I think most people would choose Option 2 or Option B, valuing the certainty of a positive outcome over an immensely rewarding but perilously risky alternative.

Even if we create scenarios with things like 50/50 odds or even favorable odds, I think most people would have a moral instinct to not choose the option with the possible calamity.

The Wagering Calamity Objection compels us to think beyond mere arithmetic. It asks us to consider the moral weight of risk and the ethical implications of near-certain negative outcomes.

Afterword

This objection seems to be related to Pascals-mugging (and infinite ethics), but it isn’t the same thing. I tried looking for it online but couldn’t find it. I asked the ‘askphilosophy’ subreddit about it and they couldn’t find it either. Please let me know if this objection already exists.

Subscribe now

Unequal money

I would like to push back against this idea a little bit.
The marginal value of money decreases as you get more of it. A hundred dollars might be a vitally important amount of money for a poor person, and not even noticeable for a rich person. So if you bet against a person with less money you are wagering less of your happiness than they are.
If they have health problems (and live in a country with bad healthcare) this bet increases their risk of death, which it doesn't for you. It seems to me that betting against someone who is poorer than you is morally dubious.

What if you think a poorer person is wrong and you think it's important to publicly signal so? Well, you still make a public bet, just without involving money. If you suspect your interlocutor isn't honest, the fact that you suggest a public bet gives them a status incentive to be honest.
If you think this isn't enough you can suggest that you put in a bit more money to compensate for the difference in wealth. This might not work if you yourself are too uncertain about the outcome, but in scenarios where you have a lot of confidence it might still be worth it in expectation (e.g. betting a $100 to $1 that the sun will rise tomorrow still seems like a good deal)

Another way is to bet with money, but give them the money once you win. However, you can only do this trick once, otherwise people might use this behavior to extract a constant stream of money from you. Alternatively, if you bet against a moderately poor person you might consider giving the profits to an extremely poor person.

Punishment for wrongthink

A bet is a tax on bullshit, but sometimes bullshit is necessary. Consider a closeted atheist living in a fundamentalist country where apostates are beheaded. I could extract a lot of value from their dishonesty by offering bets about various religious occurrences, which they'd have to take lest they out themselves. But this is cruel right?

And it's not just deceptive atheists. Say someone is poor and their only support network is the local church/religious community and/or they live in a country where apostates get beheaded. This person (unlike the atheist) might be an honest believer, but if the punishment for failing to self-deceive is ostracization or death, maybe it becomes cruel to consistently start bets with them. (This line of thinking can be extended to other scenarios where someone is under heavy pressure to think or talk a certain way.)

It’s important to note that these considerations cut both ways. If you, a non-rich person, can extract value from someone like a televangelist, I think it becomes not just prudentially-, but also morally important to take that opportunity.

The standard way to resolve this is to deny that we have inconsistent preferences.3 But what if we accept their existence? Could there be a way to deal with them by making a non-classical preference logic?

Before we can can make a logic we need to know what the operators are. We consider preference statements of the form “I prefer φ to not-φ” and denote them as “Pref φ”, where “Pref” is a new modal operator representing preference. Let us also use the natural inference rule (N):

if we have Pref ¬φ (meaning “I prefer ¬φ to φ)
then we can infer ¬Pref φ (meaning “I do not prefer φ to ¬φ”)

Aka: If I want to not have a painting, we can infer I don’t want a painting4

Okay now that we have our operators and our rule, let’s see if we can make a preference logic.
One possible objection to the existence of inconsistent preferences is something which I’ll call a ‘preference explosion’. I will first write out the argument formally, and then in prose:

1. Pref φ & Pref ¬φ              -assumption

2. Pref φ                                -from 1, conjunction elimination

3. Pref φ ∨ Pref ψ                -from 2, disjunction introduction

4. Pref ¬φ                              -from 1, conjunction elimination

5. ¬Pref φ                              -from 4, using rule (N)  

6. Pref ψ                                -from 3 & 5, disjunctive syllogism

(and ψ can be any possible preference, hence preference explosion)

In prose: I both want a painting and want to not have a painting, from which we can logically infer that I either want a painting or kill a puppy. Since we can infer from my wanting to not have a painting, that I don't want a painting, we can infer that I want to kill a puppy.

This seems strange right? Yet rule (N) is quite natural and intuitive, and all the other inferences are valid in classical logic. If we take inconsistent preferences seriously we have to give up either:

rule (N): Pref ¬φ ⊢ ¬Pref φ
(If I want to not have a painting, we can infer I don’t want a painting)

conjunction elimination: φ & ¬φ ⊢ φ
(If someone has a preference for ‘not having a painting’, and also a preference for ‘having a painting’ we can infer someone has a preference for ‘having a painting’.)

disjunction introduction: φ ⊢ φ ∨ ψ
(If someone has a preference for ‘having a painting’ we can infer that they have a preference for either ‘having a painting’ or ‘killing a puppy’)

disjunctive syllogism: ¬φ, φ ∨ ψ ⊢ ψ
(From the fact that someone has a preference for either ‘having a painting’ or ‘killing a puppy’, and the fact that they also have a preference for ‘not having a painting’, we can infer they want to kill a puppy)

I think that last one, disjunctive syllogism (DS), is the one to reject. If we believe in inconsistent preferences we can no longer see it as a logically valid rule.

So DS is invalid, but that does not mean DS is a bad argument. DS is legitimate with consistent preferences. In a situation where someone just wants a painting (and doesn’t simultaneously not want a painting) DS becomes reliable. In such situations we can treat DS as if it’s deductively valid. So DS is inductively strong, and if we are a little uncertain about whether inconsistent preferences are involved, then it makes the conclusion likely: in most such situations, the inference is valid, but in some rare situations it is not.

Our belief that DS is valid might have arisen because inconsistent situations are rare, so DS works in most situations. Compare the inference of DS with the inference:

A is a proper subset of B, so A is smaller than B.
(E.g. Ants are a type of Bug, so there are less Ants than Bugs)

This inference almost always works, only in cases of infinities does it break down. If there are infinitely many Ants there are also infinitely many Bugs, so the two sets could both be equally (infinitely) big.
This ‘subset inference’ is inductively strong, but not deductively valid, because sometimes you are talking about infinities. Perhaps this is similar to DS, it almost always works, just not when talking about a special situation (infinities and inconsistencies, respectively).5

So the “subset inference” can’t be deductively valid because it treats infinities and non-infinities the same way. Is there something similar going on with “preference explosion”? The proof for “preference explosion” might mistakenly equate two slightly different ways to understand the disjunction “φ ∨ ψ”

1. “φ ∨ ψ” follows from φ alone. In this sense, because I know φ I can infer something else. Because I know that someone prefers “having a painting” I can therefore infer that they either prefer “having a painting” or “killing a puppy”
   
   which is slightly different from
   

2. “φ ∨ ψ” is somewhat equivalent to the ‘material conditional “if ¬φ then ψ”, in the sense that if it would turn out that ¬φ, we would know that ψ. This is the interpretation we have in mind when we know that at least one of φ and ψ is true, but we don’t know which one. We don’t know whether someone prefers ‘having a painting’ or ‘killing a puppy’ and we’re trying to figure it out, so if it turns out it’s not the first, it must be the second.

In the proof of preference explosion, we infer φ ∨ ψ in the first sense (we know which one), but we need the second sense to perform the disjunctive syllogism (we don’t know).6 This distinction between the different interpretations of a disjunction is not something that is captured in classical logic, but it might be something we want to capture. Maybe the material conditionals in classical logic just don’t capture the semantics of intuitive conditionals.

Special thanks to Jobst Heitzig for greatly improving this post, and to Daan Vernieuwe for slightly improving this post. And special thanks to the people who provide free secondary literature on the topic. You guys are the reason I understand these topics and are much more deserving of my tuition money.

Utilitarian: I’m sorry, why should I believe that you will let the kid live if I give you $1000?

Gunman: I’m the ‘trustworthy gunman’, google my name and you’ll see that I’ve always kept my word.

Utilitarian: [seeing it’s true] Hmmm, alright but I’m sorry I can’t give you the money because that would set a bad precedent. If people know I always give money to gunmen that would encourage people to start taking hostages and demanding money from me.

Gunman: It will be our little secret. Look at my trackrecord. I’ve threatened deontologists, virtue ethicists and even egoists, but utilitarians are curiously absent from the public record. That’s because they’re always concerned about precedent so I always promise to keep it a secret, and I keep my promises. Plus a utilitarian gave me this AI-powered collar that will explode if I try to remove it or expose the secret agreements I made with utilitarians. Here, see for yourself [shows proof].

Utilitarian: What about the kid?

Gunman: She’s too far away, she doesn’t even know this is happening.

Utilitarian: I feel bad for her but I know of a way to save a kid for $900 dollars so I’m going to spend the money on that intervention.

Gunman: Alright, give me $899 and I’ll spare her.

Utilitarian: How do I know you won’t come back tomorrow to threaten another kid.

Gunman: I promise. [points at the promise tracking collar]

Utilitarian: Shit.

Compassionate commitments may generate cruel behavior, even in those that are not inherently cruel (e.g. egoists). Maybe this is an argument against classic utilitarianism, or maybe it’s an argument against revealing that you’re a utilitarian (or compassionate).

Then Loudacris had the brilliant idea to make this hierarchy into an easily shareable picture:

And thus internet debate was rational forever.

I joke, but I do genuinely believe that this image helped the quality of internet-debate in some small way. Even if not many people see it, making these kinds of guides is a net benefit for public discourse. It also inspires people to think about this stuff and make more guides. Like how Scott Alexander created a related guide, and how Lukeprog and Black Belt Bayesian want to expand the hierarchy.

Black Belt Bayesian writes:

If you’re interested in being on the right side of disputes, you will refute your opponents' arguments. But if you're interested in producing truth, you will fix your opponents' arguments for them. To win, you must fight not only the creature you encounter; you [also] must fight the most horrible thing that can be constructed from its corpse.

Which Lukeprog argues should be the top layer called: "Improve the Argument, then Refute Its Central Point"

I don't know about you, but to me this new top layer feels separate from the other layers. This top layer, for one, should probably only be used after you’ve already refuted the existing argument. You don't want your interlocutor to feel like you are either misrepresenting or humiliating them. Improving an argument is still desirable, but don't sour the debate.

But secondly, this layer goes beyond countering your interlocutor and ascends into the realm of active truth seeking. Whereas the other layers are a linear process of knocking down an argument, this new layer is more circular. You can continue on creating and knocking down new versions of the old argument again and again, always gaining new insights but never reaching a perfect conclusion.

Nonetheless, I think this addition is extremely important. It shows people that the purpose of debate is not an adversarial brawl, but ultimately a way to achieve better understanding. The hope is this layer will change peoples mindset into a more curious and cooperative one.

Unfortunately no one has made an easily shareable visualization of this new hierarchy. So I decided to make one myself with two additional updates.

1) I hate it when there is a guide on the internet and I click on it to copy the text, only to realize it is a picture and I have to manually copy all the information while hoping that I don't make any mistakes. I decided to make a pdf, which not only solves this issue but should also make it easier to translate into other languages.

2) The old visualization implies that you can't have a solid argument without a strong base of name calling, so a pyramid might not be the best shape for this guide. I wanted the new version to look more like a staircase, where you are encouraged to climb to the top.

Here it is:

You can find a full size image, a pdf-file and a page-file here. If this doesn't work you can also find it here.

What do we do when we’re are uncertain about what the right thing to do is? What if you’ve read multiple ethical theories and don’t know which one to pick?
Here we can turn to the philosophical study of moral uncertainty

If you’re familiar with the field you can skip this introduction. For those of you unfamiliar: theories of moral uncertainty are decision procedures that allow us to pick between different options presented to us, even though we remain morally uncertain.
Here’s an example of one such theory: the parliamentary model. Say we find some moral theories more plausible than others, so we give different credences/odds to different theories. You might find Utilitarianism quite plausible so give it 50% odds of being true, while Kantianism and Contractualism get 25% each. Now we can imagine our mind as a parliament where Utilitarianism gets 50% of the seats, while the others get 25%. When presented with a moral dilemma, you pick the option with the highest number of votes in the parliament.

The (currently) most popular theory of moral uncertainty is “maximizing expected choice-worthiness”, which is a more advanced version of the parliamentary model. I will first briefly list some objections to this theory (don’t worry if you don’t know all the terms, I’ll mostly be relying on examples in the main body of the text), before introducing some new theories that use randomization.

If you want to know why we should care about moral uncertainty, you can read this post right here.

Problems with maximizing expected choice-worthiness

A popular approach to moral uncertainty is "maximizing expected choice-worthiness" (MEC) by William MacAskill. It states we should make decisions based on expected choiceworthiness — a framework similar to expected utility theory, but applied to moral theories. However, it has some issues.

1. It requires theories to be interval-scale measurable.
   This means that a moral theory needs to provide information about the numerical difference in utility/choice-worthiness, when comparing different options.
   E.g. Stoicism says that the difference between lying and killing is ten times bigger than the difference between being honest and lying.

2. It requires intertheoretic comparisons of value.
   We need to be able to tell to what extend one theory deems an option as more important than another option, when compared to a different theory.
   E.g. Utilitarianism says the moral difference between lying and killing is three times bigger than Kantianism says it is.

3. It falls prey to the infectiousness of nihilism.
   When an agent has positive credence in nihilism then the choice-worthiness of all actions is undefined.
   E.g. If you think nihilism has 1% chance of being true, you can't evaluate one option as morally better than another.

4. Some say it has a problem with fanaticism.
   It ranks a minuscule probability of an arbitrarily large value above a guaranteed modest amount of value.
   E.g. If you think christianity has a 1% chance of being true, and it gives people infinite happiness in heaven, you should choose it above your 99% chance that a naturalist-utilitarianism is true, which only posits finite happiness.

A while back I created an approach that wouldn't fall prey to these problems: the sortition model of moral uncertainty.

Problems with the sortition model of moral uncertainty

The sortition model prescribes that if you have x% credence in a theory, then you should follow that theory in x% of cases. So if you have 20% credence in Kantianism, 30% credence in virtue ethics and 50% credence in utilitarianism, you follow Kantianism 20% of the time, virtue ethics 30% of the time and utilitarianism 50% of the time. When you act according to a theory is selected randomly with the probability of selection being the same as your credence in said theory.

This approach doesn't fall prey to the problems of interval-scale measurability, intertheoretic comparisons of value, or the infectiousness of nihilism. Furthermore, the theory is fair, computationally cheap and doesn't generate problems with fanaticism and theory-individuation. However, it does have a big problem of its own, it violates the principle of moral dominance.

Suppose you face this decision:

The principle of moral dominance states that if an option A is better than B according to at least one possible moral theory and as good as B in all others, then you should choose A instead of B. The principle of moral dominance prescribes in this scenario that we shouldn't punch someone. However, if the sortition model randomly selects theory 1, then according to the sortition model it is equally appropriate to punch or not punch someone, directly violating the principle of moral dominance.

You could solve this by saying that once a theory doesn't eliminate all the options, the leftover 'appropriate' options are presented to another randomly selected theory who eliminates the options they deem inappropriate and so on, until only one option remains. However, this doesn't solve the sentiment that some choices are just higher stakes for some theories than others. Consider this decision:

Even with this solution, it's possible that the sortition model lands on theory 1 and chooses option A, despite it only being a very slight preference for theory 1, while all the other theories have a very strong preference for option B. We could solve this by normalizing the distributions (giving the theories "equal say"), but sometimes some theories genuinely care more about a decision than others (We'll return to this problem at the end of this post). Perhaps we give the selection process a random chance to switch to another theory, with the odds of that occurring being determined by how much the other theories prefer a different option. However, this adds computational complexity. Let's look at a different approach.

Runoff randomization

We might want to do is something in between "only letting theories assign options either permissible or impermissible" and "giving a numeric value to options".
What if we let the process go through several runoff rounds? Let's call this version runoff randomization. Say you face the following decision:

If you think we should strongly focus on impermissibility (e.g. certain strands of deontology and suffering-focussed ethics) you might want to first have a selection round that eliminates all the options that at least one theory considers impermissible (options C and D). You subsequently select in a second round the best option from the ones that remain (in this case it would be option A between A and B).

However this gives some weird results. Theory 1 and 2 strongly prefer options C and D, but because Theory 3 finds them impermissible it can eliminate them and cause its (slightly) preferred option to win. This can get especially weird if there are hundreds of theories under consideration, any one of which can shut down all the options except their favorite one.

Let's be a bit milder. Suppose you face the following decision:

A notable feature of this approach is that theories can give quantitative recommendations, qualitative recommendations or a combination of the two.
This doesn't matter because the idea is that we do three rounds of runoffs:

1. First we randomly select a theory. In said theory we eliminate the "impermissible" options.
   Let's say we randomly draw theory 1 and see that it only considers option D to be impermissible, we therefore eliminate option D.

2. Secondly we randomly draw another theory. In said theory we eliminate the "neutral" options.
   Let's say that this time we draw theory 2. It considers option D, B and A permissible, but we've already eliminated option D in the previous round, so the only thing we can eliminate is option C which it is "neutral" towards.

3. Lastly we randomly draw a final theory and we eliminate its "non-optimal" options so we only end up with the "optimal" options.
   Let's say that this time we draw theory 3. While it considers option C the optimal choice we have already eliminated options C and D. Between options B and A it considers option A to be the "optimal" choice, so we end up choosing option A.

If a theory eliminates all the remaining options, you draw again. E.g. if nihilism is neutral towards all the options (because it doesn't make a value judgement), you randomly select another theory. If in this process, you have exhausted all theories without reaching the next round, we go to the next round anyway, keeping all the remaining options in.

For any round the chance that a theory is selected depends on the credence you have in said theory. So if you have 98% credence in theory 1 and only 1% in the others, you have a 98% chance of selecting it in the first round, a 98% chance in the second and a 98% in the third.

Note that value and permissibility are not the same thing. A negative value for example doesn't necessarily mean an option is impermissible. If a theory assigns option A '-1000' and option B '-1' it might be that it doesn't consider option B to be impermissible, but rather optimal. Where a theory draws the line between merely assigning a low value and also considering an option impermissible varies from theory to theory.

Why you need randomization

So why would we still use randomization and not exclusively the runoff? Let's say you face the following decision:

If you start with the theory you have the most credence in, it's possible that you'll keep choosing options that most other plausible theories don't agree with. Randomization solves this.

Not randomizing also makes your theory vulnerable to "theory-individuation". Theory-individuation occurs when one theory splits into multiple versions of the same theory. For example, say you have 60% credence in utilitarianism and 40% credence in Kantianism:

However, you soon realize that there are two slightly different versions of utilitarianism, hedonistic utilitarianism and preference utilitarianism:

Without randomization Kantianism would dictate how you behave, even though your overall credence in utilitarianism hasn't changed.

Epistemological runoff

However, the various versions of both "maximizing expected choice-worthiness" and "the sortition model" require you to be able to assign a numerical percentage of credence to each theory. What if we aren't fans of Bayesian epistemology? Or what if we aren't able to do this in the real world?

We could have the same runoff that we did for moral value, but apply it to epistemology. So all the moral theories that our epistemological theory considers "impermissible" get eliminated, then the "neutral" ones and then the "impermissible" but not "optimal" ones. What's left are the moral theories we use our theory of moral uncertainty on.

What if we are uncertain about our epistemological theory? Don't we run into an infinite regress problem? I'm not sure how fruitful this runoff business is in meta-epistemology. Let's see if we can get a theory of moral uncertainty that doesn't require a runoff.

Convex randomization

If we use a convex hull we don't have to assign probabilities. Here's how that works:

• Order the options in an arbitrary way from 1 to k.

• For each theory Z, let V(Z) be the list of choice-worthinesses of all options according to this theory. I.e., the first entry is Z's evaluation of option 1, the last entry is Z's evaluation of option k.

• Interpret the entries in the list V(Z) as the coordinates of a point in a k-dimensional space.

• Let P be the set of all these points, one for each theory.

• Let H be the "convex hull" of the set of points P, i.e., all points contained in a tight rubber envelope around P, or, more mathematically, all points that lie on some straight line between any two points in P.

• Let R be a randomly chosen point in the interior of H (i.e., in H but not on the boundary of H), chosen from the uniform distribution on H. (Note that by excluding the boundary, we in particular exclude the possibility that R equals any of the original points in P.)

• Choose the option that corresponds to the largest entry in R. I.e., if the third entry in R is the largest, choose the third option.

Let's look at an example. Suppose you face this decision:

We now need to interpret the choice-worthiness as coordinates, with option A being the x-axis and option B being the y-axis.

Now we draw a convex hull around it. Imagine it's a tight rubber envelope around all points.

Now we randomly pick a point R inside this space (without its boundary).

Point R gives option A a choice-worthiness of 2 and B a choice-worthiness of 3, so we choose option B.

If we make a diagonal line we can see it even more clearly. The area of the convex hull that is beneath the blue line (aka the green area) represents the odds that R will land on option A. The area of the convex hull that is above the blue line (aka the pink area) represents the odds that R will land on option B. As you can see there's a bigger chance that this process will choose option B.

The convex hull ensures that dominated option cannot win. This is because when S is dominated by T, then there is at least one theory that rates T better than S, and that theory has a nonzero weight in the mixture corresponding to R (because we exclude the boundary, on which that theory might make a zero contribution to R), so T will always have at least one larger entry in R than S has.

It also makes the theory clone-consistent: If someone takes a theory T, modifies it in a negligible way and adds it as another theory T', then the set H will change only negligibly and hence the winning probabilities won't change much. Similarly, it combats the problem of theory-individuation, splitting a theory into multiple similar theories only changes H very slightly.

Problems with convex randomization

Convex randomization (CR) solves the problems of violating moral dominance (because of the convex hull), theory cloning/theory-individuation, and arguably the infectiousness of nihilism (because nihilism doesn't prescribe coordinates so it doesn't get added to the graph), however it does require intertheoretic comparisons of value and ratio-scale measurability.

We could say that theories find actions either permissible, impermissible, or are neutral on the matter. We could represent this as a coordinate system where permissible actions get 1, impermissible actions get -1 and actions that are deemed neutral get 0. However, while this might be preferable for categorical theories like Kantianism, this does collapse the nuance of numerical theories. If prioritarianism gives an action a choice-worthiness of 100, while utilitarianism gives it a 2, they will both be collapsed down into permissible (1) despite the huge difference between them. Perhaps adding a runoff with an "optimal" round will solve this, but the whole point of using the convex hull is that we no longer need a runoff.

Another problem is that CR is more computationally complex than the sortition approach. However, computational complexity isn't generally seen as a weak point by philosophers.

CR also requires there to be more theories than options. If there aren't, the interior of the 'envelope' will be empty. This isn't really a problem since almost any theory will have some continuous parameter (such as a discount rate or degree of risk-aversion) which makes it become infinitely many theories. Even if you have infinitely many discrete options, the infinite continuous theories will still outnumber them (ℵ₀ < ℵ₁).

CR has a problem with fanaticism, although to a lesser extent than MEC. If a theory says option A has 10^1000 choice-worthiness, R won't always land on option A but it will be biased towards it. Similarly, if a theory says option A has an infinite amount of choice worthiness, R won't literally always land on A, but will practically always land on A. However, MEC's solution of using amplification or using normalization to give different theories 'equal say' could also be used by CR. If you don't think we should normalize at every choice, here's a different approach. Perhaps all theories get an equal choice-worthiness 'budget' over a span of time (e.g. the agents lifetime). The theories can spent this budget however they like on any options that are presented giving theories more power to influence events they consider crucial. This would tackle choice-worthiness in the same way linear and quadratic voting tackle votes (and the same debate between the linear and quadratic approach exists here).

CR makes you less single-minded, it stops a moral theory from dominating even though it only has slightly more credence (e.g. 51% vs 49%). However, because CR uses randomization you don't appear to be entirely consistent to an outsider, which might make you harder to coordinate with. This could be solved by using the centroid instead of a random point.

Using centroids 

The centroid is the "center of gravity" of the convex hull, and it can be seen as the most natural "representative point" inside that set. Interestingly, that approach can also lead to some form of  "ex-post" credence values for theories, relative to the preference data. This is because the centroid can be expressed as a convex combination of all the points representing the individual theories, similar to barycentric coordinates. Each such convex combination assigns coefficients to all theories, and these coefficients sum up to 1; so they could be interpreted as credence values for the respective theories. It remains however to clarify which convex combination is the most natural one since in general there will be several.  

However, you might conversely think that the 'centroid theory' of moral uncertainty makes you too morally rigid. One solution could be that the randomization process is biased towards the centroid (i.e., the further from the centroid a point is the lower the probability of point R appearing there) with the strength of the bias being determined by how much you value consistency. What this allows you to do is regain a kind off credence distribution (like MEC) without actually needing probabilities.

References

MacAskill, W. (2013). The Infectiousness of Nihilism. Ethics, 123(3), 508–520. https://doi.org/10.1086/669564

Macaskill, W., Bykvist, K. & Ord, T. (2020). Moral Uncertainty. OXFORD UNIV PR.

This post was co-authored with Jobst Heitzig and was first posted on the EA forum March 29th

A disrespectability cascade is the reverse: something that was once neutral or respectable becomes tainted and socially unacceptable as more people reject it. This often happens when a word gets associated with a disliked group or an idea gets labeled as offensive. For example, once a phrase is linked to extremists, even neutral people start avoiding it, pushing it further into disrepute.

Both processes can be arbitrary or strategic, driven by activists, media, or social momentum. Sometimes, a new word replaces an old one for practical reasons—other times, it's just a cascade effect, where people switch simply because they don’t want to be left behind. The key question isn’t just whether to accept or reject these cascades, but when to resist and when to go along with them.

I think it becomes more important to struggle against a cascade the more annoying the new word is. There are a variety of factors that contribute how annoying it is, here are five of them:

1. Is the new word longer? (Because we want to waste less time)

2. Is it easier to pronounce/type? (Because we care about easy of use)

3. Is it less descriptive? (Because we care about people quickly grasping the concept)

4. Is it dissimilar to the old word? (Because we want to minimize transaction costs)

5. Is it likely to cause another cascade? (Because we dislike cascades)

If tomorrow people say that "black people" is bad but "black peoples" is good, I would jump on board once 60% of the population is on board, because it scores well in all aspects (except 5). If however people want to replace it with "abcedifoguhajekilomun" that's worse in all aspects (except 5) so I would strongly push against it (maybe only jump on board at 95%).

Conversely if people want to reclaim a word and start a respectability cascade I would jump on board rather quickly (I like having the freedom to use a lot of words), but I would jump on board even more quickly the better it scores on those different aspects.

For example, I'm on board with reclaiming the word "queer" because lgbt is longer and more annoying to say, requires an explanation for a new user, and quickly causes another cascade (what about intersex? Okay we'll say lgbti. What about asexuals? Okay we'll say lgbtia etc).
Queer on the other hand scores well on all aspects (even transaction cost since we still have a lot of books lying around about "queer theory" etc).

I don't think we should be trying to put a number on this (e.g. I add 10% for every aspect it has) because a lot of it depends on social context. With rarely used jargon I value descriptiveness over brevity, with words I use in everyday life it's the other way around. Let's all agree we jump onboard a disrespectability cascade at more than 50% and onboard a respectability cascade at less than 50%. How much more or less we'll change depending on a lot of hard/impossible to quantify social factors.

Note that this is just a heuristic and moral reasoning obviously takes priority. Queer people wanting to make "queer" a respectable synonym of "lgbt" is fine. But if the nazi-party wants to start a respectability cascade to make "holocaust" a respectable synonym of "morality" you should (probably) resist jumping on board even once the majority of the population has.

This heuristic can be adapted/used for other respectability cascades too (like clothing, tattoos, style guides etc).

There have been many models proposed to resolve moral uncertainty, but I would like to introduce one more. Instead of acting in accordance to the moral theory we are most confident in (my favorite theory) or making complex electoral systems (MEC, the parliamentary model), we might want to pick a moral theory at random. Just assign to every moral theory you know the probability of how confident you are about this theory, put them in a row from least to most likely (or any sequence really) and pick a random real number between 0 and 100. E.g: Say you have 1% credence in 'Kantian ethics', 30.42% in 'Average utilitarianism' and 68.58% in 'Total utilitarianism' and you generate the random number 33, you will therefore pursue 'Average utilitarianism'. Whenever you update your probabilities you can reroll the dice (another version would be that you reroll at a fixed frequency of intervals, e.g every day). Here are some of the advantages and disadvantages of this model.
 

The Good

1. It represents your probabilities

2. It is fair, every theory gets an equal chance

3. It is easy to understand

4. It is fast

5. It stops a moral theory from dominating even though it only has slightly more credence than the second largest theory (e.g 49% vs 51%)

6. It stops a moral theory from dominating even though it has a minority credence (e.g 20%, 20%, 20%, 40%)

7. It stops the problem of “theory-individuation”

8. It has no need for “intertheoretic comparisons of value”

9. It makes you less “fanatical”

10. It is cognitively easy (no need to do complex calculations in your head)
     

The Bad

1. Humans need to use something other than their brain (dice/computers/cloud-patterns) to choose randomly (for an A.I this would not be a problem)

2. You're not considering a lot of information about the moral theories. This could lead to you violating “moral dominance” e.g picking a theory that decides on an option that it doesn't have much stake in while another theory screams from the sideline (this problem could potentially be solved by making the 'stakes' an additional metric for deciding any given option, but that increases the complexity)

3. It makes you more inconsistent and therefore harder to cooperate with

4. Someone might get a wrong impression of you because they met you on a day of very low probability

Overall I'm not really convinced this is the path to a better model of moral uncertainty (or value uncertainty, since this model could also be applied there). I think some variation of MEC is probably the best route. The reason I posted this was because:

1. Maybe someone with more expertise in moral uncertainty could expand upon this model to make it better

2. Maybe sortition elements could be included in other theories to improve them

3. Maybe sortition elements could be included in other theories to make them more useful in practice, since sortition is so easy without sacrificing fairness

EDIT: Here’s the updated model: