Integrity Objection Explained

Preliminary objection to Williams: Negative responsibility itself is not problematic
Some philosophers such as John Harris and Nancy Davis argued that negative responsibility itself is not problematic, whereas Williams said that it is the cause of the problem of utilitarianism (Harris [1974], Davis [1980]). Intuitively, it seems that negative responsibility itself might be a natural way of thinking ethically. If there is an evil agent, we will naturally try to stop them. So we need to reinterpret Williams' argument against utilitarianism so as to answer this preliminary objection.

Reading Williams: The gap between the internal viewpoint and the impartial viewpoint
Here is my reading of Williams which is similar to recent powerful readings (e.g. Chappell & Smith [2016]). What Williams objects to is not negative responsibility itself, but the utilitarian way of demanding negative responsibility. According to utilitarianism, especially in the case of negative responsibility, an agent is required to look at the situation impartially. We should impartially calculate the utility involved in the world. This is as if an agent looks at the situation from the point of the view of the universe as classical utilitarian Henry Sidgwick put it (MSH:169). This impartial viewpoint is, however, essentially different from an agent's internal viewpoint which is based on its projects to the extent that the impartial viewpoints may alienates the internal viewpoint. An agent's internal viewpoint should have an essential importance in an agent's deliberation, but its first-person importance is obliterated in the utilitarian viewpoint. This gap between two viewpoints is illustrated as follows.

Summary of this section
Williams' integrity objection should be read as the critique of the utilitarian point of view rather than the critique of negative responsibility itself. What is problematic, according to Williams, is the utilitarian way of demanding negative responsibility in which an agent's internal viewpoint loses its first-person authority, and, thus, the values of integrity and practical necessity are unintelligible. Hence, we do not ethically view the situations of George and Jim as utilitarians view it: from the point of view of the universe.

Supplement: "One thought too many" objection and Williams' critique of indirect utilitarianism
Besides the integrity objection, Williams proposed other objections to utilitarianism: the "one thought too many" objection and his critique of indirect utilitarianism, both of which are also famous. These objections become, I think, more understandable given that Williams' point is the gap between two viewpoints.

One thought too many
In his paper "Person, character and morality", Williams raised an illuminating example of man trying to save his wife over two drowning people because it is her. This kind of personal reason sounds natural, but moral theories do not admit this. By contrast, utilitarians would say that choosing one's family in this situation would create the greater utility. Williams, however, argued this kind of impartial justification as one thought too many as we are directly justified to save her because it is her (ML:18). In this objection, again, we can see the gap between the internal viewpoint and the impartial viewpoint. The personal motivation is internal, but the impartial utilitarian viewpoint cannot make sense of it.

Critique of indirect utilitarianism
There is a variant of utilitarianism, two-level theory, according to which we ought to have two levels of ethical thinking: the intuitive level in which we intuitively deliberate and the critical level in which we critically evaluate our intuitive deliberation according to utility calculus. Though two-level utilitarianism seems to avoid the one thought too many objection, it is susceptible to Williams' critique again. Williams casted doubt on the utilitarian supposition that the intuitive level is systematically subject to the critical level (ML:52-3, MSH:167). It is, according to him, absurd to suppose that the intuitive level is systematically subject to the utility calculus, as the intuitive level has its independent depth. For instance, if George has commitment to the rejection of CBW but loses its commitment as soon as he critically evaluate the utility involved in the getting his job, we can say that he did not have a deep commitment. In sum, it is absurd to suppose that our internal viewpoint (intuitive level) is subject to the impartial viewpoint (utilitarian reflection), because there is a crucial gap between them.