This book is just a philosophical/methodological disaster.
Layard cites a study by Carol Ryff that purports to show that “purpose in life, autonomy, positive relationships, personal growth and self-acceptance” are highly correlated with self-reported SWB. OK. No suprise. What does Layard think this shows? That Mill was wrong about the existence of qualitatively “higher” pleasures.
Thus Mill was right in his intuition about the true sources of happiness, but he was wrong to argue that some times of pleasure are intrinsically better than others
Of course, it doesn't even begin to establish this. It might simply establish that people who have more intrinsically valuable experiences tend to report that they are happier on the whole. That's what Mill thinks, after all.
Layard goes on to say that Mill's high/low distinction is “inherently paternalistic.” But the only reason to say that is if you, like Layard, are an irremediable paternalist, and take the existence of higher pleasures as a reason to coerce people into having more of them and less of the lower. That is, Mills distinction is paternalistic only if you think the fact that something has special value on one conception of value immediately implies that the state should do something about it. Absurd.
[S]ome unhealthy enjoyments, like that of the sadist, should be avoided because they decrease the happiness of others. But no good feeling is bad in itself–it can only be bad because of its consequences.
Now, I understand that that's just a restatement of Benthamite egalitarianism among pleasures, but it doesn't pass the straight face test, does it? Many emotions (or any “judgment sensitive attitudes”, in Scanlon's terms) are themselves morally evaluable. And it strikes me as exceedingly dubious to assert that the problem with taking pleasure in the rape of children, the torture of kittens, or the betrayal of those who trust you has to do with their consequences for happiness.
The reason Mill distinguishes between higher or lower pleasures is that the distinction is real, he's a good philosopher, and so sees that it must be accomodated within his theory. The problem with Mill's move for Mill is that it points beyond utilitarianism toward the independent value of properties, such as beauty, cognitive complexity, and truth in virtue of which higher pleasures are higher.