First, some think that the failure to specific that we are talking about carbon dioxide makes "carbon" a highly ambiguous modifier of pollution, and so quip that if carbon is pollution, we should all be getting rid of our diamonds (not to mention the carbon in each of our body's cells). However, in the context of contemporary political debate, to speak of a "price on carbon" or "carbon pollution" is an entirely understandable and acceptable shorthand. The context makes clear that we are concerned with mitigating the deleterious effects of an enhanced greenhouse effect from rising levels of carbon dioxide and other greenhouse gases. Granted, in certain circumstances this needs to be spelt out carefully and fully to avoid confusion, but in the daily cut and thrust of political debate, "carbon" is sufficient (and also manages to include a couple of the other non-CO2 GHGs, such as methane (CH4), though I suspect this is more happy accident than by design).
Second, and more importantly, some reject the phrase because they do not believe carbon dioxide ought to be classified with other harmful substances. This may be (a) because they think carbon dioxide is natural and pollution is unnatural; (b) because they believe that only substances that are directly toxic to life ought to be called pollution; or (c) because they think that carbon dioxide is harmless.
Regarding (a), this common position is based on a couple of basic scientific and philosophical confusions about the nature of pollution. Many naturally-occurring substances are classified as pollutants: mercury, asbestos, arsenic, just to name a few of the better known ones. Furthermore, almost every substance can be harmful in certain doses. Pollution is a relative term. Nothing is a pollutant in itself, but substances pollute when too much of them is found in an inappropriate location. Remember, it is possible to die of water poisoning, or oxygen poisoning.
Regarding (b), critics say that calling CO2 pollution implies that breathing ought to be regulated, as we exhale CO2 with every breath. Defenders sometimes reply by pointing to the possibility of carbon dioxide poisoning (which has historically caused a number of deaths). Yet the direct physiological effects of elevated carbon dioxide levels can be overstated in an effort to justify the use of the term "pollutant". I have seen research (can't find the link at the moment) that suggested that there would be no observable direct effect upon human physiology until over 1,000 ppm. Pre-industrial levels were about 275ppm and we're currently at 390 ppm, with the most commonly-cited goal of aiming to stabilise at 450 ppm (though this is considered by many climate scientists to still be highly dangerous; the last time the earth had CO2 concentrations above 400 ppm, sea levels were approximately 25 metres higher). So 1,000 ppm is a long way off and would mean we'd already burst through all kinds of very nasty threshholds (though remember that reaching 1,000 ppm by 2100 is not outside the realm of possibility if large positive feedbacks kick in). In the US, the Occupational Safety and Health Administration (OSHA) has set a permissible exposure limit (PEL) for CO2 of 0.5% by volume, which equates to 5,000 ppm, a level of atmospheric CO2 so staggeringly high that the last time they were anywhere near there was over 400 million years ago (for reference, dinosaurs don't appear in the fossil record until 230 million years ago). Well before we got anywhere near 5,000 ppm other effects of carbon dioxide would have wiped us out, so worrying about shortness of breath from global CO2 concentrations is a bit like worrying about how a bullet hole in your head might make it difficult to comb your hair. This is a red herring.
A better line of reply to those who believe the term "pollutant" ought to refer only to substances that are directly toxic to life is to speak of ocean acidification. Rising CO2 levels are leading to falling oceanic pH levels as the oceans and atmosphere reach a new gas exchange equilibrium. These startlingly fast (from a geological or ecological viewpoint) changes in ocean pH are already having measurable detrimental effects on a wide variety of marine life and are projected to become much worse as concentrations rise. This is a direct physiological harm of carbon dioxide that does not rely on complex human social changes and so alone justifies calling this dangerous substance a form of pollution. Nonetheless, it is directly dangerous only to certain critical forms of marine life.
And so we reach (c), which is, I suspect, what really drives this discussion. The quibbles above are really just extra confusions muddying the waters. The truly vital issue is whether the climatic effects of rising levels of CO2 and other greenhouse gases are on balance harmful or not. Considering all the likely indirect effects - increasing heat waves, droughts, floods, extinctions, sea level rise, habitat loss, surface ozone pollution, ocean acidification, public health problems, and so on (not to mention the likely knock-on effects of increased food stress, water stress, migration and conflict) - our present trajectory of substantially CO2-driven climate change will almost certainly be disastrous on human health and well-being, indeed is potentially catatrophic. So I have no qualms about labelling CO2 a pollutant when we are talking about the volumes of it currently being dumped into the atmosphere (and these enormous quantities mean it is facetious to reply with a comment about breathing or soft drinks, as some do in order to ridicule the idea of CO2 as a pollutant). If you think these impacts are implausible, then you would obviously have a problem with calling CO2 a pollutant. The physiological point becomes a distraction. But rather than having a conversation about definitions, it is far more honest and direct simply to have the debate about the impacts of climate change. This complex and evolving scientific debate continues with much energy in the peer-reviewed literature, though it must be acknowledged that, with the exception of a handful of fringe figures, the mainstream debate is not between those who think impacts will be bad and those who think they will be minor, but between those who think that impending climate changes spell human misery on a scale never before seen and those who think it is much worse than that. The debate is not between climate change being bad vs neutral (or even good); it is between disastrous and utterly catastrophic. There are many more publishing climatologists who are worried about the fall of civilisation and even the extinction of humanity than there are who believe the impacts will be minor or even beneficial. In this context, even if the outcomes resulting from complex causal chains involve other factors as well (not least human social, economic, political and cultural systems), nonetheless, calling carbon dioxide a pollutant is quite logical - as is taking action to slash our emissions as quickly as we can.