Whole Dog Journal's Blog February 1, 2018

Clean Label Project: Lots of Unfulfilled Dog Food Review Promises

Posted at 03:03PM - Comments: (20)

How legitimate is the Clean Label Project's dry dog food label ratings? WDJ editor Nancy Kerns gives us her own Clean Label Project review concerning the company's evaluation of dog food.

In the comments of last week’s blog post, several readers asked us what we thought of the pet food ratings by the Clean Label Project (CLP), a “nonprofit focused on health and transparency in consumer product labeling.”

We wish we could herald the work of the group, or even join forces with it to test foods, kick ass, and take names. The Clean Label Project's stated vision, mission, and values sound terrific.

We can’t critique any of its work other than its dog food commentary, which, in our opinion, is currently so flawed as to be without any practical use.

Also, glaringly, Clean Label Project shoots itself in the foot by offering dog food manufacturers a certification program, which is described most fully here:

“Clean Label Project’s Certification program is for the highest performing products. Not only did the individual products of these brands perform exceptionally well in our initial unannounced sampling and testing, they have voluntarily signed on to having the Clean Label Project continue to randomly sample and test a subset of their products to ensure ongoing compliance with Clean Label Project standards.”

Can we trust the Clean Label Project's dog food ratings?

It’s not stated anywhere, but our assumption is that the program is fee-based – and as soon as a manufacturer is paying for its foods to be rated, its foods will likely receive a good rating. This sort of structure is nearly impossible to implement without at least the appearance of conflict. Only a couple of dog food manufacturers are identified as having joined the certification program, and lo and behold, they have five-star ratings and appear on the top of the first page of five-star products.

But we have a lot of other nits to pick with the Clean Label Project ratings. We don’t usually comment on other sites or individuals who rate or review dog food, but we were compelled to do so in this case out of sheer disappointment. When we first heard of the Clean Label Project, we hoped that the organization had somehow managed to fund a significant number of validated, independent lab tests of dog food, searching for contaminants and nutrient levels that were out of spec, and plainly report the results.

Instead, the site assigns “ratings” of the products tested. These ratings are described as based 60 percent on the detected presence of heavy metals (the explanation calls out arsenic, cadmium, lead, and mercury); 20 percent on the detected presence of by-product contaminants (acrylamide and mycotoxins), 20 percent on the detected presence of “process contaminants” (here, the site includes ammeline, amelide, antibiotics, BPA, melamine, pesticides, and cyanuric acid).

Unfortunately, there is also an embarrassing little math problem in the graphic that explains the Clean Label Project evaluation process, explaining that the CLP also weighs “nutritional superiority” as 20 percent of the food’s score, leading to a potential score of 120 percent. Or maybe they meant that the scores for the contaminants overlap?

clean label project review

Let’s put aside the “nutritional superiority” for a moment and go back to the contaminants. There isn’t any explanation of the difference between a “by-product contaminant” and a “process contaminant,” and only some explanations of how these things might come to be present in food.

As just one example, the site names acrylamide as a “by-product contaminant” and starts its description of the substance this way: “Acrylamide is an extremely toxic chemical used in manufacturing.” We don’t know about acrylamide being “used” in manufacturing of any food; the most important thing to know is that it is created when starchy food ingredients are cooked in at high-enough temperatures. Most notoriously, it gets created when potatoes are made into potato chips and French fries; these foods have the highest levels of acrylamide in commonly consumed human foods. To be fair, the site adds that acrylamide “is used in the treatment of water, and it can sometimes wind up in soil through that route. It is also found in tobacco, and made naturally when certain plants (like potatoes) are cooked.”

But that’s burying the lede. It’s been generally acknowledged that acrylamide is produced during the cooking process of dry dog foods. But, to our knowledge, no one has attempted to quantify the amount of acrylamide that is proven to be harmful to dogs. If we knew which foods contained the most acrylamide and which foods contained the least, would we use that information when selecting a food for our dogs? We might, especially if all other factors under consideration were equal, but the Clean Label Project doesn’t give us this information “straight.” Without reporting the actual numbers - the parts per million or parts per billion that the CLP says its has measured - we have no context. How does the level of acrylamide in a particular dog food compare to foods we eat? If there is “only” as much acrylamide in our dog’s food as there is in our breakfast cereal, maybe we shouldn’t be all that concerned about it.

The same goes for antibiotics and pesticides and even the heavy metals. Without reporting the actual test results, and offering some specific values of other products we are familiar with, we don’t know how concerned to be.

But, in our opinion, the project really wanders off into the weeds when it attempts to factor “nutritional superiority” or “ingredient quality” into its ratings.

The most in-depth discussion that we could find of how Clean Label Project defines “ingredient quality” was a single paragraph on a page that discussed how it changed its ratings from its first report on dog food to its second. Here is the paragraph:

  1. We’ve factored in ingredient quality.We know you want a quality, nutritious food for your dog or cat in addition to one that is low in industrial and environmental contaminants. While it’s always best to consult your veterinarian for your pet’s specific nutritional needs, we have created a system to help. Not all pet food ingredients are created equal – some products use preservatives, artificial colors or chemicals, while other products do not. Some products are dedicated to using quality meats, vegetables, and starches, while others use loopholes to include lower quality ingredients. Our ingredient quality system captures this, rewarding products for using a smaller number of quality, transparent ingredients rather than a large number of less regulated ingredients.

We could spend a day critiquing each sentence in that paragraph, but the most glaring problem – and the one that creates the most dissonance between the CLP’s stated goals and its actual ratings – is the last sentence.

This definition of the rating for ingredient quality is comparing a bag of organic apples to a truckload of Chinese oranges; there are simply too many factors, none of them defined, in the explanation of this segment of the rating criteria.

The idea that a smaller ingredients list is a good predictor of quality is alluring – but there are so many exceptions that the number of ingredients in a food cannot be used as a reliable criterion in and of itself.

Worse, there are no indications of what the Clean Label Project might call a “transparent” ingredient, or a “less regulated ingredient.” What are they talking about? Without definitions and examples, the consumer is left without any context here, and has to look at the food ratings themselves to get a general idea of what the CLP might be measuring with these definitions.

So, you go to the ratings page and start clicking on specific foods. The ratings start off with five-star products - the highest-rated foods. When you try to select “dry dog foods,” you see there are freeze-dried foods intermingled with kibble – these are incredibly different product categories, but, okay. You click on a specific food. You already know that it has been given a five-star rating, but you want some details.

Some Clean Label Project Inconsistencies:

Unequal Category Placements of Dog Food

While each of the foods have been given an overall rating of one to five stars, some of the foods have been rated in four categories (heavy metals, process contaminants, by-products contaminants, and nutritional superiority), some have been rated in three categories (product purity, product value, and product nutrition), and some were rated in only two categories (purity and value). Huh?

clean label project review

No Test Results to Base Ratings

There are no test results posted that could put these ratings into context. The only explanation of the star ratings is this: “The star rating you see tells you whether the product is average (3 stars), below average (1 star), or above average (5 stars) in terms of the overall purity (lower levels of contaminants).”

The List Isn't Ordered

While the first page of the ratings starts out with five-star products, there is no indication of how else the products have been ordered. One might imagine that they are rated from the highest to the lowest star rating, but, in fact, as we were examining the list of dry dog foods, we found a product with a three-star rating on the first page of five-star foods, so there goes that theory. Again, without any values from test results being published, and given that they are not listed alphabetically by star rating, a viewer has no way to know why they are arranged in their order of appearance.

clean label project review

Does the CLP Even Look at Dog Food Ingredients?!

But any credibility that the project might have had goes completely out the window when you discover that foods like Ol’ Roy have been rated with five stars overall. Let’s compare what’s in Ol’ Roy with a one-star food from the last page of the dry dog food ratings: Earthborn Holistic Ocean Fusion variety.

clean label project review

clean label project review

The Clean Label Project doesn’t list product ingredients, but in case you weren’t familiar with what the two above-named foods generally contain, here are their ingredients lists:

  • Ol Roy Complete Nutrition Dry Dog Food Ingredients:Ground Yellow Corn, Meat And Bone Meal, Soybean Meal, Poultry By-Product Meal, Animal Fat (Preserved With BHA And Citric Acid), Corn Gluten Meal, Natural Flavor, Brewers Rice, Salt, Potassium Chloride, Color Added (Titanium Dioxide, Yellow #5, Yellow #6, Red #40, Blue #2), Choline Chloride, Zinc Sulfate, Vitamin E Supplement, Ferrous Sulfate, Zinc Oxide, Niacin, Copper Sulfate, Vitamin A Supplement, Biotin, Manganous Oxide, Calcium Pantothenate, Vitamin B12 Supplement, Thiamine Mononitrate, Pyridoxine Hydrochloride, Menadione Sodium Bisulfite Complex (Source of Vitamin K), Riboflavin Supplement, Sodium Selenite, Calcium Iodate, Folic Acid, Vitamin D3 Supplement, Cobalt Carbonate.
  • Earthborn Holistic Ocean Fusion Ingredients: Whitefish Meal, Sweet Potatoes, Ground Barley, Rye Flour, Potatoes, Menhaden Fish Meal, Canola Oil (preserved with Mixed Tocopherols), Ground Flaxseed, Potassium Chloride, Choline Chloride, DL-Methionine, L-Lysine, Taurine, Glucosamine Hydrochloride, Chondroitin Sulfate, Beta-Carotene, L-Carnitine, Vitamin A Supplement, Vitamin D3 Supplement, Vitamin E Supplement, Zinc Sulfate, Ferrous Sulfate, Niacin, Folic Acid, Biotin, Manganese Sulfate, Copper Sulfate, Salt, Calcium Pantothenate, Thiamine Mononitrate, Pyridoxine Hydrochloride, Riboflavin Supplement, L-Ascorbyl-2- Polyphosphate (source of Vitamin C), Zinc Proteinate, Manganese Proteinate, Copper Proteinate, Calcium Iodate, Sodium Selenite, Cobalt Carbonate, Vitamin B12 Supplement, Yucca Schidigera Extract, Rosemary Extract.

That just doesn’t make any sense! In our view, rating a food that contains artificial colors and preservatives, a cheap plant protein at the top of the ingredients list, and unnamed animal species sources of protein and fat, over a food with natural preservatives and a named animal protein source at the top of the ingredients list, is not doing dogs or consumers any favors. If they stuck to reporting the contaminant levels – not rating the products, but listing the results – we could get a little more excited. But these ratings just won’t help consumers identify good, healthy foods.

If the overall mission of this project really is “health and transparency in consumer product labeling,” more transparency is needed.

The Clean Label Project for dog food is NOT Whole Dog Journal-Approved.

Comments (20)

I went to clean label to check dry dog food and was so appalled with their 5 star ratings and wonder how in the world they arrived at these ratings. Those of us who study dog food will know that they are nuts

Posted by: slintonf@gmail.com | August 30, 2018 12:12 PM    Report this comment

And little known fun fact, dry pet food is amazing for keeping your pet's teeth clean. So if you're going to try a raw diet make sure to give them chews to grind their teeth on or brush them yourself. Febuary is National Pet Dental Health Month so I would recommend taking advantage of the cheap services. Also I'm not sure how the raw diets work, but I would recommend knowing the nutritional value of everything you're feeding and prehaps use supplements to make sure your dog's getting everything. That is another amazing thing about pet food is that it has everything your dog should need in one food.

Posted by: Ampu | May 18, 2018 5:47 PM    Report this comment

Actually the more I read the ingredients of the second brand the more I am concerned 😟 there's only fat from fish and plants and there's no fiber. Even corn has a little fiber. I guess at least the protein is probably good with the supplements combined with what the fish had

Posted by: Ampu | May 18, 2018 5:33 PM    Report this comment

I'm not sure how CLP decides it, but just from looking at the ingredients I would say that the Ol' Roys makes more sense.

A dog needs just what other mammals need so the bone, muscle, fat, organs of any is fine. And soybean is actually a good source of soluble fiber which should help the dog live longer.

The other brand has fish and simple sugars; however, it has added amino acid supplements which can actually be better since they can balance the amino acid ratios so some aren't in excess, since too much protein can sometimes be a bad thing.

Posted by: Ampu | May 18, 2018 5:21 PM    Report this comment

I know this article is a few months old but I'd like to add a few things. First off, grateful to have found this article. I was looking through the CLP site a few weeks back for their ratings on nutritional supplements. What I found odd was the same as Nancy's discovery: the organic products (USDA approved, mind) with superior ingredients were given 1 star ratings, whilst the non-organic products with inferior ingredients had 5 stars. To say that that's peculiar is an understatement, plus as Nancy said just defies common sense.

It says on the CLP website that it's a non-profit, but I suspect money's exchanging hands in some way that allows it to keep that designation, or that someone with an agenda (against organic) is a regular donor to CLP. Just an opinion of course, but it wouldn't surprise me. Since external links are not allowed, there's another good article from 2017 titled "Why won't Clean Label Project listen to concerned pet owners?" at a site called TheRawFeedingCommunity.

Posted by: TheGL | April 26, 2018 10:13 PM    Report this comment

Hi, Readers. The Clean Label Project rebutted our post, below, and encouraged those interested to see their post on their site. That's fair. But then, six times, they tried to post a repetition of their initial post, as if to keep it appearing again and again in the comments. We felt that one rebuttal and referral was quite enough, thank you, and took down the repetitive posts.

We do encourage you to go to their site and read their blog post for yourself.

Today, on their site -- upon which NO ONE can comment -- the CLP is accusing us of censoring them on OUR site. That's funny.

Their rebuttal still appears below. We'll respond to the longer rebuttal they posted on their site soon.

For now, we'll just say that it's a shame that an organization that purports to "focus on health and transparency" isn't confident enough in its test results to publish them. Scare-mongering doesn't help anyone.

Posted by: Nancy Kerns | February 7, 2018 7:02 PM    Report this comment

My 6 yr old Yorkie is allergic to chicken and would scratch himself a lot especially at night. I discussed this with the owner of a Whole Pet Foods store. He advised me to try raw frozen Primal Pronto Dog Food, Beef Formula, $29.99 for 4lb bag. I have feed him two handfulls (once in the AM and again at night) for the past year and there is no scratching and he has gained the proper weight of 10 lbs. I supplement his diet with Grain Free Holistic Formula "Evanger's Dry Food for Dogs" Meat Lover's Medley with Rabbit (one handful daily) and for treats Dr. Becker's Beef Bites (grain free liver treats). He appears and acts very healthy and active. Does anyone have any experience with these dog foods and your comments?

Posted by: Roscoe Gamble | February 7, 2018 10:54 AM    Report this comment

When Clean Label Project measured contaminants in pet food they did a big mistake when measuring Arsenic as a whole (Total Arsenic) and did not breakdown Organic Arsenic compounds v. Inorganic Arsenic compounds. Their response to my question back in May was that they are evaluating the testing method to determine if additional breakdowns should be made. When results of our Clean Label Project Pet Food Study 2.0 came, I was disappointed.

The methods of testing wich Clean Label Project chooses to perform, leads to the conclusion that fish is generally toxic. Sea food contains more or less Organic Arsenic, its unavoidable. In their product ratings fish as ingredient is bad stuff; their tests for toxins certainly measures high levels of Total Arsenic in pet food made with fish.

There are two types of arsenic compounds in water, food, air, and soil: Organic and Inorganic (these together are referred to as total arsenic). The Inorganic forms of arsenic are the forms that have been associated with long term health effects.
FDA laboratories use methods that determine different Arsenic Species in rice products and fruit juices.
FDA Advices on eating fish concerns contaminants as Mercury, PCBs, and other pesticides, Arsenic is not particularly mentioned.

CLP is not all wrong, we should concern about levels of heavy metals in pet food. If I calculate Champion Petfood's own figures on average of measured values in Acana and Orijen products for dogs, using their feeding guide (dog 30 kg, 330 g/day: My dog then will consume ≈ 400% more lead than what is average intake (g/kg body weight) for adult persons (EU). WHO has no limits for tolerable intake; all levels of lead are harmful to the organism.

(Forgive me for my english, bad translation is to blame!)

Posted by: Vilja | February 6, 2018 5:21 AM    Report this comment

The Clean Label Project believes in looking beyond the label and evaluating products on the basis of real, scientific data. Im writing this today to set the record state and to re-affirm what weve always said here at the Clean Label Project: Less environmental and industrial contaminants in the food we feed our families (including our pets) is better than more. Contrary to the absurd litany of inaccuracies within the Whole Dog Journal post about our pet food list, Clean Label Project believes that:

1) Food that is healthy and nutritious and low in industrial and environmental contaminants and toxins should not be a novel concept
2) Consumers make more informed choices when armed with data, science, and transparency as opposed to marketing
3) That brands should be held accountable for the products they put on store shelves
4) That the very best products deserve to win in the marketplace

Contrary to the beliefs and evaluation methodology of Whole Dog Journal, when given the choice between marketing puffery and hard data, Clean Label Project will take the data every time.

When all is said and done, the readers of Whole Dog Journal have a choice: they can choose to believe that the claims brands put on their labels (the foundation of Whole Dog Journal's recommendations) is the truth, the whole truth, and nothing but the truth despite their being direct evidence to the contrary or they can embrace our call to action and demand that members of the pet food industry do better.

Clean Label Project sets the record straight on this blog post on its website and social media. Visit us by going to the Clean Label Project website and clicking on "Blog" under "Resources".

Posted by: CleanLabeProjectED | February 5, 2018 6:24 PM    Report this comment

Bottom line-there are no regulations on dry dog food. If I followed their feeding advice, I'd have 7 very overweight dogs lacking in quality nutrition. Not anymore though.

Posted by: LindaKay | February 4, 2018 12:55 AM    Report this comment

Thank you for doing the work and taking the time to write this article, Nancy. "The Clean Label Project" has been around for a couple of years now. Several nutritionists and scientists, including myself, voiced serious concerns about their methods, lack of transparency, lack of evidence-based conclusions regarding potential harm, and arbitrary assignment of "scores". When posted on their FB page, these concerns were all immediately deleted. This group also responded to legitimate questions from pet food industry professionals with derision and attacks. (So, if you have not yet received a scathing response from them, whoever they are [no one seems to know], be ready for it). Great article with clearly presented concerns. Thanks again. Linda Case

Posted by: Linda Case | February 2, 2018 8:33 AM    Report this comment

Could you provide the names of all the Scientists, Vets and Nutritionists you consulted as well as their qualifications in preparing this article? I have both a scientific and legal background and it doesnt add up.

Posted by: BlueMoray | February 1, 2018 8:08 PM    Report this comment

This is why I've been making homemade dog and cat food for 6 years. I've been a subscriber to WDJ for 8-10 years and it has been an invaluable and reliable source of information. In fact, if I read another rating of food, toys, treats, etc...I will always come here to WDJ to see what your rating is on that particular category before deciding to purchase. In addition to all the valid reasons you state in this article on why NOT to trust the information given in the CLP, most pet owners will not dig deeper to find out why certain brands are rated lower than the cheap, bargain basement varieties on the list. They will see the star ratings and use this solely as their basis for buying brand Z instead of brand A. It's sad, really. Thank you for keeping on top of things and keeping us well informed!

Posted by: SueW | February 1, 2018 6:40 PM    Report this comment

I have been a WDJ subscriber since 2005. As an earlier poster said, articles like this are why. Thank you for continued excellent reporting.

Posted by: Carolyn M | February 1, 2018 6:27 PM    Report this comment

I thought the same thing and actually contacted the company for some clarification. I was surprised, but they did email me back. This is how it was explained to me. She said that their ratings weren't about the nutritional values, but were about the contaminants. So you might have a really low quality food that got a 5 star because while it wasn't good nutrition, it didn't have any contaminants. It kinda made sense to me. If you have a food that doesn't have anything but chemicals in it, there isn't anything to be contaminated. It all just makes me crazy. That's why I make my dogs food! :-)

Posted by: Vicki P. | February 1, 2018 4:53 PM    Report this comment

Articles like this are the reason why I have been subscribing to Whole Dog Journal for probably close to 20 years now. Thank you for your critical thinking and for digging deep into issues like this on our behalf, and on behalf of our dogs.

Posted by: lmh | February 1, 2018 4:39 PM    Report this comment

Thank you so much for making the effort to investigate this. It is very hard for pet stewards to make sense of contradictory ratings and your breakdown of this is very helpful. I appreciate the transparency (and lack of commercial advertising sponsorship) which gives Whole Dog Journal credibility.

Posted by: 1wyldhaven@gmail.com | February 1, 2018 4:26 PM    Report this comment

I agree that this review of the subject is accurate. Over the years I've learned to read and analyze dog food labels. Every pet owner should learn how to do this but unfortunately, most do not and if they do, they don't understand them. I had a friend who had no idea what the specific ingredients in the food did for the health and well being of her dog, but one thing she said was if she could pronounce the ingredients, they were good.

In my view, there are very few really good dog foods.
Although ingredients are listed, there is no guarantee that the ingredients are actually in there or if they substituted a cheaper product for a stated ingredient(s) for some batches. I do believe that a home made diet with enough correct and appropriate supplements for your dog is better than processed foods i.e. some breeds are notorious for having low zinc, other breeds may be low on other minerals. However, and I understand this point, most people are too busy to spend the time buying, preparing and cooking their dog food so they need to know a good product. And some will always buy the lowest priced food at the supermarket. Indeed, processed dry foods are rendered and as well as being cooked at very high temperatures which destroys all the nutrients which may have been in the uncooked product. End of my treatise. Grin

Posted by: Holly 1 | February 1, 2018 4:16 PM    Report this comment

This is utterly shocking.

Kibbles and Bits with 5 stars? Puppy Chow with 5 stars?

I'm also disturbed by the fact they use affiliate links on the review page. They are paid when someone clicks the buy link and purchases on amazon.

As someone with an internet marketing background, this entire website seems like a ploy to get traffic from google and send them to an affiliate site to make a commission. Nothing wrong with that, but if there is a guise of science behind it, and that the reviews are unbiased, its concerning.

Great article. At iHeartDogs.com we're also interested in covering the Clean Label Project and will definitely link to your piece.

Posted by: JustinPalmer | February 1, 2018 3:56 PM    Report this comment

What a shame!! Another profit motive obviously. Our pets deserve better than this! Thanks Whole Dog for making us aware of these types of hype.

Posted by: Narrowdog | February 1, 2018 3:32 PM    Report this comment

New to Whole Dog Journal? Register for Free!

Already Registered?
Log In