Grab Your Copy of Everybody Writes Here >> Now a WSJ Bestseller!

The Science Behind Quality Content: A New Study

Only a third of the world’s largest companies have quality site content based on human factors like style and clarity and tone of voice (in addition to basics like grammar and spelling.)

The study from linguistic analysis company Acrolinx is the first of its kind to try to quantify something that’s hard to pinpoint: What’s the value of writing to a business?

(It’s the first as far as I know. And I’m happy to be wrong. So let me know in the comments.)

At the very least, it underscores the need to pay attention to not just what you say, but how you say it.

Some details:

A mere 31% of brands worldwide earned a passing grade for the effectiveness of their website content—a score of 72 or higher on a scale of 0-100—according to Acrolinx’s analysis of marketing, corporate, technical, and customer support website content.

I like that the study looked at more than the marketing and corporate communications pages that we might typically think of as “content.” Because everything the light touches is content.

Ann handley lion king

Using its proprietary linguistic analytics engine (say that 10xs fast!), the company scored the content of more than 20 million sentences and 160 million words making up 150,000 Web pages from 340 global brands with more than $250 million in annual revenue. (Got all those details? There might be a quiz.) Brands like Gucci, Exxon Mobil, and Harley-Davidson, among others.

Acrolinx tackled style, tone of voice, and clarity—not just the “easy” stuff like grammar, usage, and spelling.

The grammar and usage analysis was straightforward: Acrolinx looked at subject/verb agreement and use of pronouns, and it also didn’t not look at double negatives. (Ha.)

Then, it evaluated style, based on 62 separate rules and writing practices (the kind you find in The Chicago Manual of Style or Yahoo! Style Guide, and, of course, Everybody Writes).

It judged clarity (how easy is this piece of content to read and understand?) by evaluating things like sentence length, structural complexity, and word choice.

Based on its proprietary algorithm, Acrolinx gave each company a “content impact score” using a 100-point scale to give each company—a measure of how effective the writing is. A score of 72 or higher signifies content that’s effective.

“Most companies have not yet reached that level of content sophistication,” Acrolinx concluded.

Fully 69% of brands failed the content quality test, scoring below 72. The scores of the 340 brands studied ranged from 55 to 85.

acrolinx-global-content-quality-scores-2015

Among other findings of the analysis:

  • Retail businesses exceeded the benchmark for content quality, on average scoring 73.2, followed by B2B tech with an average of 71.2; telecoms lagged with a 66.2 average.
  • From a global perspective, Germany and America tied, scoring the highest for content quality: 70.2 each, on average.

An interesting footnote: Acrolinx suggested a connection between Alexa website rankings and the effectiveness of site-content writing. Those with higher content impact scores had, on average, a 22% improvement in their Alexa rank over the past six months, while the companies with the lowest content impact scores had, on average, a 9% decrease.

Acrolinx fully admits that its data doesn’t necessarily show correlation between Alexa score and its “content impact score.” And, of course, others have pointed out that the Alexa rankings themselves can be a flawed measure of a site’s success.

But anyway, the apparent link is nonetheless an interesting footnote to consider.

See the full Global Content Impact Index here.

Total Annarchy

Join at least a handful of your peers and all of Ann's relatives. Get new posts by email.

OR Subscribe via RSS Reader

50 Responses to The Science Behind Quality Content: A New Study

  1. Very cool. So who has the best content? Who has the worst? I’d love to see names – and names of the content leaders behind the best.

    I did something like this for Mark Cuban a few years ago when he was investing in Little Bird: we discovered which sports writers were most influential among their peers and then we analyzed the Flesch-Kincaid writing levels of their content. For what it’s worth, the top 4 sports writers online, measured by influence, had declining sophistication in their content as you went up the list. Meaning, the very most influential were the least sophisticated writers (like Peter King) – but the most sophisticated writers (Darren Rovell and Bill Simmons) were still pretty darned influential among other sports writers.

    Fun stuff!

    • Ann Handley says:

      Interesting, Marshall. Insightful thought re the Little Bird study, too. I’d like to see something like that applied to content marketers. :)

      Maybe Acrolinx would share the ranking. That would be interesting.. agreed. In the report itself, they call out Kohl’s, Caterpillar, and National Australia Bank as being among the top-shelf communicators.

      Thanks for chiming in here!

  2. It seems like what they’re measuring is the readability of their content, not its impact.

    To judge the impact of content you need to have a clear understanding of the goals the content is trying to achieve, both for the business and the customer. If the content achieves those goals, it doesn’t matter whether it follows the Chicago Manual of Style or not.

    You also have to take into consideration the audience for each piece of content. Marketing content is going to have a much difference audience (and reading level) than technical content, so to judge them both on the same scale seems problematic.

    Also wondering how they “judge” tone of voice, since each company may have a different tone of voice. Are they judging whether the tone of voice is appropriate for their audience and brand? Can tone of voice be good/bad?

    I definitely agree that most of these companies probably have serious issues with their content (god knows I’ve seen it firsthand), I’m just skeptical that their “proprietary” algorithm can quantify something as complex and nuanced as the impact of content – especially since the study comes from a company that sells content services to large brands.

    • Ann Handley says:

      Thanks, Andrew. Readability seems part of it — but Acrolinx’s software also looked at specific use of language and structure to identify a
      variety of style issues, according to the report. Not sure how, exactly, but hopefully Acrolinx will chime in here and shed some light.

      I agree “quality” is a tricky thing to measure, especially since it depends heavily on intent, goal, audience. But at the same time, I wish more content marketers paid more attention to creative, engaging content. Better content, not more content, in other words.

      Big thanks for stopping by, Andrew!

      • I wholeheartedly agree with you about the goal of creating better content, not more content – and I think you’re a great Ambassador for that message.

        And while I’m always skeptical of a one-size-fits all algorithmic approach to evaluating anything, if a study like this gets people talking about improving their content and thinking about ways to improve readability and clarity, I’m all for it.

    • We aren’t just measuring readability, we really are measuring dimensions of the tone of voice of the content.

      Firstly, just to clarify, we are not “a company that sells content services to large brands”, we are technology company – and we have been using our AI-based linguistic analytics engine for many years to provide these kind of insights to our customers.

      You are absolutely right on two really important issues…

      Firstly, of course, adherence to the Chicago Manual of Style is not going to tell you whether your content is going to have any impact with your target audience. Good writing – grammar, spelling and style – are however kind of table stakes for getting read. So in addition to the standard stuff, we are also measuring how “conversational” content is, since all kinds of brands are moving in this direction to better engage with their customers.

      Secondly, you are right that a tone of voice needs to be distinctive, and we can see that B2C companies are typically more informal in their language than B2B (although there are exceptions). We are not measuring how close a particular organization is to their own stated goals – as you say, we can only do that once they tell us what those goals are.

      This is our first edition of this research, but we are going to be updating it regularly and capturing more detailed insights about trends in customer-facing language as we go along.

      Watch this space!

      • I’m sorry if I came off as dismissive of your company, that wasn’t my intention. It sounds like you’re doing really interesting things and provide a really valuable service and I’d love to see how this type of research progresses.

        The point I was trying to make (and which you acknowledged) is that without knowing the goals of the content it’s nearly impossible to judge its quality. You can create engaging, creative and tonally consistent content, but if it doesn’t help drive business goals than it’s not having an impact.

        Getting content strategists and marketers to do that extra work to really identify their client’s goals instead of just jumping into content production is huge challenge.

        For the second edition of your research I’d love to see some segmentation between types of companies and the different metrics that they use to judge content success. I know a lot of that is probably proprietary information, but it seems like you work with a lot of big firs, so maybe you have inside info :)

  3. Kennedy Grey says:

    If analytics can be compiled by an app, why not the copy itself? Oh…whoops, it is every time Google compiles a search engine result. Which really makes “content” something that we should leave to the algorhythm rather than try writing “visceral” copy based on non-visceral cold machine logic “now with added human feel approximation!”
    And won’t it be a great day for “writing”? When *FINALLY*, one measurement algorhythm examines the analytics and “style” of another writing algorhythm for its effectiveness. All that’s next is the robot love story, the robot Oprah (RoPrah, now on digital newstands near you!) and when-boy-robot-meets-girl-robot movies (Keanu Reeves isn’t busy and makes a great robot without trying!).
    Good thing that by then, all the “writers” who aren’t actually writing their own blogs, books, web copy, and speeches can rest easy that their iPhone 8 can do it for them.
    And then we can finally have a world filled with perfectly aligned, analytically-symetrical verbiage that robots worldwide will triumphantly hail as the “robot smash-hit of the summer of 2015 for you and your family driving Toyota Camry, and drinking Heineken 1.5 times per month, married but still got-it male dudes!”
    And i can finally rest. Oh shoot, they don’t give me much rest at my job serving coffee to roofers at 6am at Dunkin Donuts since all the writer jobs now pay .05 cents a word. Awesome article, btw. My app read it and loved it.

  4. Mark Evertz says:

    Ann,
    This is great information. Thanks for sharing it. This study, Marshall’s work (Thanks Marshall) and work from companies like Atomic Reach (www.atomicreach.com) seem to be sounding the alarm on Keep It Simple, Stupid! for content. That both makes complete sense and kinda bums me out a little for some reason I’m eager to learn more about components of the algorithm and what their version of simple and good is. Thanks again.
    Mark (@MarkAEvertz)

    • Ann Handley says:

      Thanks for chiming in, Mark. Curious why it bums you out? I think any platform that helps companies value strong, clear writing is a win for writers. But I’d love your take.

  5. Jeff Krimmel says:

    I really like the use of data here. Andrew is of course correct in his comment about the nuances of measuring “impact”.

    But if these correlations can spur deliberate thinking around (1) specifying goals for content creation and (2) leveraging readability to accomplish those goals, we all win.

    Really fun stuff. Thanks for sharing.

    • Ann Handley says:

      Exactly. I think the key is “deliberate thinking.” I don’t think organizations value words nearly as much as they should. So I’m happy we’re even starting to have this conversation!

  6. Mark Evertz says:

    Hey Ann bummed in a way similar to Kennedy w/o the pretty prose. (Back to the screenplay, Kennedy!). You’ll all remember this from English & journalism classes, but before content was called content it was our role as communicators to advance the language and spur active learning through our writing. Not spoon feed people Wall-E style so they become lazy regurgitates of other people’s ideas. My old j-school professors use to actually demand a few thesaurus words in stories so people would be forced to look them up.
    Maybe I’m showing my age in the 140-character world we live in , but … Long story medium that’s the nature of my discontent. That and I always bristle at people or tools that tell me to behave. Eager to watch the space & science evolve though. Thanks for that. ^me

    • Mark Evertz says:

      Said the writer with typos. The iPhone is an enemy^me

      • Ann Handley says:

        You’re awesome for swinging by again! But I guess I see things differently. I don’t think platforms like Acrolinx devalue writers or the creative process as much as put new emphasis on the importance of good, clear, customer-centric writing.

        Acrolinx doesn’t write for you — nothing will ever replace the creative writer. But tools like it helps business see why good writing matters. And that’s what interests me.

        Lots of marketers are talking about content. Fewer are focusing on quality.

        • Mark Evertz says:

          On that we completely agree. Anything that helps people communicate clearly and be understood by an audience is a big win for all of us. I’m just torn a little on the rigidness a tool can sometimes demand and the unilateral decision making of what is good or bad based on a mashup of style guides or app designer preference. Penalizing any personal style is my concern. So … I’m in … just dipping my toes in the pool first. Cheers, ME

    • Allow me to join you in the discontent department….I’m an old print newshound who’s made a living writing online for the past eight years. And I really dislike the word “content.” Can’t we think about it as writing, vs. cubes of print that fill spaces to order?

      In fact, I’m giving a talk at the New Media Expo on April 13. The title of my presentation: “Stop Calling It ‘Content’!”

      And I just launched an online writing course/blog coaching business, in which I also take out after the notion of “content.” Let’s love the language a little more, please? Otherwise everything will be doubleplusungood online in the very near future… 😉

      That said, I found this article very interesting. Just a little depressing, is all.

      • Ann Handley says:

        “Content” is a horribly bland word — I agree. But not sure what else to call stuff we create. It’s not all text, or writing per se. But I’d love to hear your suggestion for a new descriptor.

        Wish I was going to NMX to hear your talk, Donna!

      • Len Diamond says:

        The depressing thing is that the study can pass for science.
        I’m more interested in your “Content” comment, though. “Content” reduces writing to a commodity, and when you’re selling a commodity you compete on price. That’s what has helped reduce the price (and as a result, quality) of writing to poverty levels. I started a discussion once on a writers’ group site under the headline “Does Anyone Besides Me Have Trouble With the Word ‘Content’?” I guess I got my answer: only one writer did.
        It’s good to hear it said on a bigger platform than any I have. It’s an uphill fight, but I hope you pursue it. I’d gladly chip in my two cents if it would help.

  7. Thank goodness for companies like Acrolinx because they will likely help the Executive Professionals stay on their toes. Setting agendas for improvement is always important. For creatives like me who are guided by their own stars, creating is sometimes less planned or at least driven by trends, our own professional and personal experience. Personally, I do not let numbers affect what I choose to create or guide my direction in content creation in any way. When someone starts to let numbers control their creative direction then their own agenda can be lost and they fall victim to the control of those controlling the numbers and become less genuine. I never want to be that person.

  8. Pingback: Google update: Mobile-friendliness and SEO | INKsights

  9. Pingback: The Week in Social Analytics #148 - TweetReach Blog at TweetReach Blog

  10. Josh St. Aubin says:

    This is really interesting Ann. I’ve worked with a lot of clients that tend to overcomplicate their content as a way of attempting to elevate the intelligence or sophistication of their brand. Like some how the value of their product or service is directly related to the complexity of their message. The complex it is, the better it must be. I tend to have the opposite point of view – the more you can simplify and explain something that’s complex, the smarter and more sophisticated you really are. You can’t out think your audience.

  11. Pingback: Weekly Wrap-Up: What You Need To Ask To Make Your Pitch A Success | Come Recommended

  12. Pingback: The Science behind Quality Content, Jill Marler, April 10, 2015 | Evangel University Social Media Class Blog

  13. Pingback: Friday Catch-Up: The Imaginarium of the World Wide Web

  14. AJ Kohn says:

    I’m always surprised that more companies and sites don’t concentrate on readability. If you want engagement and if you want your content to be remembered (which to me is the ultimate goal) then the first step is getting it read.

    A 30 line block of text is going to chase most readers away. Studies show that most of us skim web content. You’re not writing the next Jonathan Franzen novel folks. Writing for the web is as different as grant writing is from haiku.

    So this study seems like a step in the right direction. I’d love to know if other factors such as the golden ratio of typography, presence of a font hierarchy or use of images are taken into consideration.

    Because it’s not what you say, it’s how you present it that really matters.

  15. Bibhu Panigrahi says:

    Interesting info, and discussion.

    Tools like Acolinx can help us focus on what matters, quality content, by taking care of superficial aspects of language that take a lot of our time and attention. For example, Acrolinx can check if my writing adheres to our corporate style guide, and that saves me considerable time and effort. I can put the time it saves me to better use, that is, creating impactful content.

    I agree Acrolinx cannot help us write better, but it can help us focus better on writing better.

  16. Pingback: Digital Marketing News: April 2015 | Email Marketing Tips

  17. Pingback: Digital Marketing News: April 2015 | Email Marketing Tips

  18. Pingback: Digital Marketing News: April 2015 | Unlimited Email

  19. Pingback: New study finds quality of brands' content sorely lacking - The Social Olive

  20. Pingback: New study finds quality of brands’ content sorely lackingCloud0086 Latest Tech News | Cloud0086 Latest Tech News

  21. Pingback: Digital Marketing News: April 2015 | MarketingTumbler.com

  22. Pingback: New study finds quality of brands’ content sorely lacking | Body Language

  23. Pingback: 8 Writing Tools I Use Every Day

  24. Emma Overgaard says:

    I’ve used Acrolinx IQ for several years, for varying types of text, UI, web and User guides. If, like me, you have to write texts that are translatable, highly consistent and easy to understand, then it is really a support. But if you’re after a more relaxed native or colloquial style, using plenty of idioms, slang and fragments, it’s not the tool for you. You’d be stuck in feeding the machine allowed exeptions, because it will flag your creative language as something that you should consider revising.

  25. JWermann says:

    I use Acrolinx for technical writing. I love how it keeps the text concise and clean. If I write more creatively for marketing, I would have a different set of creative writing rules in Acrolinx. Acrolinx can have separate rules for each writing purpose and audience. But in one location all the terminology nuances for consistency are kept. From a writer’s point of view, Acrolinx is a well developed tool. This tool makes the printed message clear and consistent between multiple writers and products in one company.

  26. Pingback: 2 Million Blog Posts Are Written Every Day, Here's How You Can Stand Out | Articulous.net

  27. Pingback: Top Content Marketing Blogs: 50 Must-Read Blogs on Content Marketing, Social Media, and the Intersect of All Things Digital - Docurated

  28. Pingback: Quality Content Matters. Here’s the Latest Science to Back Up this Claim. | Bill Ringle

  29. When selecting a vacuum on your rug, get a merchandise that includes a great reputation. Trying to reduce cleaning items will only run you in the end using the funds that you may have to spend about brand new carpets. Analyze the item in your house to determine when it will do the job.

  30. It is surprising that many brands were not able to pass the 72 figure. In my opinion the content has to be engaging too. This can help websites to let visitors stay on the website for a long time. Also, there are ways where content can be used to bring back the visitor if they have subscribed to our newsletter, sending them thank you notes or festive greetings.

  31. Lynette says:

    Since public relations requires familiarity with a wide variety of matters, a broad
    education is the very best preparation.

  32. mfsw.pl says:

    Governments have a central position in creating a healthy food atmosphere that allows
    folks to undertake and keep healthy dietary practices.

  33. Pingback: 5 Keys to Developing a Strong Tone of Voice in your Content Marketing

  34. Hansofetch says:

    Great post! Quality content plays a major role for a blog.
    hansoftech.com

Leave a Reply

Your email address will not be published. Required fields are marked *