How Not to Take a Survey, Part II

            Previously, in “How Not to Take a Survey,” I discussed McKinsey’s opaque approach to its survey on employer reactions to the PPACA.  Now, it seems, McKinsey has bowed to pressure from the media and the Democratic Party, and has released some of its survey data.  Those who follow this link can also download PDF’s of the survey and the resulting data (I recommend reading the questionnaire at the very least).

            Again, I’m not sure how I feel about the political dimensions of this imbroglio.  The business press has every right to ask McKinsey questions; that’s part of the role that we assign to the media in a democratic society.  But should McKinsey be made to answer to the demands of a political party?  That thought makes me more than a little nervous.

             Whatever the case, McKinsey certainly seems to feel the need to act like a child caught stealing cookies.  The beginning of the firm’s latest press release reads (all emphases are mine):

 The survey was not intended as a predictive economic analysis of the impact of the Affordable Care Act. Rather, it captured the attitudes of employers and provided an understanding of the factors that could influence decision making related to employee health benefits.

 As such, our survey results are not comparable to the healthcare research and analysis conducted by others such as the Congressional Budget Office, RAND and the Urban Institute. Each of those studies employed economic modeling, not opinion surveys, and focused on the impact of healthcare reform on individuals, not employer attitudes.

Comparing the McKinsey survey to economic estimates, such as the CBO’s, is comparing apples to oranges. While the McKinsey Quarterly article about the survey cited CBO estimates, any comparison is not apt. We understand how the language in the article could lead the reader to think the research was a prediction, but it is not. 

As the survey results reported, 30 percent of respondents who said their  companies offered employer sponsored health insurance said they would “definitely” or “probably” drop coverage in the years following 2014, the year the Affordable Care Act takes full effect. (Nine percent said “definitely,” and 21 percent said “probably.”)

I suppose we can be forgiven for thinking that McKinsey was claiming any predictive power for its initial study when the latter’s title was “How US health care reform will affect employee benefits” (my emphasis), and when the article claimed that “The Congressional Budget Office has estimated that only about 7 percent of employees currently covered by employer-sponsored insurance (ESI) will have to switch to subsidized-exchange policies in 2014. However, our early-2011 survey of more than 1,300 employers across industries, geographies, and employer sizes, as well as other proprietary research, found that reform will provoke a much greater response.”  So I guess the last step in how not to conduct a survey is to obfuscate furiously when you finally get called out on your study’s methodological shortcomings and on your own aversion to sharing data the way that you should.

        Anyway, without further ado, let’s talk about the survey itself.  First, McKinsey still has not released the response rate, nor have they tried to give any sense of how the sample was chosen.  I also can’t get a sense of how the survey was conducted by reading McKinsey’s press releases and the publicly available data.  This already makes me skeptical of the data, though I do have to admit that the actual sample chosen doesn’t look too bad when considering whether it’s representative of employer geographies, industries, etc.  And McKinsey claims that the survey data was weighted in accordance with U.S. Census data of firms by industry, which would certainly be the logical thing to do.  However, there are more than a few things in this survey that still don’t smell quite right.  In its “apology” McKinsey claims:

To qualify as a respondent, the individuals were asked a number of screener questions to ascertain that they played roles in choosing which benefits their companies provide to employees either as primary decision makers or having influence in the decision-making process.

…

All respondents indicated they had either primary decision making authority (51%) over medical benefits or influence over the decision-making process regarding employee benefits at their company.

This is a distortion at best.  The first real question of the survey asks respondents if they play any role in determining aspects of the employee benefit package (including medical benefits) with instructions that the survey terminate if the answer is “do not play a role in this decision,” and that it proceed to question 2 otherwise.  And yet, we find that question 2 has the same number of respondents as question 1 (1329 for both), despite the fact that 12.3% of respondents to question 1 answered that they played no role whatsoever in their company’s decisions regarding medical benefits.  There’s no indication that these individuals were actually eliminated from the sample the way that the survey instructions say that they should be.  Is this some kind of intentional distortion or just outright sloppiness?  I’m still scratching my head about that one, and there’s more to come:

The top five reported respondent occupations were: owner, head of human resources, head of procurement, CEO/president, vice president of  compensation or benefits director/manager.

Actually, the most common occupational category is “other,” (at 25.8%) and given that the survey already lists owner, department head, CEO, head of HR, comptroller, procurement director, VP of compensation and benefits, CFO, and advisor/analyst, I’m not sure what the other 25.8% do, and kind of wish that I did.  There’s clearly something a little fishy going on here.

            The actual survey isn’t all as bad as I expected, though I think that some of the questions are indeed leading (particularly those numbered 43-45).  Consider question 44 of the survey:

With healthcare reform, some employers may be interested in offering health insurance to their higherwage employees, but not to lower-wage employees (who can purchase government subsidized coverage on healthcare exchanges). Here are some additional facts about this option:

Healthcare reform prohibits employers from discriminating in favor of higher wage employees in their benefits offering.  However, there are other workforce strategies your company could pursue to provide different benefits to your higher vs. lower wage employees.

For the next set of questions, assume exchanges are an easy, affordable way for individuals to obtain health insurance. How likely would your company be to consider pursuing each of the following strategies?

1) Restructuring your company into two companies, one comprised of higher wage employees who would continue to be offered health insurance, and one comprised of lower-wage employees who would not be offered coverage. (If the “lower-wage company” has more than 50 employees, you would pay a penalty of $2000 per employee in this company, after the first 30 employees.)

2) Shifting your lower-wage workforce toward more part-time workers (work less than an average of 30 hours per week each month). You would not have to offer part-time employees health insurance and would not have to pay any penalties. You would continue offering health insurance to all full-time workers.

The possible answers to this question are:

Definitely would

Probably would

May or may not

Probably would not

Definitely would not

I don’t know

So, as I said, there are some slightly leading questions in the survey.  ("Did you know that you can completely reorganize your firm in order to skirt American labor and healthcare law?  Here are some options of how to do so.  Would you like to consider some of them?")

            We’re still left asking why McKinsey would release such a survey in the first place.  Since McKinsey has nothing to gain by alienating either political party (and in past elections the firm has certainly been more than willing to give money to the DNC and to Democratic candidates), this doesn’t seem to be politically motivated.  Rather, I’m starting to suspect that Rick Ungar may have the answer:

So, why would the firm rely on potentially substandard data – or at least data gained by a study conducted below the standards typically set for efforts of this nature – to embarrass the president and ‘call out’ his landmark health care reform legislation? It wasn’t about the President and it certainly was not about politics.  It was about the money.  Ask yourself this question – What does McKinsey & Company do for a living?  They consult and advise large corporations on how to solve business problems and properly innovate and organize their future.  If a major shift in- or the termination of – a corporations’ long-standing employee health care benefits program is not a major business issue requiring the greatest of care in the planning and execution, then I don’t know what is.  Who will these companies turn to in order to get advice on the benefits of unraveling their complicated and extensive employee benefit plans and for assistance in executing any new plans for the same?  McKinsey & Company.  What we have in the McKinsey survey is little more than a company ‘pitch piece’ designed to drive customers to their door.  There are, after all, endless hours to be billed for conducting reviews of corporate employee benefit plans to better assess the impact of the new health care law and even more to be banked should McKinsey conclude that a revision and revamp of a benefits plan is in order as McKinsey will be performing that revision and revamp-for a fee.

            Some final words are in order about the way that the media treats statistics.  The Washington Post’s Greg Sargent asks (correctly in my view):  “I wonder how many of the news orgs that covered this study as a prediction will now cover the concession that it wasn’t intended to be a prediction.”  As I contemplate all this, I can only think back on John Stewart’s recent remark to Chris Wallace that “I think [the main stream media’s] bias is towards sensationalism and laziness. I wouldn't say it’s towards a liberal agenda. It's light fluff so it's absolutely within the wheelhouse.”  Unfortunately, this goes even deeper than just the media’s purported love of yellow journalism; I don’t expect a well-credentialed economist (who arguably should know better) to issue a retraction of his claims that the McKinsey study proved that the PPACA would end employer sponsored insurance as we know it and bury the US economy under a mountain of debt to boot.  Somehow, once a flawed statistic enters the public sphere, we’re stuck with it until the end of time.