Why Engagement Surveys Flop...and What to Use Instead

Three words that will pull a deep sigh out of even the most dedicated employees:

“Employee Engagement Survey”

Perhaps the only more reviled annual tradition within a company is the performance appraisal. So why does something designed to improve the situation of our employees tend to drive unease in them?

Simply writing this now - as someone who has considered themselves highly engaged throughout their career - I’m thinking back to disdaining the practice myself. Even when I was the HR leader driving the survey, I recall the mental (and sometimes literal) eye rolls I would get from fellow executives about the time and energy that would get devoted to something that was hard for them and their teams to really feel the value of in their day-to-day work.

Engagement surveys can be a valuable practice. One could cite plenty of success stories where surveys have driven positive change. (Simply surveying people on a behavior, like whether you plan to buy tickets to a basketball game, or volunteer at a charity - can significantly increase the instance of the behavior itself among the survey participants.) Yet it’s unfortunate that even when designed and executed very well, an engagement survey can too often have the opposite of its intended effect.

The survey method is an excellent way to get information quickly and cheaply. But the combination of these characteristics coupled with those it lacks creates a problematic situation,

A survey is by nature reliant on assumptions of what is important from the individual writing the survey. And it may often generate more questions than answers - when its responses contain confusing information without enough context. Few survey respondents possess the unprompted chutzpah (or self-awareness) to outline the root causes of their responses in the context of organizational systems. Engagement surveys just don’t give you information that is actionable into what your employees are experiencing.

Let’s explore some of these dimensions.

1. Why can something that takes (or should take) only twenty minutes to complete feel like such an imposition?
2. Why would employees see such a survey as a negative instead of a positive?
3. And what, even when executed perfectly, are the downsides of utilizing a survey?
4. What is the alternative to an engagement survey?

Why does an engagement survey feel like an imposition?

Surveys are inherently impersonal. Especially in situations where there is critical feedback to be given - the absence of a face-to-face conversation with the people with the power to make a difference can often feel like those leaders are phoning it in.

The implications of the name “employee engagement survey” itself implies that there is some issue with the level of engagement. That implication is off-putting and can drive divisiveness when people feel that they’re being placed in one of several camps around how engaged they may or may not be.

It’s ironic that engaged employees sometimes feel like they have more important things to do than devote their full attention to a survey. And disengaged employees don’t feel connected enough to the organization to bother devoting their full attention to a survey.

When it comes down to it though, isn’t the intent behind a survey positive? It’s an attempt to find out what our employees need, and how we might make things better for them. So why would employees view it negatively?

Why would employees see the survey as a negative?

Anonymity is a routine concern with survey respondents. Individuals are 75% more likely to participate in a survey if they feel confident their anonymity is guaranteed.

People are savvy enough to recognize that there are digital identifiers in their day-to-day interactions with technology. The proposition that these have suddenly disappeared for the sake of the survey is a hard pill to swallow for many.

There are ways around this. We’ve seen companies sticking to paper surveys, ensuring the entire process is administered by external consultants, or even earnest HR professionals showing employees the dashboard where they view the results. But even a single anecdote spread through the grapevine regarding abuse of any kind of employee input makes it hard to shake the thought that there’s some kind of loophole.

The transparency of the results is another concern we see cited frequently in our work. A company that conducts a survey but does not release the results to the employee population can create more harm for themselves than good. Even more commonly, the results are released, and there is a clear lack of action on the information that’s been obtained. This leaves employee engagement worse off than it began - creating the feeling that the employer doesn’t value the input of their employees.

Managers and more engaged peers may grow frustrated with verbatim comments that they feel lack context or display susceptibility to bias. Even numerical results might create frustration in leaders who believed that energy and resources were devoted to topics that perform poorly. This is frequently the case because of the fatal flaw in the nature of engagement surveys that we will unpack below: they’re based on the assumptions of the survey designer. They’re not born out of employee experience.

What are the downsides of utilizing a survey?

The Achilles heel of the survey method is the nearly unavoidable injection of bias. Even if the surveyor masterfully crafts a set of questions without any bias - response rates and respondent mindsets may introduce staggering levels of bias into a survey’s results.

Examples of the most commonly seen biases present in engagement surveys are recency, availability, agreement, selection, and attributional bias.

Recency Bias: It is extraordinarily difficult for respondents to prevent themselves from allowing recent experiences to weigh more heavily in their answers than past ones. This skews validity toward only one point in time.

Availability Bias: Similar to recency, attribution of what immediately comes to mind as having a greater impact on responses can be dependent on several factors in addition to the timing.

Agreement Bias: One of the best-documented types of bias is the tendency of a respondent to agree with questions as they progress through the survey. Lengthy surveys become demanding on the attention spans of respondents. Even when question order is randomized, they can quickly lose their reliability.

Selection or Non-Response Bias: The representativeness of responses in a survey format is a gamble. This is where we see the “squeaky wheel” phenomenon. Even in instances where the organization places goals around the number of responses required from business units, this “forcing” of responses tends to have its own ill effects.

Attribution Bias: Individuals regularly shift their responses more positively toward a way that they believe reflects individually on them or their group in a desire to respond in a way that makes them feel good about themselves. This is especially apparent in questions about the effectiveness of a team or function, or the quality of work produced.

Is there an alternative to an engagement survey?

The most common prompt to dive deeper generally occurs in a company faced with troubling survey results. Sometimes those results are concerning because they’re clearly negative on one or more dimensions. And sometimes the results cause worry because they’ve raised more questions than answers.

Whether a company chooses to skip the survey for a more reliable and actionable method - or employs that method following the survey’s results - the employee experience assessment stands out in a number of ways.

An employee experience assessment is the development of an objective and in-depth understanding of a deceptively simple question: “What is it like to be an employee here?”

At the surface, the most obvious difference between the survey and an employee experience assessment (or EEX) is that it utilizes a one-on-one interview format to gather data.

Leveraging EEX also differs in another deceptively simple way from engagement surveys. Indeed, despite the pitfalls we’ve covered - it’s not uncommon to hear the protest, “A really good survey can capture the same information!” But that’s often not the case.

The difference lies in a paradigm shift. A survey is based on the viewpoint of the surveyor - whether that’s HR or leadership. EEX is based on the viewpoint of employees. That change in vantage point is so unassuming it’s frequently overlooked. When one is basing their understanding of what it’s really like within an organization on leadership’s point of view instead of the employee's point of view - it’s common to miss the forest for the trees.

An extreme example of this problem occurred at the United Parcel Service (UPS). The same year they received an impressively positive overall result on their employee engagement survey, UPS workers went out on strike. The one thing that was the central issue - of great importance to the employees - around what turned out to be a very costly strike for UPS, had not even been asked about in the engagement survey. So the bitterness harbored by employees on the topic went undiscovered and the issue continued to fester.

The ability to flag, evaluate, examine, dig deep, and seek specific solutions for a wide range of issues in a large organization is dependent on facetime with employees. Their anecdotes can often reveal a defect, broken or unwieldy process, or a problem with leadership’s assumptions about how planned systems truly function in the field.

It is a leader’s role to maintain skepticism when what they think they know from their data, and what their people are telling them do not match. It’s often a sign of hidden problems.

How do you conduct an employee experience assessment?

1. Identify your stakeholders. 
2. Develop your sample group. 
3. Communicate the process.
4. Conduct your interviews.
5. Compile your data.
6. Pull out the patterns.
7. Identify the relationships.

Stakeholder Analysis

Accomplishing company goals always requires change. Change often stirs emotions and encounters resistance. The larger the impact of the changes, the more important emotions and resistance are to plan for and manage.

The output of a stakeholder analysis is a representation of where impacted people stand relative to the topic areas. It is documentation of the roles and relationships of stakeholders and their stance on the topics. Through the in-depth interviews conducted in this assessment of employee experience, we also allow the employees themselves to define the landscape of topics for us.

Developing a Sample Group: Organizational Cross Section

A sample group is developed to be representative of an employee population. Whether it’s a company-wide assessment, or a targeted look into a particular division, site, or department. This is done because it’s rarely a good use of time or resources to interview every single person in a given group. Especially in large divisions or company-wide assessments, it does not improve the quality of the information.

This sample is developed utilizing the same standard statistics, quality assurance, and survey methodology any academic would employ in selecting a subset of individuals from within the statistical population to estimate the characteristics of the whole population. It’s important to consider the different planes within the organization: department, job function, shift, location, tenure within the organization, and demographic characteristics.

This is done to avoid information-gathering biases which give too much credence to outlying opinions and experiences and to directly combat the difficult “squeaky wheel” phenomenon present in surveys.

The most frequent question we get is “How many people should we interview?” This number is different for every organization. It’s highly dependent on the organizational structure and the degree of homogeneity within the employee population. But we find the average engagement tends to produce a sample size somewhere above one in six employees.

Communicating the Process

Apart from the fact that anything a company does that will affect its people should get ample communication - this process is different enough from what the average employee is used to that it’s really important.

It’s important to directly explain the EEX assessment in an effort to inform people about:

1. What the heck you’re doing,
2. Why you’re doing it, and 
3. What to expect in the process.

Perhaps the biggest question you’ll need to get in front of, given the nature of the representative sample/cross-section, is “Why did you interview Steve but you didn’t interview me?” If you get to that topic ahead of that question getting asked, most people will immediately understand when you explain, “There are five hundred of you. Interviewing every single person would take forever.” 

But aside from sample size, it’s good for people to understand what’s happening, and most importantly be put at ease by your interviewer directly. Clearly and plainly stating intentions, relaxing, and connecting with people through your demeanor is important groundwork for the interview process. We use a very brief, face-to-face, huddle-style communication practice for this stage of the game.

Beginning with Leadership

The senior leaders and a portion of those in mid-tier leadership roles should be interviewed prior to the front-line employees. This beginning step of the process may seem counterintuitive in a process that’s all about the employee - but is critical for actionability. 

Developing an objective understanding of the employee experience is rarely actionable without placing it in the context of the organization’s strategic needs.

Those needs are several-fold:

• Where the organization is headed (long-term goals)
• What the strategy is to achieve the long-term goals
• What threats and opportunities will influence that strategy in the future
• What the most immediate priorities are for the organization
• What issues is the organization wrestling with today?

This context informs the interviewer’s ability to identify the most relevant and important information and probe the information presented to them by the interviewees for potential solutions. One might argue that this context can be set in the premise of the initial engagement. But it’s rare that the entire leadership team, or middle management, is in lockstep on these items. 

Conducting the Interviews

In-depth-interviews, or IDI’s for short, are what we use at Wayforward to execute the assessment phase of our client engagements. The style of interview, psychodynamic interviewing, is a powerful tool that rests heavily on the expertise, experience, and internal compass of the interviewer. 

Psychodynamic interviewing is rooted in traditional psychoanalysis. While it is not clinical therapy, this method of interviewing utilizes a similar set of guidelines to help the individual and the organization better understand and articulate the factors influencing their current state.

In psychodynamic interviewing, the interviewer gains insight into the work (and often personal) life and present-day problems of the interviewee - sometimes simultaneously with the interviewee themselves. They evaluate patterns that people have developed over time, by reviewing certain factors such as thoughts, emotions, experiences, and beliefs. 

Recognizing recurring patterns within a representative sample of interviewees allows the organization to act on those patterns.

A trusting relationship is central to psychodynamic interviewing. Feeling secure in the interview and holding a belief in the interviewer’s genuine intent to help is key to empowering the interviewee to be forthcoming. Maintaining the confidentiality of what the interviewees share is paramount to this entire process. Without it, none of this works.

Drawbacks of EEX Assessments

The interview process is more time-consuming than surveys. Depending on the level of sophistication and support from engagement survey software, the EEX assessment process is often more expensive as well. 

They are also highly dependent on the individual practitioner conducting the interviews. An inexperienced or insincere interviewer can scuttle the data collection. It’s also of great importance that the practitioner(s) be well-versed in mining raw data. In anything but a small business, there is a heavy data analysis piece to ensure that the insights are solid and actionable. Narrative presentation of the data in an engaging but objective fashion is another critical piece to ensure that the leadership team is able to absorb what is often a significant amount of information on complex, interconnected topics.

Other Points of Value

While the interview’s main and most apparent value is generating strong data from which to make decisions on improvement initiatives and protect the core competencies of the organization - the interviews themselves generate their own value. 

The interviews are a demonstration of leadership’s genuine interest in the perspective and knowledge of employees. The opportunity to voice one’s thoughts and contribute to the larger direction of the organization is a major factor in satisfaction and a morale boost in and of itself. During the interviews, the interviewer also has the opportunity to foster faith in the leadership team’s authentic interest in what their employees have to say, and the leadership’s genuine intent to act on the information. 

It’s important to note that doing so should be done very sparingly, and based on giving context and factual information. It should never be used to debate or convince - as this jeopardizes the objectivity of the interview, as well as the trust necessary to continue to conduct further interviews.

Though less certain than the direct value highlighted above, the interviews themselves do sometimes generate organic problem-solving activity among employees. The nature of the topic discussed, and the interviewer’s focus on root cause lines of questioning often plants the seeds of initiative in individual interviewees. 

Indeed, it is frequent that interviewees reach out later to thank the interviewer for sparking their desire to act on a problem and share the good news of its solution.

Feeling overwhelmed by this or want an expert opinion? Schedule a consult with Wayforward.