The strongest objection to 360-degree feedback is that the ratings say more about the people who give them than about the person being rated. The objection has empirical support from a study by Steven Scullen, Michael Mount and Maynard Goff, who analysed ratings from two large samples of managers (2,350 and 2,142 people), each of whom was rated by seven raters. In those samples the raters' individual tendencies explained 62 and 53 % of the variance in scores, while the managers' own performance explained only 21 and 25 %.[1] The practical conclusion is that a single score in a 360 report tells you very little, and the useful information sits in agreement between independent raters, in consistent differences between rater groups and in the spread of answers within a group.
In the hypothetical case this article works through, Olivia, who leads an analytics team, was rated on five competencies by herself, her manager, four peers from neighbouring teams and six direct reports. The average rating from others across the whole report came to 3.8 out of 5, exactly the average Olivia gave herself. Her manager read that match as "her self-assessment is realistic, everything's fine", even though the same table holds a blind spot, an underrated strength and a split within her team. The case and its people are invented, and the numbers were chosen to show combinations that turn up in 360 reports again and again.
Why a 360 average has to be read alongside the spread
James Conway and Allen Huffcutt brought together, in a meta-analysis, studies in which the same employee was rated by supervisors, peers, subordinates and themselves. Average agreement within a group turned out to be moderate: 0.50 for supervisors, 0.37 for peers and 0.30 for subordinates. Agreement between groups was weaker still, at around 0.22 between subordinates and supervisors or peers and from 0.14 to 0.22 between self-ratings and others' ratings. The highest figure, 0.34, was between supervisors and peers, and the authors concluded from these data that different groups see a person's work from different vantage points.[2]
The first implication for reading a report is that differences between groups are a normal property of the method and don't signal an error on their own, because direct reports see Olivia as their manager while peers see her as a project partner. The second concerns small differences in averages, which, given this level of agreement, are easily explained by exactly who ended up among the raters when they stay within half a point on a five-point scale. That's why the case treats a gap of 0.5 points or more as meaningful, provided the group has at least three people, and this is a working rule for reading the report, since research doesn't set a specific threshold.
| Competency | Self | Manager | Peers (4) | Direct reports (6) | Others' average |
|---|---|---|---|---|---|
| Setting tasks | 5 | 4 | 3.75 | 3.17 (3 to 4) | 3.45 |
| Giving feedback | 4 | 4 | 4.25 | 3.00 (three 4s, three 2s) | 3.55 |
| Delegating | 2 | 4 | 4.25 | 4.33 (4 to 5) | 4.27 |
| Working with other teams | 4 | 3 | 3.25 | not rated | 3.20 |
| Developing people | 4 | 4 | 3.75 | 4.17 (4 to 5) | 4.00 |
In formeasure, a 360 review runs as one cycle for the employee, their manager, peers and direct reports, in anonymous or open mode, and the individual development report is put together automatically. A table like this one is therefore ready as soon as the cycle closes, and you can see how the cycle works on the 360 feedback page in formeasure.
Self-rating versus others' ratings: four combinations
Comparing the self-rating with the average from others sorts competencies into four groups, and each of them means something different for the development plan.
| Self-rating | Others' ratings | How to read it | Example from the case |
|---|---|---|---|
| High | High | Confirmed strength | Developing people: 4 and 4.00 |
| High | Noticeably lower | Blind spot | Setting tasks: 5 and 3.45 |
| Low | Noticeably higher | Hidden strength | Delegating: 2 and 4.27 |
| Low | Low | Acknowledged development area | Doesn't appear in Olivia's report |
James Smither, Manuel London and Richard Reilly, in their review of research on whether 360 feedback works, report how people in these groups behave. After upward feedback, managers who had overrated themselves lowered their self-ratings while those who had underrated themselves raised theirs, and in two studies the people who initially overrated themselves went on to improve more than others. In a study by Atwater and Brett, leaders who received low ratings and agreed with them were less motivated to change than those who received low ratings but rated themselves higher.[3] In practice this means that a blind spot, however uncomfortable the conversation about it, gives the clearest material for change. An acknowledged development area calls for more care, because the person already knows about the problem without any report and can easily lose faith that it can be fixed.
In Olivia's case the blind spot is setting tasks: she gives herself a 5, while all ten ratings from peers and direct reports fall between 3 and 4, so the signal agrees across two independent groups. Delegating works the other way round, since Olivia rates herself at 2 and others rate her at 4.27, and it makes sense to put this skill into the development plan as a strength she can build the other changes on.
Spread within a group: when a team sees its manager differently
The "Giving feedback" row shows why a gap against the self-rating isn't enough on its own. Direct reports average 3.0, the gap between Olivia's self-rating and the average from all others (−0.45) falls short of the half-point threshold, and on a quick read this competency is easy to skip. Behind the 3.0, however, sit three 4s and three 2s, which looks like a situation where half the team gets regular feedback and finds it useful while the other half hardly gets any. Six identical 3s would produce the same average, yet that picture would point to a different problem altogether, namely feedback that is uniformly middling for everyone.
A split within a group usually means the manager works differently with different people, for instance talking more often to those on her own projects or to more experienced staff. The report doesn't show who gave the 2s, and trying to work it out destroys trust in the process, so it's better to discuss with the whole team how feedback works. The distribution of answers should only be shown for groups with enough people to protect anonymity: the Public Service Commission of Canada requires a minimum of three responses in the peer, direct report and other categories.[4] In anonymous formeasure campaigns, results for groups below the minimum number of responses are combined to reduce the risk of identifying the author.
The "Working with other teams" row shows another detail that's easy to overlook. Direct reports didn't rate this competency because they don't see how Olivia works with neighbouring teams, and the "not rated" note in their column tells you this behaviour is invisible to that group, which is useful to know when planning development. The 0.8-point gap here rests on five raters whose answers consistently fall between 3 and 4, so it deserves attention, although it's a weaker signal than the one about setting tasks.
In what order to read a 360 report
Reading the report starts with checking who rated: how many people answered in each group, which groups were merged because of the anonymity threshold, and on which competencies some raters chose "can't assess". After that, look at the ratings from others without the self-rating, mark the highest and lowest competencies, and only then compare them with how the person rated themselves, so that the self-rating doesn't set the frame for the whole review.
The next step is the spread within groups, because a consistent average and a split opinion call for different actions. Comments are best read last and used to explain the numbers, noting themes that several people repeat, while a single harsh comment says more about the relationship with one particular colleague. The reading ends with a list of hypotheses about behaviour, which then get tested in a conversation with the manager, an HR business partner or the team, since the report doesn't draw those conclusions by itself.
What research says about the effect of 360 feedback
Smither, London and Reilly combined 24 longitudinal studies in which 360 feedback was run at least twice in a meta-analysis, and found that improvement in ratings from direct reports, peers and supervisors was almost always positive but very small. The authors write plainly that practitioners shouldn't expect large, widespread improvement, and they give several reasons. Raters don't distinguish well between the competencies in a questionnaire, and factor analyses of the ratings usually find one to three factors instead of the dozen listed. On top of that, an average across all items is a crude measure of change when the person has been working on two or three specific goals.[3]
The same review lists the conditions under which change is more likely: the feedback shows that change is needed, the person sees that need, reacts to the report without rejecting it, believes change is feasible, sets specific goals and takes action. According to Kluger and DeNisi's meta-analysis, discouraging feedback that threatens the recipient's self-esteem reduces the effectiveness of feedback. Having a facilitator help interpret the report wasn't significantly related to effect size in Smither's meta-analysis.[3]
The most practical finding comes from a five-year study by Alan Walker and James Smither, in which 252 managers received ratings from their direct reports every year. Managers who met with their team to discuss the results improved more than others, and they improved more in years when they discussed the previous year's feedback than in years when they didn't. There was no noticeable improvement between the first and second years, and it appeared only between the second and third years and again between the third and fourth.[3][5]
How to build a development plan from 360 results
The research suggests that a development plan should be short, tied to observable behaviour and come with a way of checking progress. A convenient structure includes one consistent blind spot, one strength to build on and one open question to clarify with the team, and for Olivia that plan looks like this.
| Focus | What the report showed | What changes at work | How to check |
|---|---|---|---|
| Setting tasks | 5 from Olivia and 3.45 from others, all ten ratings from peers and direct reports between 3 and 4 | Every task for the team states the definition of done, the deadline and the person who signs off the result | After three months, peers and direct reports rate the change on a scale from "got worse" to "got better" |
| Giving feedback | Three 4s and three 2s from direct reports | Regular one-to-ones with all six direct reports on the same schedule | After two months, a short pulse question to the team, where the spread of answers should narrow |
| Delegating | 2 from Olivia and 4.27 from others | One of the analysts takes over a quarterly project end to end, from scoping to presenting the result | A repeat 360 after a year using the same questions |
The check in the first row comes from a practice that Smither and colleagues describe. A few months after the report, raters are asked how much the person's behaviour has changed on the specific goals they set, on a scale running from "less effective" through "no change" to "more effective".[3] That question is more precise than a repeat overall score, because change on two items barely moves the average across the whole questionnaire. For a quick check between full cycles, short employee pulse surveys work well, and the general rules for building the questionnaire, choosing raters and protecting anonymity are covered in our overview of 360-degree feedback.
Following Walker and Smither's study, Olivia should tell her team what she took from the report and what she plans to change, without revealing anyone's answers or discussing individual ratings. That conversation shows people their answers were used and makes the plan public at the same time, and the repeat review is best planned for about a year later, since the same study suggests that changes in ratings take time to show.
Limits of this approach
The large share of rater tendencies in the scores means it's not sound to compare absolute ratings of different employees who were rated by different people, and a 360 report serves a development conversation about one person far better than it serves a ranking. For the same reason, 360 results shouldn't be tied directly to bonuses or promotions: raters become more guarded and the process loses trust, as the overview article explains in more detail.
The four combinations of self-rating and others' ratings are best used as a guide for the development conversation. Research on self-other agreement measures it in different ways and reaches inconsistent conclusions as a result.[6] The half-point threshold and the minimum of three raters are working rules for this review, and they should be reconsidered for a scale of a different length or a questionnaire with a different structure. Olivia's case is invented and its numbers were chosen for illustration, and the studies behind this review were carried out mostly with managers, so some conclusions should be applied with care to specialists who don't manage anyone.
Reviewing 360 results in formeasure
In formeasure, a 360 cycle combines the self-rating with ratings from the manager, peers and direct reports, lets you set up the competency framework your company uses and choose anonymous or open mode. The individual development report is put together automatically when the cycle closes, and a repeat cycle with the same questions a year later shows whether ratings have moved on the chosen focus areas.
If your company has its own way of working through 360 reports, message us on the formeasure LinkedIn page. We're collecting HR teams' practices for a separate review, participants will get it first, and their approaches will be credited with their consent.
Frequently asked questions about 360 feedback results
How do you read 360-degree feedback results?
Start by checking how many people answered in each group, then look at others' ratings without the self-rating, compare them with the self-rating, examine the spread within groups and finally read the comments as an explanation of the numbers. The outcome is a set of hypotheses about behaviour to discuss with the manager or the team.
What gap between self-rating and others' ratings counts as meaningful?
Research doesn't set a single threshold, so this article treats a gap of 0.5 points or more on a five-point scale as meaningful when the group has at least three raters. Ratings within groups agree only moderately, and small differences are often explained by who the raters were.
What is a blind spot in 360 feedback?
It's a competency a person rates noticeably higher in themselves than others do. A blind spot is especially useful for the development plan when the low ratings come consistently from different groups, for example from both peers and direct reports.
What should you do if ratings within one group differ sharply?
A split usually means the person works differently with different people. There's no need to figure out who gave the low rating; it's better to discuss with the team how the work is organised overall and to show the distribution of answers only for groups large enough to protect anonymity.
How many goals should a development plan include after a 360 review?
Two or three focus areas work well: one consistent blind spot, one strength and one question to clarify with the team. The average across the whole questionnaire barely moves when someone works on a few items, so progress is best checked on the chosen goals.
Should you discuss 360 results with your team?
In Walker and Smither's five-year study, managers who discussed the results with their direct reports improved their ratings more than others. The discussion should cover conclusions and plans without revealing what specific people said.
How often should a 360 review be repeated?
A full cycle is usually repeated after about a year, because changes in ratings take time to show. Between cycles, specific changes are easy to check with a short pulse survey or a question about how much behaviour has changed on the chosen goals.
Can you compare 360 results across employees?
Comparing the absolute scores of people rated by different raters isn't sound, because more than half the variance in ratings comes from the raters themselves. A 360 report is better used for a development conversation about one specific person.
Sources
- Scullen S. E., Mount M. K., Goff M. — Understanding the latent structure of job performance ratings // Journal of Applied Psychology. 2000. Vol. 85, No. 6. P. 956–970. https://doi.org/10.1037/0021-9010.85.6.956
- Conway J. M., Huffcutt A. I. — Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings // Human Performance. 1997. Vol. 10, No. 4. P. 331–360. https://doi.org/10.1207/s15327043hup1004_2
- Smither J. W., London M., Reilly R. R. — Does performance improve following multisource feedback? A theoretical model, meta-analysis, and review of empirical findings // Personnel Psychology. 2005. Vol. 58, No. 1. P. 33–66. https://doi.org/10.1111/j.1744-6570.2005.514_1.x
- Public Service Commission of Canada — 360° Feedback Services: Frequently Asked Questions. https://www.canada.ca/en/public-service-commission/services/staffing-assessment-tools-resources/human-resources-specialists-hiring-managers/human-resources-toolbox/personnel-psychology-centre/assessment-counselling-services/360-feedback-instruments-services/360-feedback-services-frequently-asked-questions.html
- Walker A. G., Smither J. W. — A five-year study of upward feedback: What managers do with their results matters // Personnel Psychology. 1999. Vol. 52, No. 2. P. 393–423. https://doi.org/10.1111/j.1744-6570.1999.tb00166.x
- Fleenor J. W., Smither J. W., Atwater L. E., Braddy P. W., Sturm R. E. — Self–other rating agreement in leadership: A review // The Leadership Quarterly. 2010. Vol. 21, No. 6. P. 1005–1034. https://doi.org/10.1016/j.leaqua.2010.10.006