COG #3: Too Subjective--The problem with the OES is that the current system relies too heavily on skewed qualitative data leading to bias, inflation, and an inaccurate depiction of an Airmen's performance.
How do we fix this?
How do we fix this?
Teams that bring completed COAs across the line (validated by ST) will receive 1 point
Submit your ideas here. Ideas can be anything however fully flushed out ideas will contain:
COA Title:
- What is the banner that your team will rally behind
Objectives:
- What is your COA trying to accomplish? Use action verbs.
Desired Effects:
- State how this COA will positively impact the Officer Evaluation System.
Milestones:
- Red: What we need to do right now: 1/3 of the COA has been accomplished
- Yellow: The next major step: 1/3 and 2/3 of COA have been accomplished
- Green: How do we sustain measure success: >2/3 of COA accomplished
Please label your posts to avoid confusion
i.e. The Title of the COA is ... The Objective of this COA is ...
Each team MUST identify at least 3 implications for executing their COA.
  |
| Gold Gear is the COA, the Gears coming off are Implications |
- Achieve 5 validated effects from each implication all those who contributed will receive an extra point.
Username: bet
Password: auciiteam
I will take a stab at the objective here of a COA. We should develop a system that produces concrete quantifiable results. The system must be very easy to use and almost impossible to abuse. The system also has to integrate with the previous system until members who are up for promotion on the old system have been phased out.
ReplyDeleteTitle: A Likert Scale OPR
Delete-- could be used to objectively rate an officer based on purely subjective qualities.
Desired Effect:Must be given to both Commanders and Subordinates to fill out on Ratee. If Commanders and Subordinates are within 5 pts at end of assessment than the Commanders assessment score is what ends up on the OPR. If Commander and Subordinates are outside 5 pts than the average score is taken.
DeleteWhat if you do not have any subordinates like many in the aviation field, especially as a young officer?
DeleteIt is a great idea, but I think subordinates would be very difficult to include for the reason Joe mentioned, but I think it would also be a very difficult sell to our senior leaders.
DeleteI agree its a tough sell but I believe that you could apply "filters" to certain career fields. In a specific aviation field pilots who do not supervise anyone would only be evaluated by their peers and commanders. The assessment technique Matisek discussed above could be applied. This would ensure that all afsc's get a fair shake. If you are a cop or a maintenance officer expect to be filtered by supervisor, peer, and subordinate... in my opinion it takes a completely subjective process and puts actual value to the numbers. You rate high because your peers and or subordinates believe you are a good leader... you rate low because your Peers and Commanders do not think you are a good leader. You rate average because there is a split... You can quickly see how your leadership translates to your people. What do you think?
DeleteHere are my thoughts, and please strike the down/improve on them.
ReplyDeleteObjective: Create a more objective OPR than we have today.
Desire Effect: As Thomas said, we need to have concrete quantifiable results.
Milestone 1: Ditch the subjective lines that we have today, but keep the subjective strats. Part two we would steal from the Navy, which includes 9 characteristics that a great officer should have (among them character and leadership). 1 is awful, 5 is amazing, rate each officer in each category. The default answer is 3, but anything other than a 3 needs a supporting explanation. Not bullets but a couple sentences explaining what the officer did that was so amazing. Average those 9 scores out and that is quantifier number 2 (Strat was #1). Finally, we steal from the Marine Corps the comparative assessment. You guys should check it out, but pretty much you are rated on how good you are from
average to amazing, but when the rater signs the document it shows their entire profile. Immediately you get another hard quantifier that shows how good you really are. The nice thing about this is if the commander rates everyone amazing he cant help out his best people, so he is almost forced to be honest.
Please tear apart, and/or make better. Imagine how easy board selections would be when the computers actually spit out the best quantified officers!
This would take a while as Thomas said because initially officers would have no profile and therefore the comparative assessment would be useless. After about 2 years there should be sufficient data to start using the comparative assessment.
DeleteI like the idea. If you wanted a comparative analysis we would have to implement a static closeout. I believe that this would be doable due to all the chaff that you would eliminate in using plain english vs trying to write a perfect bullet. The goodness here is that you are turning subjective data into an objective score that can be compared against others.
DeleteI think this is an interesting beginning of a COA. What are the milestones??
DeleteHow do we begin to implement this? What is the logical next step. Finally, how do we measure success?
I agree with Thomas for the most part. However, I feel like the current system is so far from repair that we should look at a complete overhaul regardless. I agree that for currently promoting officers or those that are going before a board in the time when a change is implemented should be on the old system. As soon as that process is complete though, they should be transferred to the new one.
ReplyDeleteHere is my idea for a COA:
Title: The Narrative OPR
Objective: To eliminate the bullet system completely from the OES. The bullet system itself is what is causing the skewed numbers and poorly representative statements of airman across the board. So much valuable time is wasted between the ratee and supervisor about how to fit a bullet into all of the "white space" on an OPR.
Desired Effects: The current form should be changed completely to facilitate a narrative style approach. The goal is to provide a more accurate representation of the primary and additional duties of the airman being rated on. In the current system, an airman can be a rock star at their primary job as well as their additional duty, but be considered mediocre since there is not any volunteer or community service bullets, for example. By creating a narrative about an airman, it develops a short story of how they're actually doing. The supervisor can speak in plain English to get their point across rather than inputting skewed bullets to fill space. If the ratee is a rock star, the supervisor should make it known in the narrative and back it up with their primary job accomplishments. If the ratee is definitely not a rock star, it should be apparent in the narrative. In order for this to work, it will have to come down to the supervisor being honest and truthful when writing these whether it's good or bad. That should be the case now, but we all know that doesn't happen.
I feel I'm starting to get into the weeds a bit, so I'll stop for now. I hope this spurs conversation and questions. Thanks and I look forward to reading any feedback.
I agree that the bullets need to go and plain english should be used instead, however, without any length requirements, how long should a "good narrative" be. What words should a "good narrative" contain... the problem will still remain at the board to decipher the good from the bad narratives when it comes time to promote. Currently the board averages a min and a half per record as they quickly scan bullets for the "right information". So the question still remains how do we ensure that narratives just turn into long bullets over time?
Delete@Joe, I agree that the time spent on formulating bullets is time that could be spent better evaluating the ratee. However, I'm struggling to see how going to a narrative vs. bullet format will negate the subjectivity of the OES. I would argue that it has the potential to make the OPRs more subjective because the rater will not have to spend as much time thinking about the words they are writing as they can just write their train of thought. You mentioned challenging the raters to being "honest and truthful when writing", but if that is the only true step to making the OES more objective, couldn't we just do that to the current system? I'm interested to hear your thoughts!
DeletePerhaps, could this better support the overly complex COG?
DeleteI think with the current system all the rater does is open the thesaurus and try to pry some cool
Deletesounding words out to fill white space. I honestly think that more time is spent doing that than actually focusing on what the ratee has done. The rater is not really putting any thought into it. By making it a narrative it negates that process and focuses on what they've actually done. My vision wouldn't be to keep it in single sentence format either, but rather a paragraph if that makes sense. For example, instead of lengthening a bullet into a sentence, the narrative form would be a full paragraph, or story like I wrote above. I would expect that the rater would be truthful and honest in what they write otherwise it's all for not. Some feedback I received about this is how long it would take to go through each OPR. I totally understand that, but if we're actually concerned about the progression of our outstanding officers I think the time should be taken to ensure a thorough review. Likewise, for the not so stellar officers, we should be identifying them and either helping them out or worst case hold them back.
Are bullets/narratives even necessary at all? If an OPR were simply a series criteria/attributes deemed to be the most important aspects of officership at a particular rank level and of a particular AFSC, and the rater assigns numerical values on each of these attributes, why would you also need a narrative/bullets? Save narrative and bullet writing for awards and keep the OPR simple with numbers that can be compared against other people.
DeleteBullets/narratives are necessary to accurate reflect what has been accomplished throughout the year. However, there should be a standard AFI that dictates which abbreviations are approved for the OPR. In that way everyone from any base will be able to decode the bullets. Additionally, it would help if there were a block after each one that could rate the strength of the bullet in respect to peers in that career field. Example: Does this bullet align with standard actions among those in the career field or is this an extra certification that goes above standards to help the individual stand apart from peers? Then if the bullet is used in the PRF graders would understand the significance of the actions.
DeleteI don't think a rating system 1-5 is good. That's the whole reason why they changed the EPR system. It will be abused. They will still say everyone is awesome. I don't think raters will be held accountable. I like what Joe H says at the bottom about the actual use English and know your people approach. If you were to use a 1-5 metric, I would say you need justification on why they are above a 3 just as much as why they are below. This can't be bullets either. Has to be hard and true facts on job performance.
ReplyDeleteDoug - That is exactly what the Navy does. Anything other than a 3 needs justification, and not in bullet format. I agree though that using a 1-5 scale is somewhat of a turnoff just because of what we saw with the EPRs. If we used that sort of scale we would really have to sell what its different.
ReplyDeleteJoe - Can you elaborate a little more about the narrative OPR? Specifically, how would that transition into promotion boards? Without quantitative data I feel like this could be tough.
It would require a total mindset change. PRFs I also feel should change to narrative rather than pulling bullets since with this system they would be non existent. I feel that quantitative data was born to spruce up bullets. Do we want a bunch of numbers that will remain skewed, or an actual description of the performance of our guys/gals? In my experience, the people that can tweak a bullet to show an amazing numerical value will get recognized as a top performer, while the people that actually show that they are vital to the unit through their primary duties and interpersonal interactions are forgotten. So after a long answer to your question, the BLUF would be remove the emphasis on big numbers and focus on how the individual actually is doing. People can still say "provided vital support to $7 million SAR assets" for example. The idea won't be the numbers but more so the leadership for officers and actual impact this person has on the unit. Hope that helps. Would like to have an in person conversation if possible.
ReplyDeleteI think if we are successful in creating a feedback and OPRs that are more comprehensive, honest, and able to trend over time rather than snapshots in time than we can ditch the PRF saving billions of man hours and the polar bears as well... sorry its getting late and I am delirious... great conversation everyone very happy to be apart of the discussion.
DeleteCOA Title: Create a Narrative OPR rather than bullet OPR
ReplyDeleteObjective: To create a more objective and less inflated approach to getting the information across.
Implementation:
- Change the form from bullet format to a narrative format. This will require normal language used and the ability to accurately write what an individual accomplished during the respective year.
- Focus on the whole person within the workplace. Do not include outside volunteer or extracurricular activities. This will create standardization across AFSC’s so those who have a high tempo versus those with low tempo having more time to do outside things.
- Separate Forms for CGOs and FGOs that focus on the tactical level of expertise (CGOs) and the managerial level of expertise (FGOs).
- Two sections of narrative writing: one for the ratee to write on their own personal accomplishments and those of their team; and one for the rater to discuss progress and leadership abilities from supervisor perspective based on set standards and performance requirements per AFSC.
- Accountability for what information can be depicted on the OPRs. You need to have direct influence on an item/project in order to claim credit. “I did, I led” or “I created, I managed.” There needs to be accountability for inaccurate reporting.
Desired Effect: A narrative form OPR would allow individuals to write out accomplishments in normal language without the requirement to build a one line bullet that is inflated or detriment of space required to be filled. The structure of a bullet (Action, Impact, Result) causes the inflation due to wanting to create the most attractive and eye catching bullets. We hope adopting a narrative format will allow true impacts to shine through, while eliminating as much subjectivity and inflation as possible.
Contributors: Capt Brandon Carter, Capt Joe Gotwald
I mentioned it in a reply further above, but I'll add mymain point here too. If we move to an OPR that does a better job at quantitatively measuring people, do we really even need bullets/narrative? How about we just save narrative and bullet writing for awards and keep the OPR simple with numbers that can be compared against other people.
DeleteCOA TITLE: MODIFY CURRENT STRATIFICATION TO REFLECT A CAREER VS ANNUAL PERSPECTIVE
ReplyDeleteOBJECTIVES:
- ELIMINATE CREATIVE STRATS (E.G. #1 COPILOT VS #1 CGO)
- ELIMINATE STRATS FOR LIEUTENANTS
- DELAY CAPTAIN STRATS UNTIL 6 YEAR MARK
- CHANGE RANKING ORDER TO PERCENTAGE BASED STRATIFICATION
DESIRED EFFECTS:
Our COA is to modify the current stratification implementation in the evaluation. The intent of this COA is to emphasize career progression and officer growth stratification versus an annual comparison of Airmen. Commanders have a great amount of discretion to create criteria in which they use to rank people. Many use spreadsheets to track accomplishments, education, and other items. This style of box checking focuses people to try and check as many of those items off as soon as possible in order to get the higher ranking order. Delaying the first stratification until later in their career lets young officers focus on becoming experts in their primary duty and develop and apply their leadership. Additionally, it reduces work load on commanders and staffs to track these spreadsheets every year.
The next modification is to change the stratification from a ranking ordered system to percentage of based stratification. Similar to how the EPR system limits the number of 5’s that can be handed out based on the proportion of each rank that is being evaluated, or how many “Definitely promote” commanders have to give for promotions, we can limit the number of officers who get a stratification based on the amount who are eligible and amount in their command.
The third modification is to establish specific stratification that commanders would be allowed to use. This would eliminate “creative” stratifications that are made up. By limiting the types of stratifications to ones that are established and vetted, this would put stronger emphasis on stratifications given.
Collaboration Credit: Capt. Brandon Carter, Capt. Alicia Pitts, Capt. Harold (Joe) Gotwald
For your milestones, accomplishing your first and third modification could be accomplished via a simple AFI change. Currently, AFI 36-2406 already restricts some of the strats that are allowed/not allowed. By just adding more detail to the AFI about your first and third mod. If this were to get picked up by A1, that should be feasible. Implementing your second mod would require some education of SRs in addition to a simple AFI change. I think you can measure success by getting the AFI changes accomplished and having creative strats no longer allowed.
ReplyDeleteCOA Title:
ReplyDelete- Scoping a portion of the OPR in order to assist the OES to become more objective
Objectives:
1) Start a baseline for promotion boards/DTs to observe what positions a person is best suited for (currently you get only a few characters in your push statement)
2) Ensure that the PRF process is not bogged down with additional narratives that may make the process unable to be completed in the current time line
3) Give the ability for the rater/senior rater to identify MULTIPLE positions that would suit the ratee versus one in the OPR and PRF form
Desired Effects:
- Establish a baseline of positions that an officer is best suited for as identified by their rater/senior rater within the OPR and PRF
- Establish objective section that can be referenced in order to have a board/DT be able to quickly look at potential for leadership positions (ref AF911 "Future Roles")
Milestones:
- Red: Add the section of "Future Roles" (up to 3) into the OPR and PRF. Within the PRF it would be annotated as a trend analysis item. This way it could be referenced quickly by promotion board members. Training at the unit would be minimal since we are doing this for EPRs (AF911)
- Yellow: Require this portion of the OPR and PRF to be used as a factor for how to come to an order of merit. Just because someone is not being pushed to be a Sq/DO, doesn't mean they aren't cut out for leading, it would be annotated that they could do another leadership job. A 14N example (since this is my career field) would be 1) Sq/DO; 2) Wing/IN; 3) Dep Chief of ISR Division. Again all examples of leadership positions, but some people are better suited to tactical/operational/strategic areas.
- Green: Sustainment would be to keep this section in the OPR and PRF. Success would be measured by studying placement of personnel before and after implementation to see if the records are really being used to place leaders. Next, a climate survey for the promotion panels to get the qualitative feedback of "Is this really working or is it helping identify multiple avenues that a leader can be vectored towards". Lastly, the climate surveys at each unit should be cross-checked against where the board placed the leader to measure if the trends that were identified in the OPR and PRF form are accurate projections for the member.
Title should read:
Delete- Scoping a portion of the OPR and PRF in order to assist the OES to become more objective
Title: Standardized Leadership Aptitude Assessment
ReplyDeleteObjective: Administer a written leadership aptitude exam to all members who are to be considered for particular promotion board. The leadership aptitude score would be combined with an OPR quantitative measurement, thereby providing demonstrated performance and future leadership aptitude.
Desired Effects: Measure leadership aptitude in a standardized, AFSC-neutral way, which can then also be combined with demonstrated work performance, as measured in OPRs.
DeleteThis comment has been removed by the author.
DeleteThis comment has been removed by the author.
DeleteBackground: Researchers and some executive management consulting firms have developed what they believe is a valid metric for measuring executive leadership aptitude – essentially, an executive leadership IQ. See:
ReplyDelete(1) https://www.spencerstuart.com/what-we-do/executive-assessment-services/executive-intelligence
(2) https://hbr.org/2005/11/hiring-for-smarts
(3) Executive Intelligence, by Justin Menkes: http://lib.nu.edu.sa/uploads/m1/29.pdf
The premise of executive/leadership IQ is similar to that of the original IQ exams that are were developed and administered to kids to determine aptitude for academic success. As implemented by one consulting firm (Spencer-Stuart), executive intelligence is measured by conducting a standardized 90-minute interview. However, it seems reasonable to assume that such an interview could be adapted to a written format that could be administered en masse (for promotion boards).
Interestingly, even if such a leadership IQ-specific exam could not be adapted from the proprietary interview process used by firms like Spencer-Stuart, researchers have determined that even a standard academic-type IQ exam is a valuable predictor of leadership success:
Delete“In 1998, two of the most respected researchers in assessment methodologies, Professors Frank Schmidt of the University of Iowa and John Hunter of Michigan State University, published in the American Psychological Association’s Psychological Bulletin a landmark study analyzing over eighty-five years of research findings. They compared the predictive validity of all major assessment methodologies and declared that IQ is a measure that should be included when hiring virtually any employee. In a 2004 follow-up study, Schmidt and Hunter combined the results of 515 independent studies involving over 100,000 employees. They declared that cognitive ability tests predict occupational performance better than any other ability, trait, or disposition, and better than job experience. Further, research has proven that as job complexity increases, so does the predictive validity of IQ. This finding has been borne out repeatedly. For instance, in a large-scale study in 1984 conducted by Professor John Hunter and Ronda Hunter, they combined the results of 425 independent validity studies involving over 32,000 employees. They found that for less complex jobs such as farmhand or laborer, IQ explained about 5 percent of the variance in performance. But for upper-management positions, IQ explained 34 percent. Similar results were reported again in a 2003 study published in the Journal of Applied Psychology, combining the results of 138 independent studies involving over 19,000 employees….IQ testing is at least as effective as competency interviews (the most common assessment methodology used today for hiring and promotion) as a means to predict managerial success. What’s more, IQ tests are approximately ten times as powerful as personality tests. Such results demonstrate the enormous relevance that IQ has to managerial work performance, despite the fact that IQ tests are geared to academic subjects” (Executive Intelligence, page 146-147).