AI is increasingly becoming part of our lives, personally and professionally, whether we like it or not. In addition to the social and climate impacts of its growing prevalence, we also should be considering what AI will do to professional development, hiring and the way we evaluate competence.
One of those questions involves a fairly well known human dynamic called the Dunning-Kruger effect. As a quick refresher, the Dunning-Kruger effect describes the tendency for people to overestimate their competence, particularly when they are novices or lack experience in a subject. Part of what makes the effect interesting is that the knowledge needed to be good at something is often the knowledge needed to recognize that you aren’t very good at it.
AI complicates that in a way I don’t think we’re talking about enough.
Someone with reasonable facility using ChatGPT or Claude can now produce work that would have been difficult for them to produce four years ago. They can write a coherent memo on a subject they barely understand or produce an explanation that sounds more sophisticated than anything they could have written unaided. That doesn’t necessarily mean the work is bad. Some of it will be very good. The problem is that the person reading it may not know the difference.
We used to have certain signals that helped us identify expertise. They were never perfect, but knowledge of the terminology, the ability to explain a complicated issue or the ability to write a coherent document about a subject told us something. AI weakens those signals because it can reproduce many of them on demand.
Corporate America has always had people who could talk convincingly about things they didn’t understand. We didn’t need artificial intelligence to invent that. What AI has done is give them a better way to sound smarter.
Professionals will have to work differently
Most professionals can open an AI tool and produce something. The more important question is whether they determine if what they produced is wrong, AI slop or junk.
A recent study found that 82% of people agreed with answers produced by AI. Interestingly, longer explanations could make people more confident in an answer without making the answer more accurate.
The machine isn’t overestimating its competence, we are. We’re having trouble calibrating our trust in what it produces and sometimes we’re doing that while also overestimating our own ability to judge the result. That’s the intersection of AI and the Dunning-Kruger Effect. Yikes.
That may explain why Chad in Sales just sent you a four paragraph email about the marketing campaign he thinks the company should run. It sounds somewhat informed. It uses the right terminology. Chad spent twelve minutes with Gemini and is now prepared to brief the CMO. Chad may be right. He may also have absolutely no idea what he is talking about.
The professional skill that matters is changing. Getting AI to produce an answer is becoming relatively easy. Knowing whether the answer deserves to survive contact with the rest of the organization is harder.
This is one of the reasons I think AI literacy is going to matter professionally without requiring everyone to become a technologist. Employers are already asking for it. In 2025, 2.5% of jobs mentioned at least one AI skill, and the increase is not confined to technology roles. Lightcast found a 9X increase in generative AI skills appearing in non-tech jobs between 2022 and 2024.
For our Academy business, that is obviously great. Eventually more professionals will decide they need to develop AI skills. But I often wonder, do employers know what those skills actually look like?
Employers will have to hire differently
The demand for AI capability is growing faster than many employers’ ability to evaluate it. One study found that 52% of employers have some method for evaluating AI capability. Another found that a meaningful portion of hiring managers conduct no skills evaluation at all. Even companies that do assess candidates have a new problem: the candidates have access to AI too. So asking whether someone “knows AI” is not going to get us very far.
Knowing how to use ChatGPT is not the same as knowing how to use ChatGPT well in an actual job. Knowing some elaborate prompting terminology proves even less. A candidate can memorize the vocabulary of prompt engineering in an afternoon. That doesn’t tell you whether they are any good with AI.
I ask candidates to show me where AI failed them. What did they notice? Why was the answer weak? What did they change?
Someone who has used AI seriously has stories about this. They also tend to have complaints. Lots of complaints. Spend enough time with these tools and eventually a new model release ruins something that worked perfectly last Monday or the AI develops some linguistic tic that makes you want to throw your laptop into traffic. That tells me more than hearing someone recite which models they have tried.
At exactly the moment employers are beginning to demand AI proficiency, AI is also making it easier for candidates to create the appearance of proficiency. Hiring managers that rely too heavily on sexy resumes or people parroting jargon are going to have trouble with that.
Managers will have to manage differently
The same problem doesn’t disappear once someone is hired. Managers increasingly need to evaluate work that was produced with AI. That makes AI literacy a management issue, even for managers who have no interest in becoming particularly sophisticated users themselves.
We talk frequently at HIA about preserving human judgment. I realize that can sound vague, so this is what I mean by it. Imagine an employee hands a manager a market analysis. It is well written, organized and plausible. Five years ago, the quality of the document itself would have provided at least some evidence about the employee’s understanding of the subject. Today, that’s no longer true…
Does the analysis reflect how the company really sells? Are the assumptions reasonable? Has the employee checked the underlying facts? Does the recommendation make sense once you move beyond the perfect language used to describe it?
A manager who has hands-on experience with AI is much better positioned to ask those questions because they have seen the technology fail. They know that an answer can be beautifully written and still miss something obvious. They have also learned that asking AI to critique its own work can improve things considerably, while occasionally producing an even more elaborate version of the original nonsense. That experience creates a better BS detector.
Without it, managers can end up listening for the wrong signals. A candidate or employee starts talking about agents, recursive prompting or some other currently fashionable term and everyone assumes expertise is present. Six months later they discover the organization’s resident AI guru received most of their education from a TikTok influencer who records videos in front of captions saying THIS CHANGES EVERYTHING.
So how do we recognize expertise now?
This is the question I keep coming back to. I don’t think AI makes expertise less important. I think it makes expertise harder to identify using the signals we relied on before.
If someone is talking about AI, I care less about which tools they have used than how they use them. I want to hear what they distrust. I want examples of situations where the model gave them something plausible that they rejected. I want to know what they do when the output is technically correct but ridiculous.
That is also why I would not put much weight on whether someone can produce an impressive AI generated work sample. At this point, most reasonably capable people can do that with enough time. I would pay much more attention to how they critique it.
Can they see what’s missing? Can they explain why one answer is better than another? Do they know when additional detail improves the work and when it’s just making the document longer? Can they tell the difference between a fact problem and a judgment problem?
Those are harder things to fake because they depend on the person bringing something to the interaction that the AI does not already have.
Our approach
We are obviously biased here. HIA teaches professionals and organizations how to use AI, including through seminars offered by our Academy. But this is also why our approach to AI education focuses less on learning tricks and more on using AI inside the trenches of work. Producing something with AI is not particularly impressive anymore. Producing something better because you used AI well is.
That means learning how to frame the work, evaluate and recognize when that little voice in the back of your mind should take the wheel. It also means accepting that AI can sometimes make a novice look considerably more experienced than they are. Which brings me back to Dunning-Kruger.
The problem may not be that AI makes people less competent. In many cases it will make them more capable. The problem is that capability and expertise are no longer as easy to separate by looking at the finished product. Someone can now produce expert-looking work before they have developed expert judgment.
So perhaps we should spend less time looking for people who can impress us with AI terminology and more time looking for people who can recognize when something doesn’t make sense.
I am increasingly convinced that excellent judgment and a strong BS detector are going to be considerably more valuable than an impressive vocabulary or the ability to code SQL.