On this page

← AIEd Audio

Orchestrators, Not Sages: Faculty Development in the AI Era

Episode 12 · · 22 min

Why the real bottleneck in AI-era education is the educator, not the student: faculty development as an identity shift from sage on the stage to orchestrator of a human-AI environment. The hosts work through the 40% overconfidence gap, TPACK and SAMR, trust calibration and explainable AI, discipline-specific assessment redesign (vibe coding, chemistry, the arts, nursing and law), and the protégé effect.

Episode 12 · 22 minOrchestrators, Not Sages: Faculty Development in the AI Era

Download the transcript (.txt)

Transcript

Transcribed automatically from the audio, with punctuation added and the paragraphs packed by sentence. The two voices are not labelled.

Right now, educators actually overestimate their own ability to use artificial intelligence by like a staggering 40%. It's a massive gap. Right. Because they think they're flying the plane, but then when the turbulence hits, they realize they have absolutely no idea how the navigation system actually works. Exactly. So welcome to the deep dive.

Today we are kind of flipping the script because we spend so much time talking about students, right? Oh, constantly. Students generating essays or using chat bots for homework or just the endless flood of AI tools built specifically for learners. But the real bottleneck in this whole educational revolution, it's sitting on the other side of the desk.

It really is. I mean, it is the single biggest vulnerability in the system right now because you can drop the most sophisticated, intuitive AI into a classroom. But if the educator isn't equipped to integrate it, the whole ecosystem just falls apart. Right. Which brings us to our mission today. We are focusing on what the research calls educational development or faculty development.

Yes. Which is basically the institutional machinery, right? The training, the support systems, the programs that are supposed to help educators confidently teach alongside AI. And we've got some fantastic sources to pull from today to really get into this. We do. We're pulling from a massive stack of source material today, it's four incredibly dense sections of an AI and education knowledge base, plus this really comprehensive set of FAQs.

Which is great, because they synthesize, you know, empirical research, pedagogical strategies, and even institutional governance. Yeah, and we're looking at this because whether you are a parent looking at a syllabus, or, you know, a student feeling confused by a new grading rubric, or even a manager trying to rollout AI tools in your own workplace, you're watching the exact same professional identity crisis play out.

Oh, absolutely. We're really looking at how professionals transition from feeling like completely displaced by this tech to actually becoming orchestrators of a human AI environment. And that transition is proving to be incredibly difficult because it's not just a software update, you know? Right. Let's go back to the 40% statistic you opened with.

Yeah, the 40% overestimate? Right. When researchers actually measure teachers self-reported competence with AI, so how good they think they are, against their actual performance on empirical verification tasks, the gap is just massive. Wow. Okay, so why is that? Well, the sources explain this through a divide between what they call output-directed literacy and system-directed literacy.

Okay, but wait. If a teacher is using chat GPT like every single morning to draft lesson plans or generate quiz questions and the quizzes work, how are they failing these verification tasks? And because they're relying entirely on output-directed literacy, which, to be fair, is what most of us do intuitively, right? You type a prompt, the machine spits out an essay or a lesson plan, and you just evaluate that final product.

Right. You just look at it and say, you know, is the grammar correct? Exactly. Does it flow well? Are the facts seemingly accurate? Yeah. And teachers are naturally brilliant at evaluating outputs because they've been grading papers for their entire careers. Right. They're basically grading the AIs if it were a highly productive, maybe slightly erratic student.

Yes, that's exactly it. So it's like they're judging the meal without knowing how the kitchen operates. That is a great analogy. Yeah, they lack system directed literacy. Which means what exactly? It means understanding the algorithmic construction behind the tool itself. So understanding the limitations of the training data or the data privacy implications of what they're actually typing into that prompt box.

Right, the mechanics of it. Exactly, the actual mechanics of probabilistic generation. Because they can grade the essay, but they don't fundamentally grasp that the AI is essentially just predicting the next most likely word in a sequence. So they don't get the underlying math. Right. And because of that, they don't fundamentally grasp why the AI might experience a hallucination, making up plausible sounding but entirely fake facts.

Right, right. And so they just don't know how to systematically verify the information. Man, it's like a seasoned pilot suddenly being asked to co-pilot with an automated drone they don't fully understand. Yes. Sure, they know what a good flight path looks like. That's the output, right? Right. But if they don't understand the drone's internal navigation system, they just panic when something goes wrong.

That's spot on. But I do have to ask, you know, is this really an existential crisis? Well, like can a school district just mandate a Tuesday afternoon workshop on data privacy and, you know, token prediction and just call it a day? I mean, if it were just about learning which buttons to click, sure, a workshop would be fine.

But the sources are really clear that educational development requires a profound identity shift. Identity shift. Yeah, because for their entire careers, these educators were the ultimate authority in the room they were the sole knowledge providers. Right the sage on the stage. Exactly. But now they're being asked to become orchestrators of human AI learning environments.

Which is a huge leap. It is. They have to manage a room where the students have instant access to an external synthetic intelligence they can generate answers literally faster than the teacher can speak. Okay let's unpack this because I hear a phrase like orchestrator of a human AI learning environment and immediately just think of the sheer exhaustion of a teacher on a Tuesday morning.

Oh for sure. Like they don't want a massive identity shift you know they just want to get through 30 lesson plans without losing their minds. Yep. So are institutions actually expecting teachers to fundamentally change who they are overnight or is there like a bridge to get them there? There definitely has to be a bridge and that's where specific integration frameworks come into play in faculty development.

Okay. Educational developers rely on frameworks like TPAC and SAMR to map out this transition so it just doesn't completely overwhelm the faculty. Okay, let's start with TPAC. What is that? So TPAC stands for technological pedagogical content knowledge. That's a mouthful. It really is. And in the context of our sources, it's actually being adapted into Gen AI TPAC.

Okay. But the core philosophy here is that you cannot teach technology in a vacuum. A teacher has to understand how the AI tech intersects with their specific subject matter, the content, and their specific teaching methods, which is the pedagogy. So TPAC kind of forces the teacher to admit that, hey, the AI might actually be better or faster at delivering the baseline content, and they have to figure out whether human pedagogy fits in around it.

Yes, exactly. It forces them to look at that intersection, and then to actually put that into practice, developers use the SAMR model. SAMR. Right. SAMR, which is essentially a step-by-step ladder for technology integration. It stands for substitution, augmentation, modification, and redefinition. Okay, let's ground this in a real classroom because I need a visual here.

Sure. Walk me through what a teacher actually does at each of those steps. Let's say it's a high school history teacher doing a unit on the American Civil War. What does substitution look like? Okay, so substitution is the lowest rung on the ladder. It's very safe, and it changes almost nothing about the actual learning experience.

Okay. So our history teacher takes a physical multiple choice worksheet about the Civil War and just swaps it out for a basic AI chatbot that asks the student the exact same multiple choice questions. Oh, I see. Yeah, the task hasn't changed. The pedagogy hasn't changed, just the medium. So the students are literally just clicking a screen instead of holding a pencil.

Exactly. And the problem is that a lot of educators get stuck right there. Oh, really? Yeah, they think they are integrating AI, but they're really just doing basic tool substitution. Right. So educational development programs are designed to push teachers past that comfort zone. Okay, so what's the next step? As they move up the ladder to augmentation, maybe the AI provides like real-time hints when the student gets a question wrong.

Okay, adding a little value. Right. Time to reach modification the task itself is actually changing. How so? Well maybe the student has to prompt the AI to write an essay defending the Confederacy and the student's actual assignment is to fact check and annotate the AI's logical fallacies. Oh wow that's entirely different. Exactly.

Because now the student is the evaluator not just the test taker. Exactly and then you reach the very top of the ladder which is redefinition. Okay. This is where you design learning experiences that were previously inconceivable without the technology. Like what? So the history teacher might have students use AI to create a real-time interactive simulation of the Lincoln-Douglas debates.

Oh, that's cool. Right. Where the student has to dynamically debate an AI Lincoln and the teacher grades the student on their ability to adapt historical arguments on the fly. That is amazing. But you know there is a massive collision coming here I feel like. What do you mean? Well we can train a teacher on SAMR, right?

We show them how to redefine a history lesson. Sure. But if they don't actually trust the AI to run that simulation fairly, or if they're like terrified the AI will give the student historically inaccurate information, they simply won't use it. You've hit the nail on the head. So what does this all mean?

Trust is the hinge on which all of this swings. Yeah. The research actually refers to this as trust calibration. Trust calibration. Right, because educational development has to teach faculty how to carefully calibrate their trust in these systems since both extremes are really dangerous. How so? Well if trust is too high you get what's called automation bias.

Okay. This is when a teacher uncritically accepts the AI's output even when it hallucinates a historical date or you know miscalculates a math problem simply because a computer generated it. Oh right and then on the flip side if trust is too low you just get total disengagement. Exactly. Like the teacher locks the laptops in a closet and goes back to paper and pencil out of pure skepticism.

So how do developers actually build systems that a skeptical teacher can trust? Well, they focus on a concept called explainable AI or XAI. But what's fascinating here is that the empirical findings are genuinely surprising. Oh. Yeah. For a long time, software developers assumed that the best way to build trust was to show the user the math.

Right, showing them the algorithm behind the curtain. Yes, they gave teachers technical, data-driven explanations. And it completely backfired. Wait, really? Why? It alienated the educators. Think about it. Imagine a dashboard where an AI system flags a student as needing remedial help. Okay. If the system tries to explain that recommendation by showing the teacher feature importance metrics or like vector weights or statistical probabilities, the teacher just rejects it.

I mean, I would too. Right. They don't have the statistical background to parse it, and it feels like a machine trying to dictate human behavior. So what works instead? The research shows that to build instructor trust, AI systems must provide explanations using curricular language. Okay, give me an example of what curricular language looks like in that scenario.

Sure. So instead of saying, you know, I recommend this intervention because variable x dropped below a point four threshold. Right. The AI needs to say, I recommend this intervention because the student's recent responses show a consistent struggle with synthesizing multiple sources, which aligns with objective three in your syllabus. Oh, okay. See the difference?

Yeah. When the AI uses the language of pedagogy, words like objectives, synthesis, scaffolding, the teacher can map the AI's logic under their own professional expertise. They can say, ah, I know what struggling with synthesis looks like. I agree with that assessment. Exactly. Okay, but here's where it gets really interesting. Let's say the teacher's on board.

They understand the system, they've shifted their identity. They trust the AI because it's speaking their curricular language. That trust is going to shatter the second Monday morning rolls around and they have to actually assign a grade. How do you grade a student who is using a tool you were just told to integrate?

I feel like everyone's instinct right now is just to buy a piece of software that catches the AI, you know, a plagiarism checker for the modern era. And that is why assessment redesign is the single heaviest lift in faculty development right now. Really? Oh, absolutely. And the very first thing development programs have to do is aggressively pivot teachers away from relying on those AI detection tools.

Wow, aggressively pivot them away. Yes, the research is definitive here. Using software to catch AI is just a dead end. Is that because it turns the classroom into like a surveillance state? Partly, but mostly because the technology fundamentally does not work as advertised. Oh, really? Yeah. The sources point out that AI detection tools have unacceptably high false positive rates.

They regularly flag original, human-written student work as AI-generated. Okay, unpack the mechanism there for a second. Why do they fail so often? Like what is the detector actually looking for that it gets so confused? So detectors generally look for two main things, perplexity and burstiness. Perplexity and burstiness, okay. Perplexity is basically a measure of how predictable the word choices are.

And burstiness is the variation in sentence length and structure. Got it. So human writing is typically very bursty. We write a short sentence, then we write a very long meandering sentence with multiple clauses. Right. It varies a lot. Exactly. AI, on the other hand, writes very uniformly. But the problem is that non-native English speakers or, you know, students who are just learning how to write academically, they also tend to write with low burstiness and highly predictable vocabulary.

Oh, no. Yeah. So the detectors disproportionately flag international students and non-native speakers, which obviously causes severe equity concerns and elevates student anxiety to just toxic levels. Wow. So if we take away the radar gun, how do teachers enforce the speed limit? Like, if you can't scan an essay to see if it's fake, how do you know if the student actually learned anything?

By completely changing what you are measuring. Okay. Development programs train faculty to design authentic assessments that focus on evaluative judgment. Evaluative judgment. Right. It's the capacity to evaluate the quality of a piece of work against established criteria. So instead of asking a student to generate an essay from scratch, the assessment becomes, hey, here is a deeply flawed AI-generated essay on our topic.

Critique it. Find its historical inaccuracies. Improve its logical flow. I see. It's like grading the scratch pad workings in a math class rather than just checking if they got the final answer right. That's a perfect comparison. When I was in school, you could get the right number at the bottom of the page.

But if you didn't show your work, you didn't get the points. Because the teacher wanted to see the cognitive path you took to get there. And with AI, the scratch pad isn't just numbers on a page. It's the prompt history. The focus shifts entirely to process over artifact. Teachers are being trained to design tasks whose integrity survives inspection.

So when a student submits a project, they aren't just submitting a final essay. They are submitting the transcript of their entire interaction with the AI. Wait, so the teacher is literally grading the conversation. Yes. They're looking to see if the student, like, asked a lazy question, got a bad answer, and just accepted it, versus actually challenging the AI to dig deeper.

Exactly. They look at revision decisions. When the AI provided a generic summary, how did the student refine the prompt to force the AI into a more nuanced analysis? That's fascinating. Yeah, the final standalone artifact is no longer the sole measure of learning. Learning is measured in the critical choices the student makes along the way.

Okay, I love that, and I hate to be the skeptic here. Grading a prompt history in a high school history class has to look wildly different than grading one in a college physics lab. Oh, entirely. Like, you can't just put a nursing professor and a creative writing teacher in the same seminar and say, hey, here's how to grade an AI interaction.

The logistics of that training seem kind of impossible. Well, if we connect this to the bigger picture, this is actually a major finding in the research regarding institutional change management. Okay. Change only works if professional development respects discipline-specific realities. Great. General purpose models just do not work across the board. The phrase that the sources use is AI in the disciplines, really emphasizing that one size fits none.

Okay, break that down for me. How does the training actually differ depending on the domain? Let's start with STEM. Okay, so in the STEM cluster, particularly computer science, The landscape has fundamentally changed because of a workflow called vibe coding. Vibe coding? Yeah. That sounds less like computer science and more like, I don't know, Spotify playlist.

How does a teacher grade a vibe? I know, it sounds funny. But it refers to a workflow where a programmer doesn't manually type out raw source code syntax anymore. Okay. Instead, they specify what they want the software to do using natural language prompts, and the AI generates the code. Code. The human's job is to evaluate the resulting program execution.

Oh wow. Yeah, so for computer science educators, faculty development is no longer about teaching teachers how to spot a missing semicolon in a student's code. It is about training teachers to evaluate a student's natural language logic and their systemic testing. The student is basically the project manager, the AI is the junior coder, and the teacher is grading how well the student manages and tests the AI's output.

That is wild. And what about something physical, like chemistry? Well, in chemistry, AI can generate complex molecular models, right, but it doesn't actually understand physics. Okay. So the faculty development focus there is on teaching educators how to design assignments where students must verify if the AI's output violates physical constraints or chemical logic.

I see. It's all about multi-representational verification, making sure the AI hasn't drawn a molecule that would just, you know, instantly combust in the real world. Okay, that makes sense. The hard sciences, but let's look at the humanities and arts, because bringing an automated generation machine into a creative writing or fine arts class feels inherently contradictory to the whole point of art.

It is highly sensitive, and the training definitely reflects that. For English and art faculty, development focuses on using AI for iterative drafting and conceptual scaffolding. Okay, so not doing the work for them. The training ensures that teachers know how to set boundaries, so that the automated generation doesn't displace the student's artistic intent.

They teach faculty how to use AI as a sparring partner for ideas. So the student might use the AI to generate five different structural outlines for a story, but the student still has to write the prose. Yes. Because if the AI paints the whole picture, the student isn't an artist anymore, they're just a patron commissioning the work.

Is exactly what the assessments are designed to prevent. Right. And then you have the accredited professions cluster. So fields like nursing, law, and teacher education. The stakes there are astronomical. Yeah. Like if an AI writes a clunky poem for my English class, it's a bummer. But if a nursing student relies on an AI that hallucinates a drug interaction and the professor doesn't catch that the student didn't verify it, people die.

Exactly. This is high-stakes faculty development. In these fields, instructors must be trained to design assessments that explicitly prove the candidate's certified professional judgment remains strictly their own. Right. So a law student might use AI to simulate a courtroom argument in a low-risk environment just to practice, but the faculty must design evaluations where the human's autonomous judgment is undeniably verified without the AI present because at the end of the day, you can't subpoena a large language model, right?

And you can't revoke a chatbots medical license. The human has to hold the accountability. And that is exactly why educational development is the linchpin of this entire era. We are basically asking faculty to redesign the fundamental mechanisms of accountability in their disciplines. Man, if you are listening to this and wondering why your workplace feels so chaotic right now or, you know, why your kid's syllabus suddenly looks completely different, this is why.

Integrating AI is not downloading a new app. It requires rewiring how a professional identifies themselves, you know, moving from a solo provider of answers to an orchestrator of a complex human AI ecosystem. It changes how you evaluate everything. Absolutely. You stop asking, can this tool do my work and start asking, how does this tool change the way I verify and understand my work?

It is a profound shift in epistemic agency. We are changing who holds the knowledge and how we prove it. But before we wrap up, there was a concept buried deep in the learning theory section of our sources that just kind of flips this entire dynamic on its head. Yeah, we have spent this whole time talking about training human teachers to adapt to AI.

The protégé effect. Yes. Idea that students learn best by teaching someone else. Only now, the sources mention that students can learn by teaching synthetic AI agents, like the human learner becomes the teacher and the AI becomes the student. It is a fascinating place to leave this discussion, because we are currently pouring all our resources into faculty development, training human educators how to cope with AI.

But as these models evolve into agents that human learners can actively practice on, We really have to wonder, will the next frontier of educational development be programs designed to help humans teach AI how to learn? Just imagine that. The pilot isn't just co-piloting the drone anymore, they are actually responsible for raising it.

Wow. Yeah. Something to mull over as you watch these tools evolve in your own life. Thanks for joining us on this deep dive.

Concepts in this episode

Questions this episode answers

This podcast was generated with Gemini Notebook, the AIEd notebook built from these pages.

Embed this page

Copy the code below to embed this page in a learning management system or other website. The embedded version shows only the page content — it hides the site header, navigation, and footer.