Episode Transcript
[00:00:18] Speaker A: Clint, welcome to the show.
[00:00:20] Speaker B: Thank you very much for having me around. It's always nice to see you, dude.
[00:00:24] Speaker A: I am so stoked to have you here. And you know, I'm reflecting on how hilarious it was, like how we were reconnected.
It was about a year ago, would you say?
[00:00:37] Speaker B: Yeah, we were at Delis and Saracen, like basically right around the corner from like my apartment. I was like, hey, good to see you. You should not eat here. You should eat somewhere different.
[00:00:46] Speaker A: Yeah, that's right. But like, you were like literally sitting within a foot of me and I got up to go to the bathroom and you're like.
And then two guys that have known each other for, I don't know, like 15 years, who hadn't seen each other in a while were suddenly like, oh, end up having dinner, hanging out, and we've been chatting ever since. So. What an amazing. Just crossing of paths.
[00:01:09] Speaker B: Yeah, it's. I think it's one of those things that New York has been really good for. Cause people wander through here and I mean, I don't know, I eat at the same like five vegan restaurants all the time. And they usually tend to wander through one of them. So it's always nice to reconnect that way.
[00:01:23] Speaker A: And then you introduced me to what has now become mine and Monica's favorite restaurant, which is Anixie.
[00:01:30] Speaker B: Do you know Anthony Amour by chance?
A long time ago, he used to tour with Newfound glory and stuff. He's like a vegan, straight edge dude from Florida. But he's in town today and we have a reservation at anixie tonight at 8:15.
It is also my favorite.
[00:01:48] Speaker A: It's like insane. Anyways, we can go on and on. Let's talk about you. So for the uninitiated, for those who don't know, who are you and what do you do?
[00:01:59] Speaker B: Great. I'll start with what I'm up to now and then go back just a little bit. So I am an assistant professor of applied statistics at New York University.
So I've been studying how we sort of measure cognitive traits in people through tests.
So often this is a lot of modeling, It's a lot of test design.
That's sort of what I do at the core these days. I also do sort of quantitative work in general, social science sense.
I got into this because I was honestly a high school physics teacher for about eight years before that.
So it was a way to sort of think about blending my quantitative skills and my educational background. And then before that, the way we know each other is through hardcore, playing music, and being big youth at today. Hex.
[00:02:48] Speaker A: Heck, yeah.
What is the meaningful difference between an assistant professor and a professor?
[00:02:55] Speaker B: Yeah, there's actually.
So there's really three levels of professorship, because assistant, associate, full, or like just professor. And so assistant professor is pre tenure, but associate and full professor are post tenure. So I'm entering my third year, so in a couple years I'll go up for my big tenure review. So they have essentially two opportunities to fire me from this job. The first one is coming this next year. And if they don't elect to fire me, then they can fire me three years after that. And if they don't elect to fire me, then they can never fire me unless I do something very, very bad.
[00:03:29] Speaker A: But what about from a workload perspective, what people are engaged in?
[00:03:33] Speaker B: Oh, that's a really good question. So typically you're going to be as a full or even associate professor, you're going to be doing less research and more sort of service, serving on committees, chairing faculty searches. Just sort of like doing odds and ends that keep the university and department kind of functioning.
Oftentimes, too, more advanced professors will teach less because they have grant funding to buy them out of some of that time. So a misconception is that teaching is a big portion of the sort of job that I do.
In actuality, I'm really a. A researcher who teaches courses so that the university gives me an office to work out of.
[00:04:17] Speaker A: It's funny because, like, you've got this. I would imagine many people, myself included, have this idea about professors that it's like they're calling to teach and that's what they want to do, but it's not quite that. Right. It's more like. About research.
[00:04:30] Speaker B: Yeah, for sure. And so it's really a research position. That's sort of the core thing that I'm evaluated on when I go up for tenure. But I do really care about teaching. It was one of the things that actually drew me to this particular department I ended up taking the job with.
So I do teach courses. I really love teaching courses. And that's sort of an extension of the time I spent in the classroom when I was in Philly, too.
[00:04:54] Speaker A: So how does it work, that relationship between the university and a professor or an assistant professor like yourself?
If teaching's only one of the components, what's that relationship? And what's the benefit for the university of having someone like you in the mix?
[00:05:10] Speaker B: Yeah, I think there's two major benefits. The first is, at least from the university's perspective. I do teach some courses, and if I'm not teaching courses, I have to pay them the money I would have gotten to teach those courses, essentially buy myself out of them so they can hire someone else to do it.
So at the very least, it's like there is that sort of teaching as part of the function of the university as far as students are concerned.
But the other part is being able to attach the New York University name to the research I do. Hopefully the research I do is good and it's the thing they do want to attach their name to, but that's really sort of the two core bits. And so a lot of the prestige that universities get is because of the professors they attract. And then the professors they attract, you know, teach some amount of courses which sort of like does the university function that most people kind of think about.
[00:06:02] Speaker A: Right. So like a. A professor that had a great body of work would attract students who would want to learn with them, which would attract revenue and that. But also that prefer professor through grants they would get would be able to. I like the term you're using, like buy themselves out of having to teach more and they'd be able to focus more on research. Is that. Is that how it works?
[00:06:23] Speaker B: Exactly. And so the main thing is that I'm bringing in grant money eventually. And that grant money is a huge boon to the university. It's going to be hundreds of thousands of dollars, probably millions over the course of a career or millions over the course of a single grant, if you're in the biological sciences or something.
[00:06:41] Speaker A: Well, then what is a university?
It sounds like a university is kind of like a revenue generator.
[00:06:48] Speaker B: Okay, so there's a couple of ways to think about this. And actually this is a conversation I was having recently in light of sort of some of the ways that universities are currently under attack and some of the ways that university administrators are really responding to those attacks.
Universities are not really a collection of courses or a collection of buildings. They're a collection of people that sort of try to form some kind of intellectual community. That's the way I think about it. But prior to this, I did my PhD at Stanford, and there the joke was always, Stanford is a hedge fund that also offers some courses as well.
So there is a huge sort of revenue generation, endowment, kind of endowment, growing portion to the work that universities do. But I don't know, they're just these really bizarre institutions.
[00:07:37] Speaker A: Well, I mean, I feel there's more we got to talk about here, so I'll give you an example. I Went to ubc. I did.
I did a couple degrees at ubc, University of British Columbia for, For those who don't know. And my experience there was like, oh, I was fine, but it just felt like really transactional, you know, like, I come in, I spend. I spend a bunch of money for the courses. I'm essentially buying a. Buying a degree that of course I have to work for.
Yeah.
And I get that degree from UBC as a rapid university, and it makes me more marketable.
At the same time, I've got all these student fees that are going to all of these funding these things. But it's also the university is renting out parts of the university to film. Like X Files was filmed. A lot of X Files was filmed at ubc. And like, parts of the university would be like, shut down. You'd have to navigate things.
And then afterwards, the university continues to ask me for money constantly. And I'm not like against any of these things. Like, it's like a transactional thing.
But I always kind of reflect, like it really felt.
It felt like I was consumer the whole time. And I really wish I'd viewed it a little bit more like that at the time. Like, I should be a little bit pickier about the school I go to, about the professors I use, like, that I'm engaging with, like, how I'm building my profile. But at the time I was like, I don't know, I guess I go to university when I'm done school and I kind of figure it out from there.
[00:09:02] Speaker B: Yeah, I totally agree with that. I think a lot of people really recognize that consumer relationship, especially during undergraduate degrees. You're just like, this is the next thing that I do in sort of the path I'm supposed to take to end up somewhere as an adult.
But the thing I've really learned here, working with master's students, is that a lot of master's students, especially the international students we have here, really do treat it in that kind of transactional way. And they're very clear about their goals up front. And that's really changed the way I actually do a lot of my mentorship even too, where when I sit down with a new student who's maybe looking to do a research project or wants to get involved in some of the work that I do, I say, what are your goals? What kind of career thing do you want to do? Do you want to leave this degree and go into industry? Do you want to leave this degree and go into a PhD program? Because the work products that we're going to need to produce for you to realize those goals are going to be really different. And it's also going to change the way I spend our time and manage our time and the types of things I want you to do because I want you to gain the skills that are important to you. I don't know that everyone necessarily has that view with mentorship, but I think recognizing that degrees are crazy expensive and that things like GPAs, which we often view as really important signifiers, are measures of nothing more than really compliance. They're just like, how well did you do the role of being student in school?
And so really trying to think about what the value add I have for students and what sort of also I can get in return from mentoring students both in sort of a work product way and also sort of just like I enjoy mentoring people and think it's an important thing for people to do.
[00:10:44] Speaker A: Yeah, it's really interesting that you say that.
So when I was doing my, I did a BA in English lit and the people I dealt with, the instructors I dealt with were fine. Like, fine. But they would run the gambit of being completely disinterested and just like, I'm going to teach some stuff.
Give me your paper.
See you later. Right. And it's like, I won't remember you. You won't remember me to being, I'd say like moderately engaged.
But when I started to go into higher levels, like when I started doing my psychology and my counseling stuff, it was way more personal. Like I actually remember those professors and they were very interested in what was next for me and presenting me with a lot of ideas. And it actually like surprised me because I did two back to back BAs and the English one, English lit one was like, literally like, it was like the subway of education. I just went in there like, what do you want on your sandwich? And like, that was it. I don't remember one professor except for the worst one.
My psych degree was a lot different, was a much, much more tailored. And of course we're talking about classes that have like, like 100 people in them sometimes. Right? So I get it. But when I started getting into those smaller and more advanced classes, I did feel a real difference. And that was where the first time I really started thinking about the monetary aspect. And this is actually a purchase I'm making. It's also when my, you know, I had to start taking out student loans, so I actually had to like use money. I had to pay back. So I started becoming more thoughtful about the monetary side of It.
[00:12:16] Speaker B: Yeah, I think the value of a college degree or a university degree is something that people don't particularly understand, and for some people, I don't think is really worth the amount of debt they're about to incur.
I think experiences that you seek out are really important to align to what your eventual goals are.
What do you want to do? Where do you see yourself in 5, 10, 20 years?
And then what are the steps you actually need to take to do that? Because oftentimes there will be ways to reach those steps that don't require a particular degree. Like, I can think of one very clear example. You talked to Cammie, my partner, earlier, and, you know, I think about the career she's in now, and that's a career that's often sort of taken on by people with PhDs.
But she was able to find a path to navigate to that same level with only a master's degree. And that was a thing where she saved five, six years of her life not doing the PhD and going this other route. And so I think having clearer goals is hard because we're often asking young people to look very far in the future when they haven't had that much life experience. But it's an important thing to do to really, actually weigh the possible value proposition of any big purchase you're making.
[00:13:42] Speaker A: Well, it's interesting to me because it seems like there has been a shift. So you'd mentioned earlier, and we get into this more universities and university administrators are under a different kind of pressure now. And there's the whole political side, but also a lot of the questioning whether or not the formal education system is actually worth what it used to be worth. Is it, like, are people actually learning things that is portable? Is it just assuming a bunch of debt?
I have a bunch of different perspectives on it, and the easiest one that I'd go with is, let's say, my English lit degree.
Do I use it in a professional sense? Am I actually out trying to write things for a lit magazine or anything like that? No, not at all. But I definitely use it every single day. I use it every day where I'm writing an email, when I'm generating a report, when I'm reading things, when I'm analyzing things, looking for themes. I literally use it every single. I cannot think of a day I don't actually use it, like, super active, but it's not like a leading foot. Like, it wasn't something that I set out from a career path on. I thought I was going to be a high school Teacher, which is why I did English Lit. Thought I was going to be a high school English teacher. But it's highly portable. It's created a great, a great skill set for me to lean into. But like, had I had, I tried to like go make a career just based off of that, I would have had a hard go of it. But my psych degree combined with that, combined with some of the other trainings has given me a skill set that's like super portable. I could go to all sorts of things, especially including things that have nothing to do with English, like English lit on the surface or psychology. Like, I could actually go into something that would seemingly be completely unrelated, but I have a skill set that would allow me to position myself very, very well for that. Plus you've got degrees from like reputable university, so people would be at least interested. So from like just a value proposition, I see. Like, no, like, I think, I think a university education, even something that might be kind of generalist, could still be very useful and really portable. But what are your thoughts on it?
[00:15:44] Speaker B: I think the generalist sort of liberal arts style education, especially at that sort of bachelor's degree level, is an amazing experience.
And I think it's something that I think more people should really value.
In the sense that you're learning a lot of skills that, you know, am I going to use classical philosophy or continental philosophy, or am I going to use English literature day to day in my work? The answer is probably no. But the things you're learning are writing skills, critical thinking skills, how to read, how to focus, how to do research or evaluate things, how to evaluate arguments, statistical reasoning. All of these sorts of skills, I think are really, really important and having a time to learn and specifically train those skills while you're also learning to be an independent adult in a place where you're like, still a little bit babysat but kind of on your own more than you probably were before. I think is a really valuable experience for almost everyone.
[00:16:44] Speaker A: I'll give you the flip side of the coin.
The best therapist I ever worked with that I learned the most from was someone who had been kind of grandfathered in, had been a therapist, and basically took like some course in the, I don't know, like the 70s or something like that, but did not have any kind of university degree, had just kind of learned through doing.
And at least in Canada and at least in bc, where I live, to be like a proper therapist at this point, you have to have a master's degree, have like certain designation out here. Called an rcc.
But when I first became a counselor, you didn't have to have any of that. You just basically had to go get like this like thing that you could get done on the weekend. Over a year, this certificate, it's scaled up a lot. And part of me appreciates that. Like, like, you know, the, the barrier to entry is quite a bit higher. I appreciate that because it tells you that this person's done a certain level of work to get there. But on the flip side, I think that barrier to entry could keep really talented people out. And also, frankly, like some of the worst therapists I ever met were the people who had the highest level of education, like totally like out of their minds, like really bad therapists. And some of the best were people who actually had a much like lower level of like let's say formal education, but were just good with people.
[00:18:05] Speaker B: You know what's funny? I think that some of the best therapists I've ever had tended to be MSWs as opposed to PhDs.
That's like something I noticed sort of right away when I first started doing therapy. But to your point, I think there's no better way to learn a job or a skill or a particular thing than an apprenticeship model where you're doing the job under someone who has also been doing the job.
There's nothing better than that. The problem is that's super duper labor intensive. It's often a large amount of one on one hours with another person. And so I think in some cases that initial sort of wider scale or broader upskilling to get people to the point where they're ready for that apprenticeship is really helpful.
But I think about PhD programs or one on one apprenticeship models, tattoo apprenticeships, apprenticeships in the trades. These are how you build people and how you train people to do skilled specific labor. And there's just no substitute for that.
[00:19:12] Speaker A: Yeah, I love what you're saying. You know the type of training that I got on job when I was a therapist, it was real interesting just spending time around someone that was like a mentor to me.
It was this person named Diane. She was like unbelievable. Like really, really, really strong therapist challenged my thinking all the time. Like really knew how to like tease out the back fast in me. I could go and get some good supervision from her. Even though she wasn't my boss. I could get like a level of clinical supervision from her. Really strong. Whereas like my, my actual boss at the time. It's like the trainings were these just like real box checky, like dude what, what, what does this actually do for anyone? And it really stood out to me. And, and so he was like a master's level and, and she didn't have a formal education. And again, like, you know, it's just as like the one example, but from like a therapeutic place. I do value some guidelines of who and how can work in a therapeutic setting. But also that, like, barrier to entry for really good people. I think that space for, like, mentorship and bringing someone along almost like a trades. Trades person.
I feel like there needs to be more space for that because it's like, I think you're keeping great people out of the industry.
[00:20:26] Speaker B: I agree with you completely. I think there are so many people who have such a degree of interpersonal knowledge, care for other human beings and personal experience that could be in, say, therapy, doing the work much better than people with a large amount of formal book learning.
But because there's a barrier to entry, you need to do some sort of coursework up front. You need to do some sort of certification program up front. That stuff's all super expensive. And a lot of times you're going to take decades of your life to realize enough economic gain to even make that start to become sort of worth your time. One of the things that I remember when I was doing my master's in computer science is it was framed to me as like, they're training you to get the job, but you're not going to know how to do the job until you really have the job, because everything is so idiosyncratic. So it's like you're being trained to prove to people that you're smart enough to learn and do the task that they're hiring you for. I found that to be like, a really unsatisfying way to think about that sort of educational experience.
[00:21:39] Speaker A: Like, to build on that. And I'm glad you brought in the computer science as part of it. I also think of things like engineering where it would be like, well, no, you gotta have a, you gotta go to school for engineering. It's like, what are you talking about? Like, where do you think engineering came from?
Like, you know, like, it's just like people kind of figuring things out over time and, and a lot of those things that say, like, early, early in, like, the early scope of engineering, it'd be a lot of people coming up as laborers and just kind of figuring it out as they went along. I use my dad as an example. My dad was born and raised in Syria and had no, no formal education outside of high School and worked at quite a, quite a high level as a water engineer and traveled all over the world, like, essentially like figuring out how to bring water to places.
And the stuff he did was like super cool.
Until the standards of education rose to a place where a company needed, you know, during the Canadian recession, a company needed to make decisions like, who do we keep? Do we keep this person who doesn't have a formal education or do we keep someone who has a formal education and who we can justify the salary? I remember for my dad, that was like a real blow for him because he never worked as an, as a, as an engineer ever again.
And he at that point had kids and he was like, you know, in his 40s. He didn't really have a chance then to go to school. And, and also like the Canadian school system versus what he grew up with. All of, all of these things.
That idea that I really value my formal education and the degrees that I have, but I also like, just recognize that that's just not a, it's a barrier to so many people who really could be doing super high level work in the most incredible way. I'm real interested in how we bridge that gap where that idea that a degree or formal education is the next step for everybody, because I just think it's a redundant way of thinking with you completely.
[00:23:26] Speaker B: And it's funny, I think about a lot of the master students I work with now who are right now on the job market and they're like, oh, I'm applying for these jobs. They're sort of entry level jobs as statisticians, data scientists or whatever it is.
And they all want five years of relevant experience. And they're like, how can I get five years of experience? I just came out of school. But the flip side in your dad's case is your dad had many years of relevant experience to prove that he should have been able to do this job as well, if not better than most people.
But that wasn't enough. And so it's like, it's almost like a chicken and egg thing, right? How do you get the experience?
It's. If you can't get the job and how do you get, you know, I don't know, Never mind.
I lost myself on that one.
I was like, I was going somewhere with that.
[00:24:21] Speaker A: Let me hit a question that I, I think I understand, but I, you know, I think it'd be cool to tease out for the audience, like, yeah, for sure. What exactly is tenure and why does it even exist?
[00:24:34] Speaker B: Oh, that's a very, very Very, very, very good question.
So the thing to remember is that a university professor job is a contract job. You sign a contract and that contract is typically for some span of time. So the assistant professor contract is typically six years.
And you have sort of a midpoint review, at which point you can be dismissed and you have an end review, at which point you can be dismissed after that sort of six years is up. At the end of the sort of assistant professor contract, you either need to be essentially rehired at the associate level or dismissed.
[00:25:12] Speaker A: Now.
[00:25:12] Speaker B: The associate professor contract has no expiration. So it's not a time limited contract. And so the tenure process is really related to that. There aren't formal scheduled reviews anymore. There isn't an end to your service.
It's just you're hired now, you're part of us. And so the purpose that most people typically talk about is that it's meant to give professors the opportunity to do sort of the work that they find inspiring to do, work that they find interesting, maybe scope out longer term projects than you can really do in those first six years when you know that they might not necessarily pan out for you research product wise.
But also it's meant to sort of preserve some degree of kind of academic and intellectual freedom. So the point being that a tenured professor can sort of say, I'm an expert in my field, these are my beliefs, this is what I've learned, this is the research that I've done. And if it does or does not agree with sort of public opinion or previous work or something like that, they're sort of protected in the sense that like, oh, I can continue to do my work, I can continue to sort of output things that are meant to be evaluated by the scientific community.
And I can't be dismissed for doing things that are unpopular.
And so it's a little bit weird the way that kind of tenure is framed these days. I think of tenure as a really good thing. I think of it as an important part of developing the profession. I think, for example, there are research projects that I might like to do that are going to take probably five, six years to really start to produce any fruit, sort of any papers get any real results. I'm going to need to assemble a team, I'm going to need money, I'm going to need all these different things to sort of realize these projects.
If I start a project like that now, when it comes time for my review at six years, I have literally nothing to show for this, despite having sunk all this time in. And so you end up at a spot where there's no incentive for me as an early stage professor to do this type of work, even if that work is super important or could move the field forward or might help us develop some generalizable knowledge. And so that lack of.
That lack of an ending date on that contract is really meant to allow you to do that type of work. And I think that sort of work is especially important in the humanities. It's especially important in some of the natural sciences where people are doing these enormous, long experiments.
It's also important to let people ask questions that might not be popular but might be of interest to them, or push their thinking in new directions.
There's this weird thing happening now, though, where there's kind of this projection of like, oh, we don't care about academic freedom or freedom of speech or something like that.
And the way that people seem to want to fix that is by, like, waging war on tenured professors, where the idea is, I personally don't agree, like, tenured professors and university professors tend to skew fairly liberal. Like, that's just a sort of way of the world.
And so if you don't like the way that those sorts of people think, you're like, oh, well, maybe I can attack tenure and then get them out and start to replace them with people who might agree more with what I want to do or blah, blah, blah. And it's weird because I think the way that the whole system is supposed to function is that people who think particular things or think different things are meant to be sort of somewhat protected or insulated by the system. And so you're. You're saying, like, I want to encourage freedom of thought, freedom of speech, sort of like campus freedom or whatever it is.
And I want to do that by dismantling the thing that preserves that for professors.
It seems, like, super illogical to me.
[00:29:26] Speaker A: So what's the counter argument to what you're saying? Like, where is that coming from?
[00:29:30] Speaker B: Yeah, okay, so there's one of the counterarguments is, and I think you can totally speak to this, given what you've already talked about with your sort of English lit experience, is that like.
And this happens also in sort of like K12 education, where people get so installed in a system they stop doing their job well and they're just sort of able to kind of like sit back and coast. And it's not really like they're not teaching well, they're not doing as much research, they're not doing all of these sorts of things.
And so the counterargument I guess is then like, well, if this is a system that props up people who don't want to do their job, like do their job, whatever that means, right.
This is a thing that's net bad for the sort of consumers of courses. It's net bad for sort of the universities, it's net bad for students, and it's really driving up cost of an education without giving the students additional value. I think that's kind of one of the things.
The other thing that I think is a counterargument too is that the tenure system, the way it's built, also protects people who do things that are really, really, really objectionable.
It could protect people who are actively studying things like eugenics or scientific racism or things like that.
Those, I think are counterarguments.
But I don't know, at least I don't want to believe, I think is probably a better way to say it. I don't want to believe that those are really the norm.
But I do know that they are real because we've all had old professors who don't really seem to want to be doing the thing that they're doing anymore. But also I think there are, we talked about that buyout system a little bit earlier where I think there are lots of people who are doing work, sort of grant funded work, and they're just paying other people, paying for the university to find other people to teach courses. And so that's a thing that's sort of maybe offsetting some of that dereliction of duty, if you want to talk about it that way.
[00:31:29] Speaker A: One of the things that seems to be really bubbling especially I'd say this year, is the idea of politics and college university education.
And it's to me, like, to me, university is like college university is like where I really like got into that stuff. And I think that's for most, most people like high school, like late high school is like when I got introduced to, you know, dead Kennedys and all that kind of stuff, like junior high, and it was like very just kind of broad thinking. But then in high school it got a little bit more nuanced. And then a university got, got much more nuanced and I was much more active.
So I find it interesting that there seems to be this, this push. It's like kind of a 2024, 2025, maybe even a 2023 thing where it's like, well, universities have become this like ideological like brainwashing machine. It's like, what do you what?
And like it shouldn't be that and you should just be going to classes and just learning things. It's like I, like I. It seems counterintuitive to at least what I experienced in university and what I think most people would expect.
University, that's where you're at the depth of those ideas. But I'm real interested in your thoughts on that.
[00:32:49] Speaker B: Oh, so I agree with you completely.
College is where I got really interested in veganism and animal rights and really started to go down that route. I entered university as a vegetarian and within a couple of months I was vegan and I've been vegans ever since. And so thinking about having access to those ideas, being able to again be an adult, making these sort of decisions on my own and taking care of sort of like my own nutrition and my own sort of like grocery buying and all of that stuff was really important.
But I also remember like, and I don't know if you had this at ubc, but like there would always be like tables set up outside of the cafeteria with like some crackpot pedaling something where it was like Lyndon LaRouche trying to talk about the transcontinental railroad or it was, I can't think of their name. But there's this one particular campus socialist group that's real crack body campus socialist groups, super common, but they're even the fringe campus socialist group or the people who would have the signs with the chunks of aborted babies being super pro life. And I think universities are important because it's a time to be exposed to that stuff and make your own decisions about what you do or don't believe.
You get to weigh those political options. You get to see more of what's out there than just what you've been exposed to. And I think that speaks back to what I said earlier, that universities are a collection of people. And collections of people facilitate this sharing of ideas and sharing of experiences and sharing of culture that is really important. And often a thing like universities are often the first time people get to really experience other cultures and other viewpoints. And where you go from that is up to you is totally fine as long as you're making your own decisions.
[00:34:53] Speaker A: Yeah, so I've got a counterpoint question, but also just to support what you just said.
I'll never forget I was in the student union building at Yale UBC and I was just sit. Like it was just this big, like big open space basically and I was sitting in this like corner of this big open space.
And I mean big open space. It was not Like a closed meeting room or anything. And I'm just sitting there steadying. I had earplugs in.
I didn't like going to the library at the time. It's just like I needed more activity around me as I, as I was reading.
I was just sitting there reading and I feel a tap on my shoulder. I look up and this guy was like, hey, are you here for the meeting?
I was like, what? He's like, oh, it's the, the, the communist group of ubc. I was like, oh, no, no, no, I'm just here studying. He's like, oh, well, I'm gonna have to get you to move because this whole area has been reserved for the, for the communist meeting. And I was like, what? She's like, it's like a big, I mean big. He was like, yeah, we've reverse reserved this area. I was like, well, how many communists are you expecting?
He was like, oh, I don't know, like it could be a lot of people. I was like, dude, this area could easily take like more than 100 people. I'm just sitting in a corner. What does it matter? He's like, no, if you're not involved in the meeting, you can't sit here. And I was like, are you sure you're a communist?
Is this actually like your political thing? Because I don't see like you would care that I'd be sitting here. We start going back and forth and I'm enjoying the hell out of this conversation because it's kind of like funny and like I'm having fun with it and homeboy is like serious, like real rulesy. Like, no, we booked this space. You cannot sit in this space. We cannot spare even three feet of this space. And I was like, yeah, man, I'm not going to move back in the cab. And like, I don't know, like 15 people showed up in this space that could hold a hundred people. It's like that to me is like university, like that university kind of like push and pull around stuff. And it was funny and I like, I loved it. And every time I saw that guy, we'd kind of have a laugh about it because he realized how, how funny it was. But like, dude, yeah, like that seems to me to be the place where you figure this stuff out. But here's the counter argument, and it's certainly not my, my counter argument, but I think it's the thing that gets fasted a lot is, and you'd already said it, is that like a lot of university professionals tend to trend Liberal.
And I think the thing that gets pushed out by a part of a society would be, well, yeah, so if you only have liberal idea, if you have liberal ideas, those are the acceptable ideas. But you can't have different kinds of ideas than that actually engage in campus life fully. And I haven't been in university for a long time, so I can't speak to that. I can certainly say when I was young that wasn't at all the case. But what are your thoughts on that now?
[00:37:39] Speaker B: I think that's a fallacy. There's always campus Republican groups. There's also always business schools, economics departments where professors are going to skew more conservative. You have lots of different religious and political views within the professoriate.
I think maybe humanities departments or sociologists or sort of critical theorists. I think critical theory is a thing that's become a lot more common in research and also a lot more sort of public in research lately. And so people often want to attack whatever kind of nonsense they want to call it. But you know, there's, I think about nyu, right? NYU is often viewed as probably one of the most liberal universities in the city.
And we have plenty of campus Republican groups. We have students who are extremely conservative.
You know, I think there's, the other ideas are represented. There's, you know, sort of campus religious groups. There are, you know, and those religious groups range from sort of like super Catholic to you know, sort of anything else. And so all of those things are represented here. But also you have to realize that like when you're at a university, you're not confined to only be at that university. Right? A university again is a collection of people, but those collection of people are within a physical place and there are other people sharing that physical space that might not be like at the university or doing university things. So if you need a particular sort of, you have a particular ethnic identity that's really important to you, you can often find a group of people who are related to that. Or if you have a particular religious identity, they might not be at the university, but you can probably find them in the community.
And if those things are important to you and you're trying to select a university and you, and you pick a university where none of that stuff is around or available.
I kind of think you've picked wrong, if that makes sense. Right. Like you should think about what you need as a person to grow again, thinking more about those long term kind of goals. Like if you, if you're like, oh, I really want to connect more with sort of, you know, like you said your dad was from Syria. Like your Syrian heritage. Like maybe you should pick a university where there's. This is in a city with a Syrian community or at least something else that's sort of important to you.
[00:40:11] Speaker A: My dad's actually Armenian. He just grew up in Syria.
[00:40:14] Speaker B: I knew that. I'm sorry about that.
[00:40:16] Speaker A: No, no, no, no. No worries.
Okay. I want to switch tracks here, but is there anything else you want to say on this piece?
[00:40:24] Speaker B: No, I think this is really fun. I like that you asked me counterpoint questions and want to push back. I think that's really fun.
[00:40:32] Speaker A: Well, dude, the whole like, the whole university thing is such an. It just, it's fascinating to me because like, you know, it's all, it's all in the media now and like all different news channels. Like, like, of course, you know, like all protests and all that kind of stuff. But this, this big storyline about how university education has changed. It's not valid anymore. It's an ideology factory, all these things. It's like, I don't know, man. Has it changed that much since I went to school? I, I don't, I don't think so. But like, I wanted to know, I wanted your thoughts.
[00:41:03] Speaker B: Yeah, I think, you know, it's, there's.
Universities are a really big institution and they're meant to train people for really particular things. And I don't believe enough care has put into, been put into the training programs for like other career pathways. And so I don't see that as a failing of the university. I see that as a failing of sort of like education in a sort of broader sense.
We should have more opportunities for people to be doing these kind of one on one apprenticeship models or train people in a more direct way for interacting with things that are often considered the trades or maybe lower skill or less intellectual work. I think that stuff's really important and it's a good experience.
[00:41:45] Speaker A: Well, what's the fix?
[00:41:49] Speaker B: People have to give money to things other than artificial intelligence to set up situations where people can actually do this. They have to devote time and energy to sort of building up these sort of systems. They have to make these systems priced in a way that's a good value proposition for the people they're targeted for. You know, like you need to be able to sort of see like I have to pay this much upfront or I have to take out this much debt, but I'm going to recoup that over X number of years doing the job that I expect to be doing.
And without that sort of clear value proposition. I don't think anyone will actually buy into that.
[00:42:30] Speaker A: Yeah, I get asked a lot. So I get asked a ton of career path questions from people who've been longtime professionals or people entering in. And the thing that I often say to people is, go out and get as much experience as you can and get education along the way, rather than putting pouring a bunch of education first and then trying to figure out what you do. And not because I don't think education is valuable. I think it's. It's been a huge boon to me. But if you can get into the workforce as quick as you can and like, start making gains, it just helps focus the decisions you make about, like, where. What kind of training you want, where you want to spend your time. I will say that going to university right after high school and not knowing what I wanted to do, I didn't have like a clear picture.
I just wasted a ton of time and a time and a ton of money. And there's nothing wrong with that, honestly. It allowed me to like, focus on playing music. And like, that's. I think the reason I did that is more I wanted to focus on playing like hardcore, but also, like, have some kind of, like, I'm actually doing something with my life fantasy happening, like illusion happening. But I spent a lot of time taking classes. I didn't get any value out. I don't remember at all. I spent a ton of money, like, and then I had to, like, I was buried in debt when I was done university and I had to really work my way out of it. And I was extremely lucky with my career, how it unfolded that I could do that. And I really encourage people, like, get into the workforce and just get into it, get into a job, try some stuff like start building a career and take courses on the side until you get a real sense of what you want to do, then go for it and totally execute. And that's not the only way to do it. There's a lot of ways to do it, but I find that that's the one that allows people to make better choices around where they want to spend their educational money.
[00:44:18] Speaker B: I agree with you so so much. Like, I did my undergrad in physics, and I did it, picked physics because it was hard and mathy and I liked it in high school. And I picked a school based on wanting to move to Philadelphia to go to a place where there was a lot of hardcore going on, really good hardcore scene at the time. And so that's what I wanted to do. And I was, like, studying physics and doing hard work. Did I want to be a physicist?
I didn't really know what being a physicist actually entailed until I was like, a couple years in. And then at that point I realized, no, I don't want to be a physicist. I don't want to do a PhD in this. I don't want to do this as a career. It's just not for me. And so taking that time off to, like you said, play music, tour, work some jobs that were bad, learn about what I don't want as much as what I do want for my life really helped me personally come back to education, do that training to become a teacher, and really say, this is what I want for my life and this is the training I need to do it, and this is what I want out of this experience and really think about it in more of that transactional way. It's like I'm spending time and or money to do this work. What do I need to get out of it in order to make that time and money really worth it for me? The Master's program we have here attracts a lot of also students who have worked for a couple of years or are doing career switching and want to do this sort of quantitative upskilling. And I find that those students often are much more successful in the way that they spend their time here. They know what they want, they know what they need a lot better than other people, and I think that's really a good boon for them. The students who come to our master's program straight out of undergrad are also great. They're brilliant, they're wonderful. I love them very much. But oftentimes they're still feeling it out and they're not as sure about what they want to do.
[00:46:12] Speaker A: All right, so like I said earlier, let's do a switch.
[00:46:16] Speaker B: Yeah, for sure. Sorry about that.
[00:46:18] Speaker A: No, no, I love it.
Tell us about your body of work.
What is it and why?
[00:46:27] Speaker B: Say more about. What do you mean by body of work?
[00:46:29] Speaker A: So what is your focus of your research? And then what's the point of it?
[00:46:34] Speaker B: Yeah, yeah, for sure. So I am primarily trained as what's called a psychometrician. So if you think about taking something like the SATs or the GREs, those tests are sort of designed and scored by people in my field. What it is is trying to measure concepts that you can observe directly. So if I'm like, Aram, I want to know how tall you are, it's really easy to pull out a tape measure. Have you Stand against the wall and make a little mark and measure. And it's like, oh, I can do that. But if I'm saying, aram, I need to know how good you are at math, and I need to know how depressed you are, I can't just crack your skull open and put the tape measure up to the little folds in your brain and be like, oh, he's four inches good at math. And so there needs to be some way that we actually sort of elicit this information.
So what psychometricians do is sometimes it's developing the questions that we ask to try to tease this knowledge out, and sometimes it's the statistical models under the hood that kind of let us go from responses to a score. And we want those scores to have particular properties.
So if you think about, for example, SATs, SATs are scored on a scale from typically 800 to 1600. There's a period of time where it's up to 24, but whatever, it's not important.
It's like this arbitrary number, and you're like, where does this number come from? It's not the number of questions I got correct. It's not the percentage of questions I got correct. And so what we do is sort of go through the data and go through the questions that are asked, and we look at how people respond to them and how people interact with them to say that, hey, this question is harder than this other question, and so it should be worth a little bit more compared to this one. Additionally, we also do all of the work to make sure that when you take the test and you get a certain number of questions or a certain set of questions, and I'm sitting next to you and I get a different set of questions, our scores are comparable to each other.
There's a whole bunch of different sort of, like, linking and equating mechanisms that we sort of use behind the scenes to make sure these inferences are true.
But that's kind of the core of the work that I do. So the way I do it and the way I sort of innovate on this is by looking at including behavioral data collected during computerized tests with this information of just your test responses to maybe improve the way we think about measurement, measure new constructs, or sort of do a better job of measuring things about people while exposing people to less sort of fewer items and just less testing in general.
So some of the work that I've done includes there's a test in Brazil called nm. It's their sort of large national test. And one of the things that's weird about it is everyone takes the same questions, but they're mixed up into booklets in different orders. And as it turns out, people will perform pretty differently year to year or even within the same year, depending on what booklet they're assigned to. Is that this points to the idea that the order you experience questions has some effect on your ability to respond to them.
So I developed a statistical model that accounts for that ordering and gives you scores that are comparable across forms, even though people are experiencing items in a different order.
I've done work looking at response times and sort of what response times tell us about both people's item responses, sort of the constructs we're trying to measure, but also what we can expect from response times as people go through the course of the test. Like, do people speed up as they go through tests? Do people slow down? These sorts of things are also sort of important questions that are not always super well understood.
Some of the work that I'm doing now is related to computer adaptive testing. And so computer adaptive testing is this idea that I can give you fewer questions and get a good measurement of the thing I care about if I pick the next question based on what you've already responded to. So you're getting all my questions right, I'm going to give you harder questions. You're getting all my questions wrong, I'm going to give you easier questions and sort of lets us do a more precise numerical measurement in fewer items.
The problem with computer adaptive testing is that it requires you to have these sort of many thousands of items, banks of questions that you can ask.
Question development is really, really expensive. I think the quote I'd heard once is that one single GRE item costs about $10,000 to develop.
That's from pilot testing, reviews, content reviews, expert reviews, retesting, all that sort of stuff. And so thinking about are there ways we can deliver items to people that let these items sort of exist within these item pools for longer periods of time before they need to be rotated out, making sure that item parameters we sort of. We calibrate difficulty parameters for every single item, that they're well calibrated relative to each other. And sort of algorithms we can use for both item selection and sort of parameter updates that let us do this sort of measurement task a little bit better.
I have a student now that I've been working on some data with the National Assessment for Educational Progress. It's called naep. So if you're from the US it's the thing that produces the Nation's report card, we have data on what students do while they're interacting with different items.
Do they click on the calculator? What do they type into the calculator? Do they use assistive tools like definitions or the screen readers or things like that? And so what we've been doing is looking at these sort of items that they respond to and developing these sort of behavioral profiles based on tool usage and response patterns.
Not correct or incorrect, but looking at sort of like, what are students physically doing while they're interacting with this computer software and then trying to understand how those response profiles evolve over the course of a test and then seeing who falls into these. So looking at potentially sort of specific learning disabilities that students might have, or different sorts of familiarity with English, or are they high ability, low ability? And so I think these sorts of questions have a lot of implications for sort of like equitable testing, reducing bias in testing, and making sure that we produce more comparable scores from test to test. And so there's some of the things that I do, I've done work looking at keystroke logs of students writing essays for sort of high school equivalency exams and trying to understand their sort of essay construction strategies. And then every once in a while, too, I also do a lot of contracting with other either companies or other researchers who just need sort of statistical analysis on the works that's sort of important to them. So I've been working with a group that does actually online intensive outpatient group therapy and sort of trying to understand how they sort of measure outcomes and sort of how they measure things at intake to sort of make sure that people are getting routed to the correct groups for them. And so that's the kind of stuff that I do day to day.
[00:53:51] Speaker A: So this field of work, why, what's the point of this? And I'm asking you about a huge field, but, you know, as you're speaking, I was like, hell, yeah, that's super cool. But I'm also, I deal with a lot of data in my work, and everything in my work is about predictability models and how to alter outcomes and all these things. So what you're saying works with. Works with my thinking. But someone listening to this would be like, why? Why the hell would you do any of that?
[00:54:20] Speaker B: Totally. I have like a canned response to that. And that's, you know, all good science is predicated on good measurement. If we don't have good estimates of the things that we care about, predictive models downstream don't work as well as they should. It's harder to derive insights from the stuff you're learning about people.
You might be trying to make generalizations about groups of people based on how they answer 10 math questions that you've written. But if you've done a bad job of writing them or they're not measuring the thing you actually care about measuring, you're going to be making decisions with much noisier information than you really need to. And so it's a weird.
It's definitely a strange field that I arrived at kind of accidentally, if that makes sense.
I originally went to grad school to study physics education because that's sort of. I was teaching physics, that's what I did. And this is a thing I care about.
My first project that I worked on with one of my advisors was a physics assessment project. And so we were trying to understand and measure misconceptions using physics questions. And I was like, oh, the actual machinery and design of how we actually tried to measure this very specific, narrow construct of one, misconceptions. And two, the degree to which teaching can reduce these misconceptions is way more interesting to me than the sort of classroom practice of teaching physics. And at that point I just sort of pivoted.
It's also helpful that a lot of my quantitative skills really carried over into this realm. But for me, I think of my work as being really sort of foundational to let other people do their jobs much better. Other researchers understand their data much better, other clinicians sort of make better decisions.
That sort of stuff, I think, is really quite important. And like, I don't think of myself as a.
[00:56:17] Speaker A: But why? Like, what's the end result? So, like, you're creating something that's foundational so that another researcher or a clinician could.
Could make better decisions based on the data them getting, like essentially correct data. So what's that? Like, what's the output of all that?
[00:56:33] Speaker B: Sure. So let me give an example of something I worked on recently with a colleague and a student.
So you have this sort of situation where sometimes there are, you know, sort of these long screeners or forms, you know, like depression screeners. They might have like 30 questions on it, but you might be in a situation where you don't have time to ask a person 30 questions. So you need to sort of decide what are the best 10 that are going to get me most of the way there. And so we developed an algorithm that does this sort of item selection, but the purpose of it is actually to reproduce clinician diagnoses. And so we Took an alcohol use disorder screener that had 20 items on it and we said, okay, we want to now pick the subset of items that best reproduces a clinician diagnosis of alcohol use disorder. One of the things that we found in doing this is that we can produce versions of this particular form that are seven items long that reproduce clinician diagnoses better than the full 20 items because some of them sort of add more noise than signal than, you know, to the kind of model that you're doing.
And so the output I think is a set of practices, a set of best practices, a set of principles, tools that people can use, you know, shortened forms or screeners or tests that other people can use in their work. And so I really think of my work as tool building and I think of my work as providing guidance for other researchers and practitioners to do their work better.
[00:58:11] Speaker A: Can I add the human component to that?
[00:58:14] Speaker B: For sure.
[00:58:15] Speaker A: So when I was working in addiction and mental health, we had these really long intake forms. So when someone would come in to see a therapist, they would have to do an intake just in the lobby. And then when you're actually meeting with someone, you have multi, multi page assessment to go through. And it would have all of these different measurements and a lot of those measurements would actually kind of be measuring almost the same thing.
And we would see you're dealing with street entrenched youth, so at risk youth who if they're in there and they have a bad experience, they are never coming back or they have been mandated because they're on probation and they're already viewing you as an arm of, of the law essentially, rather than being someone there to help them. So again, they have a bad experience, they're just going to think you're there as a consequence, or you have people with complex mental health needs or people who are under house populations.
You do not have minutes, let alone seconds to burn with really bulky assessments. And all it's doing is creating a barrier to service for people. Nobody wants to see a therapist and have to do like 30 to 40 minutes of paperwork before you actually get to talk about how you're doing.
And this was like a real issue, like 15 years ago when I was a therapist and there was always this kind of conversation, how do we streamline this? How do we do it in a way where we're hitting what we need to hit to make sure that our funding is intact, but also that we're creating this better experience for our clients? So the human component was that is anything if you gave me a form with 10 questions and then you can give me a form with eight, I'd be like, hell, yeah, that's great. Give me that. If you can give me one that has six questions, fantastic. If you can give me one that actually combines four different forms into one that's like one form that's like shorter and more precise. You just saved me time. And not only did you save me time, you saved me the ability of potentially creating a situation where this client just disappears and never comes back. So the human component of that is making it about the person rather than about getting a bunch of paperwork done 100%.
[01:00:19] Speaker B: And that's exactly sort of the work that I'm doing now with this firm that I'm currently consulting with.
They're dealing with young people who have had some sort of traumatic issue and are getting put into these sort of group outpatient, online or virtual counseling sessions. And so patient dropout is huge for them. Making sure that people are ending up in the right places is huge for them. And if you have to click 40 minutes of buttons on a website before you can get access to just talking to another human being, there's no way in hell you're going to stick with this. And so that's a huge barrier to access for people. And so some of the things we're doing there are like looking at conversational transcripts and can we maybe pull some of the same information that we would pull from this sort of screener out of just conversational transcripts so it feels like a more human to human interaction?
Can we shorten this intake? Can we do better monitoring along the way in a way that's not a burden for respondents so that we can make sure that they're improving or getting the support that they need and being able to quite literally just help people live better lives. And I think that sort of work is so, so, so important. And I think it's one of the sort of core contributions of my entire field.
[01:01:43] Speaker A: It's interesting you say that because I'd add one more piece on top of that is if you're meeting with someone who is perhaps at risk or have had a bunch of like, they've been justice involved, they've had about a bunch of bad experiences, or if they've had a really bad experience with the therapist before, so they're like they've been burnt.
People can be reticent to answer a lot of questions and can be quite understandably so defensive about sharing this stuff. So again, like, if you've got a bunch of burdens burdensome assessments. And for people who wouldn't, who would know, it's like for therapists, at least in Canada and at least in bc, your paperwork is like central for you to be able to keep funding. And if you're seeing like being non compliant with types of paperwork and those things, you can have funding pulled, like, which is crazy, like really intense. So it's not that even not for profits or just your general therapist wants to be doing all these forms. It's actually just a part of, of being able to stay compliant, recognizing that in staying compliant you could actually be creating a barrier. So there's a lot of tension around this topic. So I think the work that you're doing is ultra valuable.
[01:02:51] Speaker B: It's also really important to remember that while we need data from assessment, any time that a sort of either clinician or teacher is spending doing assessments is time that they're not spending teaching working with people one on one actually sort of doing the core missions of their work. And so the extent to which we can reduce testing time while still providing similar levels of sort of assessment information, I think just helps people one do their jobs, but also just do them better.
[01:03:25] Speaker A: All right, so what is something that you're working on right now from like a research perspective that you're super passionate about and like what's the, the intended output of it?
[01:03:35] Speaker B: Okay, super good question. So I started writing a grant with one of my colleagues. She was actually my faculty mentor when I first came here, and she's really interested in early language acquisition in pre kindergarten children.
So what we're doing is we wrote a grant to do some sort of secondary data analysis and look at transcripts of interactions between children and their parents to try to measure both expressive vocabulary, sort of like how much they can say, but also receptive vocabulary, how much they can sort of understand.
And so what we're trying to do is make this pitch that receptive vocabulary, especially at this age, is maybe a better indicator of risk for sort of later language or later delayed language development. And so this is sort of like the work that we're doing. And so I'm super excited about this because one, it's kind of a slightly different field. I get to work with a slightly different type of data. I do like working with this sort of Sanskrit freeform text data in addition to these sort of more traditional assessments. But when we think about what the work projects are going to be, there's sort of three, right? Like, one is just the core argument that maybe the way we try to measure Vocabulary or sort of language acquisition now is not the most fruitful. And because we have different technologies now than we had in the past, we can do it in a better way. And this will help us understand children better and maybe find earlier places to intervene to sort of help them in their language development.
The second, I think, is the part that's really associated with how we're evaluated. And that's just going to be sort of academic papers. So writing academic papers about that, publishing in journals about that, I think is sort of kind of the core work product when you talk about this stuff with other academics.
But in addition to sort of that, to go back to that first one for a second, I think one of the things we want to do is really do sort of training, get people trained on either protocols or software or analysis techniques to kind of do this work if it does pan out. And I think that actually wraps into what the third work product would be, which is essentially another grant to study this further in a larger population to maybe see if we can take these sort of other vocabulary measures and try to better understand how they might evolve in children over time, what they might be indicative of, what other sorts of behavioral measures can they be combined with to better understand this early language acquisition and then hopefully have a place where quite literally more children are reading and speaking fluently at younger ages, which I think is a net good.
[01:06:19] Speaker A: Yeah. So a bigger question, if you think about what are some of the advancements, we could be in education, or it could be in social work, or it could just be in society in general. What are some of the advancements that we take for granted now?
But that came as a result of the type of work that you or your colleagues would do.
[01:06:43] Speaker B: So one of them, I think, is computerized testing.
I think about the way people interact with tests. Tests can be much shorter now than they used to be. So there was a period of time where, say, onto the GRE was entirely paper based. So you sat down in a room with a whole bunch of other people in an auditorium and you filled in little bubble sheets.
Nowadays it's administered on a computer. And so the test can actually be much shorter. And so this is going to reduce both time burden for individuals. It's going to reduce test fatigue. So it allows individuals to sort of perform at a higher level for a longer period of time.
In theory, it could reduce cost, but that's someone else's decision to make, not necessarily mine. And so sort of the move from paper based to computerized testing, I think was really huge.
It also opened us up for sort of these adaptive algorithms collecting these other forms of data. That stuff's all super duper important.
Additionally, I think also we've learned a lot about intelligence testing, and I think one of the big outputs of that is that it's maybe less useful than people originally thought it was when they originally developed a lot of this sort of like IQ type stuff. One thing that's always worth talking about is my field was quite literally developed by a pile of eugenicists and scientific racists who were like, we need to find quantitative measures for the superiority of some people over others.
And that's horrible.
As it turns out, the statistical tools they developed and the methods they put together tell a quite different story than the one that they were hoping to tell. And I think that's actually one of the real triumphs of my field, the fact that we've learned about the limitations of things that we thought we were measuring. We've learned about the limitations of the degree to which something like IQ can predict sort of like future outcomes for individuals. We've learned about the limitations of our ability to measure constructs in diverse populations because we don't account for things that like, you know, cultural differences, linguistic differences that are really, really important. And so I think a lot of the assessment we've done over time has, has improved because of the bad ideas that my field kind of started with.
[01:09:27] Speaker A: Stay tuned for part two of this interview. Next week on One Step Beyond.
[01:09:31] Speaker B: One step.
[01:09:34] Speaker A: One step, One step beyond.
