Hi welcome everyone so for today the Chief Evaluation Office is celebrating 10 years of
the Clearinghouse for Labor Evaluation and Research. We would like to welcome you to
today's webinar, "How to use CLEAR for decision making." CLEAR is funded through the Department
of Labor's Chief Evaluation Office and CLEAR is supported by various independent contractors.
For today to submit content related questions select MSG Webex A from the
drop down menu in the chat panel and enter your question in the chat box and send.
The event is being recorded. By continuing to participate you are consenting to being recorded.
I will now turn it over to Megan Lizik. Ms. Lizik is with DOL's Chief Evaluation Office
and CLEAR's team lead. She has nearly 20 years of analytical and evidence-building experience
in both the private and public sectors and currently oversees a portfolio of behavioral
science, evidence translation, and other labor related research and evaluation projects for DOL. Megan?
Great thank you so much Brittany. Hello and welcome to all of you. It's really great to see
so many of you joining us today for one of the topics that's near and dear to my heart
really getting into the nitty-gritty on how to use CLEAR in your own decision making
on a day-to-day basis. So to kick us off I wanted to invite our deputy director Lauren Damme
who is here to share some opening thoughts with us. Lauren is responsible for helping
oversee CEO's evaluation research portfolio staff and office functions and before joining
CEO in 2019 she served as the senior evaluation research advisor in the Department's Bureau of
International labor Affairs Office of Child Labor, Forced Labor, and Human Trafficking.
She has over 18 years of experience in quantitative and qualitative research and evaluation
primarily in International Development and has worked in around 30 countries. Lauren.
Thanks so much Megan I didn't know you were going to do a whole introduction. Thank you. So thanks
so much for having me and welcome everyone I'm really excited to be here today. Today's session is
going to focus on how to use CLEAR for decision making and by the end of our time together you
will learn how to use clarifying evidence that you can use right away including to quickly gain
an understanding of the state of the field on a particular topic and we're also going to hear from
peers from workforce agencies in the field about how they're already using CLEAR. But first I do
want to thank Megan and the entire CLEAR team at CEO as well as our contracting partners for putting
on these public events. They do just an incredible job relative to our time and resources and it's
amazing that we are celebrating the 10-year anniversary of CLEAR this year and I know that
there are some others I saw on who are joining us today from outside of DOL who were involved
in some of the inception activities of CLEAR. So I would also like to thank our presenters for
agreeing to share their stories about how they've used CLEAR. I've heard from some of them before
including Ben for example and I think that their experiences that they will share really bring to
life how people can leverage CLEAR to improve the way that we serve and support workers. So I think
this could be a great session and I'm looking forward to learning something also but before we
get going I wanted to reflect on the importance and the challenges of using research evidence
and decision making which is really the reason why DOL created CLEAR. So many of you on this call have
probably heard of or at this event have heard of the Foundations for Evidence-Based Policy Making
Act of 2019 or The Evidence Act and you know that there has been a huge growth in recent decades of
federal, state, and local efforts to bring science to bear and program and policy decision-making.
However, I think all of us also know that there are real challenges to using research and evaluations
in your everyday decision-making so it can be really hard to find where that evidence lives
or recall when you need it it can also just be hard to keep up given the proliferation
of research and evaluation studies so earlier this week for example I heard from a leading scholar on
synthesis research based at Monash University in Australia who works on synthesis work that similar
to that which CLEAR does and they said that in their work they found that research output is
doubling something like every 10 or 15 years like exponential growth and adding to that volume issue
with keeping up is just that going deep into any particular single study can also be a daunting
prospect for busy practitioners, policymakers, and others like yourselves who are attempting to make
or who are interested in making evidence-informed decisions. So CLEAR
really helps address this problem. The team sits through tens of thousands of research studies
to find and review the most credible ones that examine effective and promising programs across
a number of labor-related areas. CLEAR does not just conduct literature reviews and that's a
question that I know that I've gotten before the team it's not just a literature review the
team looks at every single relevant report in the English language during the specific time period.
It's extremely systematic and then CLEAR provides you with a plain language summary of the studies
that they have found meets specific criteria to help you understand more about what works as well
as how where and for whom. So the other last thing I'll mention about CLEAR for those of
you who are not familiar with it is that CLEAR assigns ratings to causal studies in particular in
order so where we're trying to make a connection between a program or policy and an effect that it
has had on something for example increasing a training program increasing the wages of workers
in order that those causal ratings or the ratings for causal studies help
you understand how confident you can be in the findings of those studies and to date I think
CLEAR has conducted more than 25 evidence reviews and summarized nearly 1300 studies but again to
whittle down to those credible studies the team has reviewed you know thousands and thousands of
papers and reports. So you can really trust CLEAR and other federal clearinghouses to find research
and evaluation evidence that you need when you need it and to that end as I mentioned before
we're going to hear from some workforce leaders about how they use CLEAR and we hope that
inspires some ideas on how you can use CLEAR to support your own work and that may also include
identifying where there are still gaps in our knowledge. So I'm really excited about today
CLEAR fills a very big need and hunger for accessible reliable information when you need
it and I hope that you find the session really helpful. Let me turn it back over to you, Megan,
to introduce the presenters and panelists for today.
Great, thank you so much, Lauren.
So without further ado, I'll share a little bit more about who you'll be hearing from. So first
we have some of our presenters from one of our CLEAR contract teams. Chris Weiss is
a principal associate at Abt Associates and he is the principal investigator for Abt's work managing
CLEAR. We also have Andrew Clarkwest with us and he is a principal associate at Abt Associates as
well and he directs Abt's CLEAR contract with DOL. And then finally we have Rhaia Hull and she is an
analyst at Abt Associates. She oversees the study review process and Abt's team of study reviewers.
So welcome to all of you. And then I'd also like to introduce our panelists for today if we could
go to the next slide, thank you. We couldn't be more excited to be joined by some workforce
leaders Ben Holquist and Lisa Salazar to hear more about their experience using CLEAR. Ben has been
a leading partner in developing much of the Texas Workforce Commission's evidence and outcomes-based
grant-making processes and specifically he's been leveraging CLEAR as part of those efforts.
He's been working on Texas's evidence-based grant-making project since 2019. And Lisa is
the first Executive Director of the City of Los Angeles's Youth Development Department and prior to
working there Ms. Salazar served as the Director of Workforce Development and Economic Opportunity
to Los Angeles Mayor Eric Garcetti, responsible for leading the development of policies programs
and partnerships to advance the mayor's agenda on workforce development, educational career pathways,
and youth development citywide since 2000. She has spent more than two decades in leadership
positions working for the city developing and implementing public policies and programs on
youth and adult employment job training and skills development, career pathways, and the future of work.
So welcome to our panelists. Welcome to our presenters. We're thrilled to have you with us.
Next slide. So before we get going I wanted to share a little bit more about some
background of DOL's Chief Evaluation Office. Our office was established in 2010 as an independent
Departmental-level evaluation shop here at DOL. We coordinate manage and implement the
Department's evaluation program, working closely with colleagues in about 15 DOL agencies to help
address their research questions and support their learning and continuous improvement priorities
by conducting evaluation studies, data analysis, and evaluation technical assistance. We do most
of our work through contracts like the ones we have for CLEAR, and currently we have about 60
active projects that we're working on with DOL agencies. You can learn more about those as well
as view our past reports by clicking at some of the links here on the bottom of our screen.
Next slide. So as we get going we'd like to kind of hear from you sort of who you are where you're
from and your experience with CLEAR so we've got a couple of polls that Brittni's going to pull
up for us, if you wouldn't mind launching that Brittni, and we're gonna spend just
a couple of seconds hearing what organization you're from you can see it there in the chat.
And when you have the results, Brittni, you can go ahead and share those with us
Okay and then we have a second poll
and this poll is really to understand what
your experience is with CLEAR so we want to know a little bit more about
whether this is your first experience hearing about CLEAR, whether you've used it a few times
but it's been a while you could use a refresher, whether you use it regularly so whenever you're
looking for this kind of research and evaluation evidence to inform your work, or if you're really
kind of not sure yet and you're waiting to see more from today so take a few minutes and if you
could look in the the polling on the right hand side of your screen you'll see our questions there.
And Brittany do you have those results for us?
They're loading. 10 more seconds.
OK. So while there are loading,
we can go ahead and go to the next slide I think
we want to start by sharing a little bit about the goals for today's webinar.
Thank you. So really our goal is to try to help share with you some insights about how
to navigate CLEAR, how to find the key information that you are looking for when you need it. We will
also offer multiple examples of how you might use it, so we're going to walk through one theoretical
one from a community college perspective and then we'll have our mini panelists offer their
insights about how they use CLEAR, and then I'll go through a little bit at the end of our time
together today to understand where to learn or to look to learn more. Next slide. So this is a little
bit of an overview of our agenda. We're going to do a short recap of some concepts from the 101
webinar that we held a couple of weeks ago just to kind of provide an overview and grounding for
everybody joining today regardless of whether you heard that webinar just about kind of what CLEAR
is and does and how we do it. Then we're going to launch into the content of today's webinar,
hear from our panelists, do the resource recap, and then have a few closing poll questions and
then plenty of time for discussion at the end so please do hang on with us. As you have questions
feel free to put them in the chat and we will revisit them at the end of the webinar. Next slide.
Great, so now I'd like to turn things over to Chris to recap all about CLEAR 101. Chris?
Hi, thank you Megan. It's a pleasure to be with you all today. If you could go to the next slide, please.
Great there are three parts of the prior webinar that I want to briefly highlight about
CLEAR by way of setup. You see them listed here: CLEAR's mission, the evidence review process, and
study summary icons. We'll have more to say about each of these and use these as examples in part
of the presentation later on but at the outset wanted to cover or if you joined for the previous
presentation recover the some of these critical elements of it. If you go to the next slide, please?
Thanks. First, the mission. CLEAR's mission is listed here: "To make research on labor
topics more accessible to practitioners, policymakers, researchers, and the public so
that it can inform decisions about labor policies and programs." That's a broad goal,
and there's lots of things lots of resources on CLEAR that will help you to achieve those
goals. CLEAR is a place where you could go to see for example how much research is
out there. Is there a great deal known about a particular topic or or not much or very little?
You can also learn what the research says about particular strategies, policies, or programs, and
if they're reaching their goals and the third is out how confident we can be in the findings
of the research in a particular programmatic area. If you could go to the next slide, please.
Here's a brief, very brief, overview of the evidence review process but let me
walk you through a little bit of this briefly. CLEAR conducts systematic evidence review on
labor topics including studies funded by the Department of Labor as well as other studies
from published reports. Then, we summarize the methodologies, findings, and policy or program
implications of these studies. CLEAR's topics are determined by the DOL's Chief Evaluation Office
that works often in consultation with DOL agencies, CLEAR contractor project staff,
technical advisors, and topic area subject matter experts. You see the different stages here listed
we can talk in greater depth about these or we'd invite you to watch the webinar watch
the recording of the webinar when it's available. And as Lauren mentioned at the outset, CLEAR
has completed evidence reviews of nearly 25 labor topics and currently houses summaries
of almost 1300 studies with more studies being added continuously as we continue to grow and
grow in the number of topics and studies that have been reviewed. Next slide, please.
Third point is a brief, brief recap of the study summary icons. You see two different
dimensions of study quality or excuse me of icons that are presented that you'll see on CLEAR
study pages. The first of these are the study evidence ratings or study quality icons.
CLEAR's causal evidence ratings provide information about the quality of
the causal evidence in a study and there are three different levels you see here
presented in that top... the kind of like half-moon dial. High, moderate, and low.
For studies that receive a High rating, we can feel confident that the findings reported
in the study are due to the intervention or program understudy and not to other factors.
For Moderate rated studies, we can be somewhat confident that the findings are due to the program
or intervention examined in the study and not to other factors, but it may be that other factors
contributed as well. Finally for Low rated studies, we cannot be confident that the findings are due
to the program or intervention that other factors are likely to have contributed to the observed
outcomes. You can find more information explaining this and the process by which the criteria by
which these ratings are applied on the CLEAR website. The second part of the panel on the
right displays Effectiveness Icons. These icons provide a quick overview of the study's impact
findings as shown on the slide there are five, four, excuse me, designations: Favorable, Mixed, Unfavorable,
and None. These designations are provided for each outcome domain that the study examined.
I recognize that if you're new you didn't see the previous or if you're
a relative novice user to CLEAR, many of the words I have just said probably are
the words themselves may not be may be familiar but the meaning and the the details of how of what
they mean in the context of CLEAR may not be. But it's my pleasure now at this point to pass the
mic over to Andrew Clarkwest, who will discuss how the resources of CLEAR can be used to
inform decision-making. Andrew?
Right. Thanks a lot, Chris. I'm happy to be here to have a chance to
talk about CLEAR's resources that can support evidence-informed decision-making. Next slide.
So if you're someone who's interested in using evidence to support decision making
around some policy or program question, which I assume applies to a lot of us who are in this
meeting right now, in theory, it sounds fairly straightforward. You go out and find research
evidence, see what it says, then incorporate those insights into your decision-making.
But as Lauren alluded to, and as indicated on the next slide, in practice it tends to be slightly
more complicated than that. When you're looking into research onto a topic you may depending
on the subject run into a lot of studies or alternatively you may find it hard to find a lot
of the research on a topic since studies may have been published in a variety of different places.
Beyond that the studies on a topic may differ in a lot of ways. They may have used different
methods to look at interventions or outcomes. Those interventions or outcomes may also vary somewhat
and the study may have looked at them in contexts that are different from yours or different from
one another in terms of their geography or the economy at the time or in the local place.
They may have also served different types of populations. And even worse, when you look at the
studies you may see that their findings aren't all congruent with one another or at least
they may not seem to be congruent. So that leads to a lot of heterogeneity that makes it challenging
to get insights out of an evidence base to make sense of what exactly does this mean
and what does it mean not just in generally but also specifically for me. Next slide
And this slide here shows three types of challenges that you'll run into or three
types of questions that are key for someone who's trying to make sense of an evidence base
and figure out how to apply it to their decision-making. The first one is substance. That's just
what have the studies actually found, which isn't always as easy to tell as you might expect.
The second is credibility, and this gets into very technical issues as
Chris mentioned particularly when looking at causal studies the studies can vary a lot in
how confident we can be that there are study that our findings really do reflect the impact
of the intervention as opposed to reflecting some other factor so if I look at a finding in a study
can I really believe it? I mean studies vary in there... in how they were conducted in ways that
can be very technical and can be difficult to sort out for a non-specialist audience.
A third issue is applicability, which gets at, in colloquial terms, whether your mileage
may differ. That is, at an initial level, wanting to know or that's important to know is enough
about the intervention to tell if it seems like something that you could implement reasonably
well in your circumstances. And beyond that when looking at research findings you may want to
understand the time, place, and circumstances where an intervention has been studied
and whether those differ from yours in ways that might lead results that you get from implementing
the approach to differ from the results that were found in prior research. Next slide.
So CLEAR not only brings together research on labor-related topics in a single clearinghouse
it then goes on to provide help with each of those three questions that
are key to making sense of the evidence base. You'll see the three icons on the
left of the slide here reappear throughout today's presentation as we walk through how
to find information for decision-making in CLEAR's evidence resources. Next slide.
And we'll group those evidence resources into three types as we're walking through it.
That'll be signaled by the colored boxes that you see here,
which constitute a sort of navigation key. The boxes will appear at the top right
of the subsequent slides as we cover each of those kinds of resources. Next.
So to just walk through each of the three briefly, profile summaries provide details
on the individual studies that CLEAR has reviewed, so they provide study-specific information
and usually studies are reviewed as part of a targeted search that's targeted by topic
or by time. And then the second box, an evidence review is a collection of research on a topic
so a collection of studies that may be defined by looking at a similar population
like low-income adults or older workers or their strategies to improve a particular outcome like
literacy, or re-employment, or equitable post-COVID recovery, or potentially by intervention type so
CLEAR's interview has assessed research on apprenticeship and work-based training approaches
and remote service delivery approaches of various kinds. Finally, the evidence syntheses that clear
creates are short reports that describe findings about what works from the body of evidence on a
topic and these syntheses also note gaps where more evidence is needed. Next slide.
So when you come to CLEAR, you'll typically first hit the the front page here. You'll see at
the top there's a set of CLEAR navigation tabs in a navigation bar and if we go to the next slide
we'll see that we zoom in to make the evidence of the navigation bar a bit more readable for
you and the arrows highlight three elements in that bar. The first arrow points to topic areas,
clicking on that will provide a drop down menu that you can browse of existing clear topic areas.
The second arrow points to the search for studies page and that's a place that you'll often likely
want to start when you're looking for research evidence, and we're going to in subsequent slides
in this presentation going to walk through the search function at a couple different points.
The third arrow points to the "About CLEAR" element of the navigation menu. We aren't
going to walk through that directly in today's presentation but for those of
you who are curious about all of the details of CLEAR's procedures and how it rates studies
the kinds of things that Chris was talking about, you can find that documentation there
which you can peruse to your heart's content whenever you have some free time. Next slide.
So Megan mentioned we're going to walk through an example of a hypothetical,
a hypothetical example of using CLEAR to inform a particular program question
and in this case the question is "How can we find evidence on strategies to increase
trainee persistence?" So suppose in this case that we have a user named Teri who
is a community college administrator and they're administering a training program
or running a training program where they're having issues with students not completing
the program and want to know how can we improve low training program persistence and completion?
So in that case thinking okay we have some thoughts would it be useful
to know what's been tried elsewhere and in particular what's been studied before?
Sp we're going to walk through how she might use CLEAR to identify research-based insights
on interventions that improve program persistence and completion. Next slide.
So as I mentioned, she might start with the search for studies
function so she would click on search for studies link on the navigation bar which
would bring her as we see in the next slide to a page that looks like this I realize that the
words are a little small but you can see the at the top you have a search by keyword
and then there are a number of ways that you can filter studies to try to get to the kind
of studies that are for particular interest. So you could filter by the date that the
study was published, by the outcomes that were examined, by the target population,
by the geographic setting, or by the evidence rating, which, as Chris mentioned, that's the
ratings of this credibility of the findings, so if you want to focus on only the most credible High
ratings or you can also look at Moderate or Low rated depending on your particular interest and
how much evidence actually happens to be available. So in the next slide we'll see a hypothetical search
suppose that Teri comes in, searches by training completion because that's what she's interested in,
and you'll see that the search for studies will turn up not just relevant studies but
also research syntheses that summarize evidence from across the research literature on the topic
and in this case the first search result is a Community College Synthesis which seems
very relevant given where Teri is coming from as a community college administrator.
You'll see that in the words there or in the description that the findings mention
educational outcomes, persistence outcomes, which is what we're interested in, so let's click on
the Community College Synthesis link there to see more
and the next slide shows you what happens if you click on that link you'll see a
page like this there's a landing page for every research synthesis at CLEAR
which provides a bit of information from the synthesis. It also provides
links to two PDFs in this case there's a synthesis and a supplement. Next slide.
The synthesis provides an overview of findings from CLEAR's review of the research in the area
and then the supplement provides additional study-specific detail on the studies that
were viewed reviewed on the topic. Now let's click the link to the research synthesis
which will bring us to a PDF which at the top looks like this at the very beginning it talks
the the synthesis it'll describe the scope of what was covered in the review and then below
that we'll start listing the interventions that were examined by literature in this topic area.
On the next slide, we can see a little bit of information from that introduction that
describes the scope. So you can see that it describes the years of
research that were covered, in this case 1994 to 2019. And it just notes the fact that
this synthesis includes only High and Moderate rated studies so it excludes Low rated studies.
And syntheses will vary in cases where there's a smaller research literature, we
may also include Low rated studies. But when there's a large research literature, we have
the luxury of focusing on the more highly rated studies, the more credible studies.
We'll see in the next slide that if you scroll down a research synthesis and
you can see a full table of the interventions that were examined in the topic area.
Some of these are whole program models like career pathways and work-based training.
That's beyond what Teri is looking for, but if we're looking... interested in things
we can do to improve completion in existing training programs rather than implementing
new kinds of training programs, then the four at the top look potentially relevant.
And if we scroll further down the table, we can see a summary of what
research is available and has found on each of those four topics. Next slide.
So when evidence synthesis will include tables that look like this one that concisely summarize
what studies of an intervention have found each row or a particular... sorry each row is for
a particular intervention and the cells in a row count the study findings. Next slide, please.
The findings are provided by whatever outcome categories the topic area focuses on.
In the case of a community college synthesis, those are first, education and skill gains, second,
earnings and wages, and third, employment. Each vertical panel in the table presents findings
for a given outcome domain and you see the individual cells are color coded. The counts
in the green cells note how many studies of an intervention found evidence of favorable impacts
or outcomes in the outcome domain. Counts in the red cells note how many studies found unfavorable
impacts. Counts in gray show how many studies had null findings, that is there weren't any studies
that were statistically significant or weren't any findings that were statistically significant.
And then the counts in the yellow show how many studies has mixed findings which means
there were some favorable and some unfavorable findings in that domain.
We can see in the next slide that when we look at the education and skill gains panel
that the four interventions we're looking at here have many studies that had
findings that had favorable impacts. And in the next slide,
if we look at accelerated learning in particular, we can that may pique
your interest to dig into more deeply. It does seem to have a large body of favorable findings.
In this case, 10 out of the 14 studies found favorable effects on educational skill gains.
And so if we want to find a little bit more information on accelerated
learning and what the findings of these studies were the research synthesis will
provide a little more narrative information if we scroll down, so we go to the next slide,
we can see that the research synthesis notes specifically that accelerated learning
interventions were found to have increased rates of course enrollment and completion
as well as having improved rates of degree or certificate completion. That sounds like the
kind of thing that Teri's looking for so that's promising. The synthesis also notes that studies
of accelerated learning interventions typically serve individuals in
developmental learning courses who traditionally have lower rates of academic persistence and
degree completion and that sounds like it may align with the kinds of students that Teri
has in mind to support. So that's all promising, but suppose we want to dig into specific studies
of accelerated learning interventions. If we want to do that we can go back to
the landing page for the Community College Synthesis, which we can see on the next slide
is the second link right there so we click on that we'll be taken
to a document that looks like what we see in the next slide.
Typically... the supplement will organize studies by the intervention
type so in this case we've gone down to the part of the supplement that has the
list studies of accelerated learning we can see here the first study in that list
and as you see here the supplement provides study-specific information including a full
citation, the findings by outcome domain, the causal evidence rating of the study, and also
a link to the full study profile so you can see additional information on that study.
And at this point, I am going to turn things over to Rhaia to talk a little
more about finding individual studies of interest.
Thank you, Andrew. Can we go to the next slide, please?
You could get straight to relevant study profiles through the links in the supplement
as Andrew noted but I also want to go back to the search function available through
CLEAR to walk through some of the other features of find studies of interest.
And because clear review studies on an ongoing basis it's always worth checking whether new
studies have been reviewed that have not yet been incorporated into a synthesis. Next slide, please.
Here let's search specifically for accelerated learning. Note that the images to the right is a
magnification of the circled part of the filtering menu to make it a little bit easier to read. Next slide, please.
In addition to searching by the name of the intervention let's filter by the
outcome domain of interest, education gains, and include only High or Moderate rated studies to
focus on the most credible studies. Next slide, please. As you can see on the right side of the
screen, this pulls up 16 results. You can save them by using the export results function.
Next slide. Thanks, the export studies function produces a spreadsheet file with a row for each
item in the search result, including the sole citation, all of the characteristics that you
can filter on, and links to the full text of the study if they're available. Next slide, please.
But let's go back to the search results and click on the first
result to see CLEAR's profile page for that study. Next slide.
Here we can see the study's profile page the box at the top left summarizes the study's findings
by outcome domain. In this case, the study only examined outcomes within one outcome domain
defined by CLEAR. Findings are characterized as favorable which again means at least some
statistically significant favorable outcomes and no unfavorable outcomes and the study received a
moderate causal evidence rating. Again, that means CLEAR assesses that the studies findings likely
reflect the causal impact of the intervention although it may reflect some other factors as
well. The blue button provides a link to the full text if it's available to help you get additional
study details that you might want beyond what's available in the study profile. Next slide, please.
Further down the profile, we find more details of the intervention the study that can help us
understand the applicability. The sections below highlight real information that gives you a sense
of the intervention without reading the full article. So again, for example, this intervention served
college students from summer 1999 to fall 2010 and condensed developmental reading and writing
English courses from two semesters into one to create the accelerated reading program.
So the treatment took eight credits worth of classes over the course of one semester
whereas the control group attend the classes as they were originally designed four credits
worth of classes in one semester. Outcomes were measured at one year after the intervention
started however you also learn the important caveat that the self-selected students...
the students self-selected to the accelerated program or the business as usual program. Next slide, please.
Further down in just a few bullets you learn the study's key findings. Here we
learn that the students who were in the accelerated program were significantly
more likely to enroll and complete the college courses and increase
credit accumulation with higher level GPAs than those in the comparison group.
However, the study profiles have additional layers of detail to help make sense of studies
findings. Here we are reminded that the treatment group self-selected into the accelerated program
meaning that the estimated impacts seen between the treatment and control may have at least in
part have something to do with that. In other words, students who self-selected into the accelerated
program may already be more confident in their skills or be more motivated to get through their
degree program faster meaning that the students who self-selected into the treatment might have
had higher enrollment and completion rates with or without the intervention. Next slide, please.
So as we think about what Teri the Community College administrator might do, through her initial
search for CLEAR, she has found lots of research on multiple strategies from the synthesis report
she has learned about linked learning communities, paid performance incentives, and
the intervention we focus on in just now, the accelerated learning program. These
interventions have been evaluated in studies that CLEAR has reviewed and found to be credible
and those studies have frequently found those interventions to be effective increasing students
course completion rates. So each seems like a promising intervention for further exploration.
So from here there are a few steps that Teri may want to take if she wants to move forward
with implementing an intervention such as the accelerated learning course. First, she could look
more closely at individual studies and other publicly available resources to learn
more details about what it might take to design and implement an accelerated learning course.
For example, this may include learning what staff expertise is needed or if
there are any associated costs for implementation. She may also want to
look back at the original study to see how large the impacts are and think critically
about how they compare with the likely cost of implementing the intervention.
Equipped with this information, Terry will be in a strong position to discuss with her colleagues
what she has found, the pros and cons of different options, and be able to support the program's
decision with what strategies or strategy to implement. And go ahead to the next slide, please.
So again to recap, some further steps she can take is to review individual studies
and publicly available resources, gather more information on designing and implementing the
intervention, potentially read the full article from the CLEAR profile page and
discuss pros and cons with her colleagues. And now I'd like to turn it back to Megan
and the panelists to discuss real life experiences using CLEAR. Megan?
Great, thank you so much, Rhaia. And as we turn our attention to hearing from our panelists,
Lisa and Ben, I want to just take a moment and remind folks to go ahead and use the chat for
any questions that you have and we will cover them in a few minutes once we hear more from
our panelists. So with that I would invite Lisa and Ben to go ahead and turn on your cameras.
I'm going to start with Lisa and then turn to Ben to ask a couple of questions just about
how you use CLEAR or have used CLEAR in your work. So, Lisa, thank you so much for being with us
today. I wonder if you might just share a little bit about how you've used CLEAR in
the past and perhaps give a few examples?
Alright. Thanks, Megan. Well it's so funny because I think
I am Teri in that example. That's pretty much very similar to my journey. So good afternoon
to everyone. I'm Lisa Salazar with the City of Los Angeles and in the introductions shared
that my connection to CLEAR really came through my work in the mayor's office quite
often we are asked for ideas on how best to serve folks with multiple barriers to employment
and of course you know having just 20+ years of experience in the field in direct
program, I have all my good ideas about what works and what doesn't work but you know in
in a mayor's office and when sitting in front of other elected officials, that's all nice
to know that I have my own perspective, but it's such a much stronger argument when I'm able to
cite actual examples of what is working in the field and coming
forward with research-based evidence. So that is really how I came to know CLEAR
and it's just... it's just so easy to use so instead of me just saying
here's an approach to how we can better serve or connect folks experiencing homeless or involved
with the justice system or opportunity youth to employment opportunities, apprenticeships...
I would just go to CLEAR and find examples of what's working across the nation and borrow,
I'll say, those best practices and start to formulate sort of our own program design work on
that evidence that's out there and then also sharing with the mayor and our city council
members how these approaches have worked in other cities and having that data behind it.
So that's how I've come to know CLEAR and have been using CLEAR and are a big fan and
I have to say I have no background in research and evaluation. I am not a data scientist. I'm really a
workforce practitioner, and I have worked at the community level for a number of years so I'm
more of a program implementer I'm more of the person that CLEAR folks evaluate. So again,
Megan, thank you for the opportunity. It's great to be here.
Thanks so, much Lisa. I wonder if you could
build on something you just said about how you're not a data scientist, right? So can you tell
me a little more about like from that perspective what do you particularly like about CLEAR?
I would say two things I really like how easy it is to use. I started using CLEAR
before there were webinars about how to use CLEAR or that I was aware of
webinars. It's very intuitive. It's very easy to just click around and filter.
It's also... it's great in how the information is presented. It's not so dense
that it's, you know, it's very easy to understand especially for me the summaries because that
you know, I hate to admit, is about how much time I'm willing to spend to just
kind of understand what a particular study is about there's just looking at the summaries.
The second thing I would say is that there's just so much information
in CLEAR. You know all of the filters not only by
the types of workers and their particular barriers but also the number of approaches
and supportive services that are in there. Before I knew about clear I would just Google
a particular topic and just find random reports and then have to read like 28-page
reports. With CLEAR, you can just like go in and just narrow down your search and pull up
everything that's in there and then you know and then I choose by the rating system
which ones to read so if it's got Low impact I'm probably not going to read it unless it's
kind of the only thing out there it's such like a niche study but you know those are a couple
things of what I really like in particular about CLEAR.
Thanks, that's really helpful.
I wonder if you've thought ahead at all about, you know, how you might use it in the future?
You know, do you think you'll continue using it in the ways you've described? Are there any
you know new ways that you're thinking you might want to try to use it?
Yeah, sure. So I'd say three ways. Of most recent... in the city
in the Workforce Development Area and also in
Youth Development, we are trying to attach research and evaluation to all of the grant-making that
we do out of the city. So again, not being a data scientist, how we are also looking at CLEAR is and
understanding the evaluation process, understanding the different types of evaluations that are
out there, what it takes to have a high impact on an evaluation. We've really just been learning
more about this field from reading the reports that are in there and understanding
what it takes to have a very strong report that would encourage and I think
help our elected officials make decisions about, you know, whether or not they should fund something
or continue to find something. The other was just this morning, I have a new staff on board
and I just said do you know about CLEAR? And you know this is a very educated person
and she did not know about it and I said take a look at it and let me know what you think and
she was like wow this is great. I've bookmarked about seven tabs that I want to go back and read.
So I will continue to share CLEAR with my staff because you know for folks who work in
in a government setting, you know, I could try to procure the help of a
consultant to help me put together an evaluation or find evidence-based practices but that
would take like six months, eight months for me to do that. I don't need to do that anymore I can
just go to CLEAR and CLEAR provides enough information for me to just grab onto and and
it's just great that way. I mean we're saving tons of time and tons of money by using CLEAR.
That's really great to hear, Lisa. And I love -- hearing you say this like this is I should say
And they didn't tell me to say this! Like this is...
Yes, I didn't tell you to say it! But it's really great to hear how you and your staff are using it
and you know how you're thinking about using it going forward too. So, thank you so much for
those insights. If folks have questions for Lisa, feel free to put them in the chat and
we'll turn to Ben to hear how he's been using it, which I think is a little bit different.
Right. So Ben, I wonder if you could share a little more about how you've been
using CLEAR and provide a few examples for us?
Sure. So, the Texas Workforce Commission
has been exploring and developing an approach to grant-making that incorporates an evidence
evaluation so that we are providing additional points in our applications to those programs
which are evidence-based and have evidence of effectiveness. When we were developing that
that grant-making process, we were really looking for a system to evaluate the evidence of proposals
given that, like I think most people, we don't have a dedicated research team or a dedicated
analysis team that can go in and do these and evaluate all the different studies that are out
there and everybody's proposal and so on. And we found CLEAR along with a couple of other
federal databases or federal clearinghouses to be a really great way for us to find studies
that other people could use to explain what they're doing. So basically, we can ask applicants
please go and find a study that you are replicating, that your evidence is replicating,
and then we can use the CLEAR evaluation of that study to say that yes, your approach
has evidence base or not, and we can we go into some depth about how is your
proposal like this study or different from the study. But we were very pleased with
CLEAR's approach because the primary goal and the primary thing that we were thinking about
during this process was well, obviously the primary thing we think about was finding good
evidence, the primary thing from a grant-making point of view is that we didn't want to scare
off any applicants. We wanted something that was very user friendly we wanted something that was
easy to understand and we wanted something that people could use and review and access
quickly so that it was not a significant additional burden on top of the grant-making
process which occasionally people have complained is tiring. So we are looking... we were looking for
something that was easy to understand for people that weren't data scientists. There's always been
you know meta studies and so on they could go out and find a lot of information about a specific
topic but the ability for someone like me who is also excuse me like Lisa not a data scientist
to go out and quickly review and be able to say yes, this is a High quality study, this is not
a High quality study you know whatever we want to interpret was very convenient and very
user friendly and that was sort of what was carried out was we started to talk to our
grantees about how easy it was to use this system. So we were looking at CLEAR
not strictly for internal use, obviously these are TWC-funded programs that we are trying to
increase our evidence base, but we were looking for something that external partners, some of them very
small, could still look at and quickly find results and to then use this as a way to encourage them to
bring evidence into their programming and their own internal processes, and that's I think the
other side of this is that yes, we want to increase our evidence base for our programs, I think, but
we also want to increase the understanding of evidence and the use of evidence in the
entire ecosystem in Texas that that we are finding ways to support the development of
evidence for programs. And our system doesn't just have the High and Moderate that CLEAR has, we also
have some lower stages that we developed, excuse me, based on our analysis of what our grantees were
doing so that we could genuinely say yes, you know, there are intermediate stages
between being a completely new program that has no evidence and has never been tested before
to a Moderate study that has gotten through the entire process of a complete comprehensive study.
But obviously the ultimate goal is always to move people upwards and we sort of think of our
scoring system as this sort of chain so that you know maybe this year you can only
get the experience and testimonial stage but you can move on to data collection,
you can move on to data analysis, you can move on to using an actual studied technique.
That's really interesting, you know, and I think there's a lot to unpack there.
I love hearing how it's a little bit different from how Lisa's using it, right? I think it shows that
there's a variety of ways to use the resources and the references and the other materials that
we have in CLEAR. I'm wondering if you can talk about what you particularly like about CLEAR again
sort of as a non-data scientist.
I think the two things in 2019 when I reviewed a lot of cleaning houses in 2019, I think one of the things that I really
liked about CLEAR was that it was very easy to quickly make a decision about whether a study had
what you wanted. You know as Lisa was saying previously you would Google a study and then
you had to read it you know you just quickly skim the uh synopsis and the methodology maybe
or you know something like that and try and get a sense of this is what I want but if you can't
tell for sure then you've got to wade through, you know, however long pages and again I'm not a
data scientist so sometimes the papers are a little bit daunting. But with this
with the ability to say, you know, here is what they were testing, here's you know what they were trying
to do, and then here's whether it was done you know to the High or the Moderate standard and just be
able to say yes, this is, they know they were looking at homeless veterans, just let's
say, and this study had a Moderate effect. We can immediately say well that's in if we want to talk
about homeless veterans and it might not be, but it also gives us the opportunity to say
well maybe I'm not working with homeless veterans maybe I'm working with um people who are homeless
but have some other background but maybe there's some interesting information here or conversely
if I'm just looking at veterans maybe I want to I want to see you know this is addressing homeless
veterans but it's a well-done study so it may also give me some insight into the veterans' challenges.
And so it's that cross-referencing and the breadth of data but also just the fact that it's
presented in the these little bite-sized chunks that I can quickly, you know, interpret because
again as Lisa said, you know, we all have lots going on. We can't read 30 papers. And so we would
be able to do a quick study and say well here's eight studies that meet a Moderate or a High
standard and we can go through and pull those and then I can dedicate my time to reading those
rather than reading maybe seven or eight papers that that aren't particularly valuable.
But I wouldn't know that until after the fact. So I think that's the biggest thing for me.
Yeah, you know, that's certainly something that CLEAR tries to do right to make it easier
to pick what you want when you need it. So, you've been working on this evidence-based grant
making initiative for a while, right? Maybe about a year or more. How's it going so far? Do you have
impressions, things you're hearing from your grantees that you'd like to share?
Yeah, so we launched our first evidence-based grant making request for applications in 2021.
And we have since then launched a further five so we've launched six now.
And we continue to develop new ones but we did take the time to
conduct a webinar with the grantees for our first 2021 initial grant and we also took the
opportunity this was this is unique for us we took the opportunity because this was a grant
opportunity that had been offered in the past, we were able to reach out to some
potential eligible applicants who chose not to apply. And so we were able to hear both what
the people who applied thought about it and then also what... why people chose not to apply.
We had decided in 2019, actually I think it was very late 2018, but we really buckled
down and started working in 2019, on designing this system and we were going to publish our very first...
the initial plan was to publish our very first evidence-based grant-making project in March of
2020. Due to external reasons, we decided not to do that and postponed it a little while but
we were really pleased when we got back together with our grant he's at the at the one year mark.
We spoke with them and got some information from them about how they were doing and how they
felt experienced that the application process and we were really pleased with how easy
they said I mean this is the actual grantees saying that I mean, yes, it was more work
than not doing an evidence evaluation, but it was easy to do. It was easy to understand how to get
the information that we needed, and that what you were asking for was in line with what the federal
government has been asking for, so this was not some surprise. And so we were very pleased
that we were able to to see this and that we heard from both community colleges and from small
non-profits saying that this was this was doable for our staff and then while we haven't done
that same kind of detailed webinar with other applications, we've continued to see quality
applications and not see a significant decrease in applications for our results or
requests for application, and that we are seeing a diversity of response. One of the things that we
set out and this is less related to CLEAR, but just to say that we know that most grantees aren't
necessarily going to have a High or a Moderate study that examines their intervention. but we
want to set that up as an ambitious goal that you know maybe someday everybody can be moving
in that direction. And so we've continued to see people at all points on our scale but what was
surprising to me and I'm going to be up front and say that I was the most gloomy person
when our development team, when we were talking about what we would see from our applicants,
but we actually saw in our very first one that we had people who had Moderate and High levels
of evidence supporting their applications and we've continued to see Moderate and High levels of
evidence coming up. Now again, it's not the majority but we are really pleased with how much
having the ability to get that Moderate and High rating has really
supported some of our our grantees and applicants.
That's really exciting. I, you know, that sort of culture change,
that building that culture of sort of continuous improvement and you know evidence-building is
you know something that does take time. I think it reminds me a lot of what we're seeing
in the re-employment services and eligibility assessments
world you know RESEA for those of you that are working and thinking in that space too.
And you know I know we've got a variety of folks on the phone here that are
thinking about some of those things. So you know there's definitely some crossover lessons
there that that could be interesting to explore more. I wonder if you might
share how you are thinking you'll like to use CLEAR in the future, Ben?
So I think we're always looking at the first thing we want to do is find ways to expand our
projects you know we fund projects that are not necessarily as narrowly focused
on workforce in the sense of you're taking an unemployed person and moving them to employment
but there's a variety of other projects that we do and we are continuing to look at how we can use
CLEAR data to support those you know more precise interventions and how we can work with
that we're also moving in the direction of sort of more what Lisa was talking about
that that we want to start using CLEAR internally for our own things not just in the evaluation of
grants but in the original development of requests for application, that how can we
present our applications in a way that promotes solid design and solid development
of these programs and how can we, you know, not ask people to reinvent the wheel when we...
if we already know that there is a kind of a gold standard out there for how to solve one
specific aspect of it, we can simply, you know, build that into our initial design concept.
We're also very interested in the overall research summaries that CLEAR is
doing to sort of give us a broad spectrum to say well where is some place where maybe we
aren't currently doing research or currently doing grant-making but maybe there's enough research to
say well maybe we should be doing something in that these are all potential opportunities
that we have in front of us and, you know, to just add to that, one of the other things
that we've used that isn't strictly technically using CLEAR, but one of the things we
liked about CLEAR was the very detailed and easily available standards that CLEAR was using
we've had the opportunity to use CLEAR standards when we have hired
external evaluators to say we want you to design a study that will meet at least the Moderate or
maybe the High level of CLEAR evidence, just so that we can for example as we look at
you know as we look at our own internal research and our and bringing in evaluators for our own
programs to know that we're getting the kind of quality evaluation that we want and they
could potentially be supporting long-term other people's evaluations and other people's research.
So those are I think the key points for us looking at the future of CLEAR
with us and again also finding ways to expand into other categories, how can we support youth
interventions, how can we support other ones that may not have that easy connection between
the intervention and the final outcome that we want. If you're intervening with a 14 year
old it's hard to say and then they got a job and we can say with confidence that there's
a connection there. So just finding ways to look at these much more complex studies than
Texas could ever do in the short term to see if we can find some guidance and support there.
That's great, Ben. Thank you for sharing that. Before we move to the next slide there was
one question that came in that I think is best to answer just right now while we have you
here and this is really this is for you, Ben, if you could tell us a bit more about
the kinds of programs that were the focus of your evidence-based grants. The question is
were they all service providing programs or were some research projects?
So far they have all been service projects. We've done... our pilot program was the
building construction trades RFA that we have done and it's basically an
intervention to help people go from under or unemployment into the skilled trades and the
construction industry and the different varieties of interventions and support services and and so
on that could be offered there. We've also done some studies on adult education services
and some additional ones for other service design programs like that.
Thanks, that's really helpful. I want to thank both of you, Ben and Lisa, for joining us in this
panel. We're going to get to more questions in a few minutes and so if folks can keep entering
those in the chat, but if we could go to the next slide, I wanted to take a quick minute and share a
few resources about sort of where to go to get more information, right? So next slide, please.
Now that you've heard a little bit about a couple of different ways to use CLEAR, right, both our
example from the study team where we walked through navigating CLEAR as well as some of
the experiences that Lisa and Ben have shared. If you want to get more into the nitty-gritty here
on the left-hand side we have a list of links to some of our reference materials, right, how we
do some of these things that we're talking about beyond the profile summaries and the syntheses
and things that we talked about earlier. I also wanted to remind everybody to save the date we
have another webinar coming up in December that we'll talk about using CLEAR to find strategies
that address equity. And while today's webinar was sort of more focused at the workforce
in the workforce perspective this webinar is going to really be trying to include both
workforce as well as compliance assistance, worker protection, and other areas that are in the labor
sphere that are of interest that are seeking to address equity. You can find any event recordings
that we have we are posting to the CEO events page and then also we're developing a page in
CLEAR where we can post these and other resources for you, and if you have a question at any time we
do invite you to just reach out and contact CLEAR. So I have the link here it's also easily found on
our website. If you want to hear more about CEO, we've got a few ways here that you can
do that easily. You can click on the link here and look for future events that we're hosting
that are public. You can also sign up for our periodic newsletter, Building the Evidence Base,
and that will tell you a little bit about the latest work that CEO is doing. Next slide.
So again, I wanted to just recap kind of what we went through today. We learned how to navigate
CLEAR to find information, we heard multiple examples of how some of their research and
evaluation-based evidence and CLEAR is or could be used to support policy and program decision-making,
and we talked about where to look to learn more. So before we launch into our questions. we're going to
cover just a few polls so we can get your feedback about how you thought today's webinar went and
what you want to hear more about from us in the future.