WEBVTT
Kind: captions
Language: en

00:00:01.080 --> 00:00:08.400
Hi welcome everyone so for today the Chief 
Evaluation Office is celebrating 10 years of  

00:00:08.400 --> 00:00:13.560
the Clearinghouse for Labor Evaluation and 
Research. We would like to welcome you to  

00:00:13.560 --> 00:00:19.800
today's webinar, "How to use CLEAR for decision 
making." CLEAR is funded through the Department  

00:00:19.800 --> 00:00:26.100
of Labor's Chief Evaluation Office and CLEAR 
is supported by various independent contractors.

00:00:30.000 --> 00:00:36.660
For today to submit content related 
questions select MSG Webex A from the  

00:00:36.660 --> 00:00:41.760
drop down menu in the chat panel and enter 
your question in the chat box and send.

00:00:42.960 --> 00:00:48.960
The event is being recorded. By continuing to 
participate you are consenting to being recorded.  

00:00:49.980 --> 00:00:56.520
I will now turn it over to Megan Lizik. Ms. 
Lizik is with DOL's Chief Evaluation Office  

00:00:56.520 --> 00:01:03.240
and CLEAR's team lead. She has nearly 20 years 
of analytical and evidence-building experience  

00:01:03.240 --> 00:01:10.080
in both the private and public sectors and 
currently oversees a portfolio of behavioral  

00:01:10.080 --> 00:01:16.980
science, evidence translation, and other labor 
related research and evaluation projects for DOL. Megan?

00:01:17.880 --> 00:01:25.560
Great thank you so much Brittany. Hello and 
welcome to all of you. It's really great to see  

00:01:25.560 --> 00:01:30.960
so many of you joining us today for one 
of the topics that's near and dear to my heart  

00:01:31.620 --> 00:01:36.360
really getting into the nitty-gritty on 
how to use CLEAR in your own decision making  

00:01:36.360 --> 00:01:45.660
on a day-to-day basis. So to kick us off I 
wanted to invite our deputy director Lauren Damme

00:01:46.620 --> 00:01:52.980
who is here to share some opening thoughts 
with us. Lauren is responsible for helping  

00:01:52.980 --> 00:01:58.920
oversee CEO's evaluation research portfolio 
staff and office functions and before joining  

00:01:58.920 --> 00:02:05.160
CEO in 2019 she served as the senior evaluation 
research advisor in the Department's Bureau of  

00:02:05.160 --> 00:02:05.660
International labor Affairs Office of Child 
Labor, Forced Labor, and Human Trafficking.

00:02:05.660 --> 00:02:15.840
She has over 18 years of experience in quantitative 
and qualitative research and evaluation

00:02:15.840 --> 00:02:20.820
primarily in International Development and 
has worked in around 30 countries. Lauren. 

00:02:22.620 --> 00:02:27.540
Thanks so much Megan I didn't know you were going 
to do a whole introduction. Thank you. So thanks  

00:02:27.540 --> 00:02:32.820
so much for having me and welcome everyone I'm 
really excited to be here today. Today's session is  

00:02:32.820 --> 00:02:38.460
going to focus on how to use CLEAR for decision 
making and by the end of our time together you  

00:02:38.460 --> 00:02:42.540
will learn how to use clarifying evidence that 
you can use right away including to quickly gain  

00:02:42.540 --> 00:02:47.340
an understanding of the state of the field on a 
particular topic and we're also going to hear from  

00:02:47.340 --> 00:02:53.160
peers from workforce agencies in the field about 
how they're already using CLEAR. But first I do  

00:02:53.160 --> 00:02:58.800
want to thank Megan and the entire CLEAR team at CEO 
as well as our contracting partners for putting  

00:02:58.800 --> 00:03:03.960
on these public events. They do just an incredible 
job relative to our time and resources and it's  

00:03:03.960 --> 00:03:07.980
amazing that we are celebrating the 10-year 
anniversary of CLEAR this year and I know that  

00:03:07.980 --> 00:03:12.840
there are some others I saw on who are joining 
us today from outside of DOL who were involved  

00:03:12.840 --> 00:03:18.420
in some of the inception activities of CLEAR. So 
I would also like to thank our presenters for  

00:03:18.420 --> 00:03:22.140
agreeing to share their stories about how they've 
used CLEAR. I've heard from some of them before  

00:03:22.140 --> 00:03:27.540
including Ben for example and I think that their 
experiences that they will share really bring to  

00:03:27.540 --> 00:03:33.060
life how people can leverage CLEAR to improve the 
way that we serve and support workers. So I think  

00:03:33.060 --> 00:03:37.380
this could be a great session and I'm looking 
forward to learning something also but before we  

00:03:37.380 --> 00:03:42.060
get going I wanted to reflect on the importance 
and the challenges of using research evidence  

00:03:42.060 --> 00:03:47.640
and decision making which is really the reason why 
DOL created CLEAR. So many of you on this call have  

00:03:47.640 --> 00:03:52.560
probably heard of or at this event have heard of 
the Foundations for Evidence-Based Policy Making  

00:03:52.560 --> 00:03:59.040
Act of 2019 or The Evidence Act and you know that 
there has been a huge growth in recent decades of  

00:03:59.040 --> 00:04:04.200
federal, state, and local efforts to bring science 
to bear and program and policy decision-making.  

00:04:04.200 --> 00:04:09.840
However, I think all of us also know that there are 
real challenges to using research and evaluations  

00:04:09.840 --> 00:04:15.000
in your everyday decision-making so it can be 
really hard to find where that evidence lives  

00:04:15.000 --> 00:04:19.260
or recall when you need it it can also just 
be hard to keep up given the proliferation  

00:04:19.260 --> 00:04:25.620
of research and evaluation studies so earlier this 
week for example I heard from a leading scholar on  

00:04:25.620 --> 00:04:32.220
synthesis research based at Monash University in 
Australia who works on synthesis work that similar  

00:04:32.220 --> 00:04:36.360
to that which CLEAR does and they said that in 
their work they found that research output is  

00:04:36.360 --> 00:04:42.540
doubling something like every 10 or 15 years like 
exponential growth and adding to that volume issue  

00:04:42.540 --> 00:04:48.240
with keeping up is just that going deep into any 
particular single study can also be a daunting  

00:04:48.240 --> 00:04:54.960
prospect for busy practitioners, policymakers, and 
others like yourselves who are attempting to make  

00:04:54.960 --> 00:04:59.280
or who are interested in making 
evidence-informed decisions. So CLEAR  

00:04:59.280 --> 00:05:04.200
really helps address this problem. The team sits 
through tens of thousands of research studies  

00:05:04.200 --> 00:05:09.960
to find and review the most credible ones that 
examine effective and promising programs across  

00:05:09.960 --> 00:05:15.780
a number of labor-related areas. CLEAR does not 
just conduct literature reviews and that's a  

00:05:15.780 --> 00:05:20.580
question that I know that I've gotten before 
the team it's not just a literature review the  

00:05:20.580 --> 00:05:25.500
team looks at every single relevant report in the 
English language during the specific time period.

00:05:25.500 --> 00:05:31.140
It's extremely systematic and then CLEAR provides 
you with a plain language summary of the studies  

00:05:31.140 --> 00:05:36.720
that they have found meets specific criteria to 
help you understand more about what works as well  

00:05:36.720 --> 00:05:42.600
as how where and for whom. So the other 
last thing I'll mention about CLEAR for those of  

00:05:42.600 --> 00:05:46.980
you who are not familiar with it is that CLEAR
assigns ratings to causal studies in particular in  

00:05:46.980 --> 00:05:52.800
order so where we're trying to make a connection 
between a program or policy and an effect that it  

00:05:52.800 --> 00:06:00.420
has had on something for example increasing 
a training program increasing the wages of workers  

00:06:01.080 --> 00:06:07.020
in order that those causal ratings 
or the ratings for causal studies help  

00:06:07.020 --> 00:06:12.180
you understand how confident you can be in the 
findings of those studies and to date I think  

00:06:12.180 --> 00:06:18.360
CLEAR has conducted more than 25 evidence reviews 
and summarized nearly 1300 studies but again to  

00:06:18.360 --> 00:06:22.020
whittle down to those credible studies the team 
has reviewed you know thousands and thousands of  

00:06:22.020 --> 00:06:27.720
papers and reports. So you can really trust CLEAR
and other federal clearinghouses to find research  

00:06:27.720 --> 00:06:33.780
and evaluation evidence that you need when you 
need it and to that end as I mentioned before  

00:06:33.780 --> 00:06:39.000
we're going to hear from some workforce leaders 
about how they use CLEAR and we hope that  

00:06:39.000 --> 00:06:44.880
inspires some ideas on how you can use CLEAR to 
support your own work and that may also include  

00:06:44.880 --> 00:06:49.620
identifying where there are still gaps in our 
knowledge. So I'm really excited about today  

00:06:49.620 --> 00:06:55.800
CLEAR fills a very big need and hunger for 
accessible reliable information when you need  

00:06:55.800 --> 00:07:01.000
it and I hope that you find the session really 
helpful. Let me turn it back over to you, Megan, 

00:07:01.000 --> 00:07:04.040
to introduce the presenters and panelists for today. 

00:07:04.040 --> 00:07:06.960
Great, thank you so much, Lauren.

00:07:07.920 --> 00:07:13.620
So without further ado, I'll share a little 
bit more about who you'll be hearing from. So first  

00:07:13.620 --> 00:07:19.860
we have some of our presenters from one of 
our CLEAR contract teams. Chris Weiss is  

00:07:19.860 --> 00:07:25.920
a principal associate at Abt Associates and he is 
the principal investigator for Abt's work managing  

00:07:25.920 --> 00:07:32.880
CLEAR. We also have Andrew Clarkwest with us and 
he is a principal associate at Abt Associates as  

00:07:32.880 --> 00:07:39.600
well and he directs Abt's CLEAR contract with DOL.
And then finally we have Rhaia Hull and she is an  

00:07:39.600 --> 00:07:45.720
analyst at Abt Associates. She oversees the study 
review process and Abt's team of study reviewers. 

00:07:45.720 --> 00:07:53.220
So welcome to all of you. And then I'd also like 
to introduce our panelists for today if we could  

00:07:53.220 --> 00:07:59.760
go to the next slide, thank you. We couldn't 
be more excited to be joined by some workforce  

00:07:59.760 --> 00:08:07.260
leaders Ben Holquist and Lisa Salazar to hear more 
about their experience using CLEAR. Ben has been  

00:08:07.260 --> 00:08:12.060
a leading partner in developing much of the Texas 
Workforce Commission's evidence and outcomes-based  

00:08:12.060 --> 00:08:17.400
grant-making processes and specifically he's 
been leveraging CLEAR as part of those efforts. 

00:08:17.400 --> 00:08:24.480
He's been working on Texas's evidence-based 
grant-making project since 2019. And Lisa is  

00:08:24.480 --> 00:08:31.380
the first Executive Director of the City of Los 
Angeles's Youth Development Department and prior to  

00:08:31.380 --> 00:08:37.080
working there Ms. Salazar served as the Director 
of Workforce Development and Economic Opportunity  

00:08:37.080 --> 00:08:43.020
to Los Angeles Mayor Eric Garcetti, responsible 
for leading the development of policies programs  

00:08:43.020 --> 00:08:48.300
and partnerships to advance the mayor's agenda on 
workforce development, educational career pathways,

00:08:48.300 --> 00:08:53.340
and youth development citywide since 2000. She 
has spent more than two decades in leadership  

00:08:53.340 --> 00:08:57.720
positions working for the city developing and 
implementing public policies and programs on  

00:08:57.720 --> 00:09:02.460
youth and adult employment job training and skills 
development, career pathways, and the future of work.

00:09:03.060 --> 00:09:07.740
So welcome to our panelists. Welcome to our 
presenters. We're thrilled to have you with us. 

00:09:08.820 --> 00:09:16.680
Next slide. So before we get going I wanted 
to share a little bit more about some  

00:09:16.680 --> 00:09:22.860
background of DOL's Chief Evaluation Office. Our 
office was established in 2010 as an independent  

00:09:22.860 --> 00:09:28.140
Departmental-level evaluation shop here at 
DOL. We coordinate manage and implement the  

00:09:28.140 --> 00:09:36.120
Department's evaluation program, working closely 
with colleagues in about 15 DOL agencies to help  

00:09:36.120 --> 00:09:41.460
address their research questions and support their 
learning and continuous improvement priorities  

00:09:41.460 --> 00:09:48.120
by conducting evaluation studies, data analysis, 
and evaluation technical assistance. We do most  

00:09:48.120 --> 00:09:54.840
of our work through contracts like the ones we 
have for CLEAR, and currently we have about 60  

00:09:54.840 --> 00:10:01.020
active projects that we're working on with DOL
agencies. You can learn more about those as well  

00:10:01.020 --> 00:10:05.460
as view our past reports by clicking at some 
of the links here on the bottom of our screen.

00:10:07.560 --> 00:10:15.240
Next slide. So as we get going we'd like to kind 
of hear from you sort of who you are where you're  

00:10:15.240 --> 00:10:20.280
from and your experience with CLEAR so we've got 
a couple of polls that Brittni's going to pull  

00:10:20.280 --> 00:10:25.320
up for us, if you wouldn't mind launching 
that Brittni, and we're gonna spend just  

00:10:25.320 --> 00:10:32.160
a couple of seconds hearing what organization 
you're from you can see it there in the chat.

00:10:39.960 --> 00:10:43.500
And when you have the results, Brittni,
you can go ahead and share those with us

00:10:53.820 --> 00:10:56.520
Okay and then we have a second poll

00:10:59.640 --> 00:11:02.640
and this poll is really to understand what  

00:11:02.640 --> 00:11:07.260
your experience is with CLEAR so we 
want to know a little bit more about  

00:11:08.760 --> 00:11:17.460
whether this is your first experience hearing 
about CLEAR, whether you've used it a few times  

00:11:17.460 --> 00:11:23.340
but it's been a while you could use a refresher, 
whether you use it regularly so whenever you're  

00:11:23.340 --> 00:11:28.380
looking for this kind of research and evaluation 
evidence to inform your work, or if you're really  

00:11:28.380 --> 00:11:34.500
kind of not sure yet and you're waiting to see 
more from today so take a few minutes and if you  

00:11:34.500 --> 00:11:40.080
could look in the the polling on the right hand 
side of your screen you'll see our questions there.

00:11:46.380 --> 00:11:48.780
And Brittany do you have those results for us?

00:11:52.500 --> 00:11:54.000
They're loading. 10 more seconds.

00:12:00.120 --> 00:12:03.060
OK. So while there are loading,

00:12:05.580 --> 00:12:09.180
we can go ahead and go 
to the next slide I think  

00:12:09.840 --> 00:12:14.220
we want to start by sharing a little 
bit about the goals for today's webinar.

00:12:16.440 --> 00:12:25.920
Thank you. So really our goal is to try to 
help share with you some insights about how  

00:12:25.920 --> 00:12:33.300
to navigate CLEAR, how to find the key information 
that you are looking for when you need it. We will  

00:12:33.300 --> 00:12:39.120
also offer multiple examples of how you might use 
it, so we're going to walk through one theoretical  

00:12:39.120 --> 00:12:44.580
one from a community college perspective and 
then we'll have our mini panelists offer their  

00:12:44.580 --> 00:12:49.380
insights about how they use CLEAR, and then I'll 
go through a little bit at the end of our time  

00:12:49.380 --> 00:12:58.920
together today to understand where to learn or to 
look to learn more. Next slide. So this is a little  

00:12:58.920 --> 00:13:05.220
bit of an overview of our agenda. We're going to 
do a short recap of some concepts from the 101  

00:13:05.220 --> 00:13:10.380
webinar that we held a couple of weeks ago just 
to kind of provide an overview and grounding for  

00:13:10.380 --> 00:13:15.240
everybody joining today regardless of whether you 
heard that webinar just about kind of what CLEAR

00:13:15.240 --> 00:13:20.580
is and does and how we do it. Then we're going 
to launch into the content of today's webinar, 

00:13:20.580 --> 00:13:28.200
hear from our panelists, do the resource recap, 
and then have a few closing poll questions and  

00:13:28.200 --> 00:13:35.400
then plenty of time for discussion at the end so 
please do hang on with us. As you have questions  

00:13:35.400 --> 00:13:42.300
feel free to put them in the chat and we will 
revisit them at the end of the webinar. Next slide.

00:13:44.400 --> 00:13:52.000
Great, so now I'd like to turn things over to 
Chris to recap all about CLEAR 101. Chris?  

00:13:52.040 --> 00:13:58.560
Hi, thank you Megan. It's a pleasure to be with you 
all today. If you could go to the next slide, please. 

00:13:59.760 --> 00:14:06.060
Great there are three parts of the prior 
webinar that I want to briefly highlight about  

00:14:06.060 --> 00:14:12.240
CLEAR by way of setup. You see them listed here: 
CLEAR's mission, the evidence review process, and  

00:14:12.240 --> 00:14:16.920
study summary icons. We'll have more to say about 
each of these and use these as examples in part  

00:14:16.920 --> 00:14:22.800
of the presentation later on but at the outset 
wanted to cover or if you joined for the previous  

00:14:22.800 --> 00:14:28.440
presentation recover the some of these critical 
elements of it. If you go to the next slide, please? 

00:14:30.360 --> 00:14:35.580
Thanks. First, the mission. CLEAR's mission 
is listed here: "To make research on labor  

00:14:35.580 --> 00:14:41.640
topics more accessible to practitioners, 
policymakers, researchers, and the public so  

00:14:41.640 --> 00:14:48.720
that it can inform decisions about labor 
policies and programs." That's a broad goal,  

00:14:49.560 --> 00:14:55.080
and there's lots of things lots of resources 
on CLEAR that will help you to achieve those  

00:14:55.080 --> 00:14:59.340
goals. CLEAR is a place where you could go 
to see for example how much research is  

00:14:59.340 --> 00:15:04.740
out there. Is there a great deal known about a 
particular topic or or not much or very little?

00:15:05.460 --> 00:15:10.500
You can also learn what the research says about 
particular strategies, policies, or programs, and  

00:15:10.500 --> 00:15:16.080
if they're reaching their goals and the third 
is out how confident we can be in the findings  

00:15:16.680 --> 00:15:21.300
of the research in a particular programmatic 
area. If you could go to the next slide, please.

00:15:23.700 --> 00:15:28.800
Here's a brief, very brief, overview of the 
evidence review process but let me

00:15:28.800 --> 00:15:33.720
walk you through a little bit of this briefly. 
CLEAR conducts systematic evidence review on  

00:15:33.720 --> 00:15:38.640
labor topics including studies funded by the 
Department of Labor as well as other studies  

00:15:38.640 --> 00:15:44.940
from published reports. Then, we summarize the 
methodologies, findings, and policy or program  

00:15:44.940 --> 00:15:52.000
implications of these studies. CLEAR's topics 
are determined by the DOL's Chief Evaluation Office

00:15:52.040 --> 00:15:57.480
that works often in consultation with 
DOL agencies, CLEAR contractor project staff,  

00:15:57.480 --> 00:16:05.040
technical advisors, and topic area subject matter 
experts. You see the different stages here listed  

00:16:05.040 --> 00:16:10.920
we can talk in greater depth about these or 
we'd invite you to watch the webinar watch  

00:16:10.920 --> 00:16:17.280
the recording of the webinar when it's available. 
And as Lauren mentioned at the outset, CLEAR

00:16:17.280 --> 00:16:23.340
has completed evidence reviews of nearly 25 
labor topics and currently houses summaries  

00:16:23.340 --> 00:16:30.120
of almost 1300 studies with more studies being 
added continuously as we continue to grow and  

00:16:30.120 --> 00:16:35.520
grow in the number of topics and studies 
that have been reviewed. Next slide, please.

00:16:39.480 --> 00:16:44.460
Third point is a brief, brief recap of the 
study summary icons. You see two different  

00:16:45.180 --> 00:16:50.400
dimensions of study quality or excuse me of 
icons that are presented that you'll see on CLEAR

00:16:50.400 --> 00:16:57.120
study pages. The first of these are the study 
evidence ratings or study quality icons.

00:16:58.320 --> 00:17:02.580
CLEAR's causal evidence ratings 
provide information about the quality of  

00:17:02.580 --> 00:17:06.420
the causal evidence in a study and there 
are three different levels you see here  

00:17:06.420 --> 00:17:12.300
presented in that top... the kind of 
like half-moon dial. High, moderate, and low.

00:17:13.380 --> 00:17:17.520
For studies that receive a High rating, we can 
feel confident that the findings reported  

00:17:17.520 --> 00:17:22.920
in the study are due to the intervention or 
program understudy and not to other factors.

00:17:24.120 --> 00:17:28.440
For Moderate rated studies, we can be somewhat 
confident that the findings are due to the program  

00:17:28.440 --> 00:17:33.360
or intervention examined in the study and not to 
other factors, but it may be that other factors  

00:17:33.360 --> 00:17:39.540
contributed as well. Finally for Low rated studies,
we cannot be confident that the findings are due  

00:17:39.540 --> 00:17:44.580
to the program or intervention that other factors 
are likely to have contributed to the observed  

00:17:44.580 --> 00:17:49.680
outcomes. You can find more information explaining 
this and the process by which the criteria by  

00:17:49.680 --> 00:17:56.700
which these ratings are applied on the CLEAR
website. The second part of the panel on the  

00:17:56.700 --> 00:18:02.340
right displays Effectiveness Icons. These icons 
provide a quick overview of the study's impact  

00:18:02.340 --> 00:18:10.800
findings as shown on the slide there are five, four, 
excuse me, designations: Favorable, Mixed, Unfavorable,  

00:18:10.800 --> 00:18:17.160
and None. These designations are provided for 
each outcome domain that the study examined. 

00:18:18.360 --> 00:18:22.800
I recognize that if you're new 
you didn't see the previous or if you're  

00:18:22.800 --> 00:18:29.040
a relative novice user to CLEAR, many of 
the words I have just said probably are  

00:18:29.040 --> 00:18:34.320
the words themselves may not be may be familiar 
but the meaning and the the details of how of what  

00:18:34.320 --> 00:18:39.840
they mean in the context of CLEAR may not be. But 
it's my pleasure now at this point to pass the  

00:18:39.840 --> 00:18:45.240
mic over to Andrew Clarkwest, who will 
discuss how the resources of CLEAR can be used to  

00:18:45.240 --> 00:18:48.040
inform decision-making. Andrew? 

00:18:48.240 --> 00:18:52.320
Right. Thanks a lot, Chris. 
I'm happy to be here to have a chance to  

00:18:52.320 --> 00:18:58.260
talk about CLEAR's resources that can support 
evidence-informed decision-making. Next slide.

00:19:01.500 --> 00:19:06.180
So if you're someone who's interested in 
using evidence to support decision making  

00:19:06.180 --> 00:19:12.240
around some policy or program question, which I 
assume applies to a lot of us who are in this  

00:19:12.240 --> 00:19:18.360
meeting right now, in theory, it sounds fairly 
straightforward. You go out and find research  

00:19:18.360 --> 00:19:21.960
evidence, see what it says, then incorporate 
those insights into your decision-making.  

00:19:23.820 --> 00:19:32.940
But as Lauren alluded to, and as indicated on 
the next slide, in practice it tends to be slightly  

00:19:32.940 --> 00:19:37.920
more complicated than that. When you're looking 
into research onto a topic you may depending  

00:19:37.920 --> 00:19:42.900
on the subject run into a lot of studies or 
alternatively you may find it hard to find a lot  

00:19:42.900 --> 00:19:47.160
of the research on a topic since studies may have 
been published in a variety of different places.  

00:19:48.780 --> 00:19:53.760
Beyond that the studies on a topic may differ 
in a lot of ways. They may have used different  

00:19:53.760 --> 00:20:00.480
methods to look at interventions or outcomes. Those 
interventions or outcomes may also vary somewhat  

00:20:01.200 --> 00:20:08.640
and the study may have looked at them in contexts 
that are different from yours or different from  

00:20:08.640 --> 00:20:14.940
one another in terms of their geography or 
the economy at the time or in the local place.

00:20:14.940 --> 00:20:21.240
They may have also served different types of 
populations. And even worse, when you look at the  

00:20:21.240 --> 00:20:27.060
studies you may see that their findings aren't 
all congruent with one another or at least  

00:20:27.060 --> 00:20:34.260
they may not seem to be congruent. So that leads to 
a lot of heterogeneity that makes it challenging  

00:20:34.260 --> 00:20:40.200
to get insights out of an evidence base 
to make sense of what exactly does this mean  

00:20:40.200 --> 00:20:45.360
and what does it mean not just in generally 
but also specifically for me. Next slide

00:20:49.620 --> 00:20:55.140
And this slide here shows three types of 
challenges that you'll run into or three  

00:20:55.140 --> 00:21:00.000
types of questions that are key for someone 
who's trying to make sense of an evidence base  

00:21:00.000 --> 00:21:06.900
and figure out how to apply it to their decision-making.
The first one is substance. That's just  

00:21:06.900 --> 00:21:11.700
what have the studies actually found, which isn't 
always as easy to tell as you might expect. 

00:21:12.900 --> 00:21:17.580
The second is credibility, and this 
gets into very technical issues as  

00:21:17.580 --> 00:21:21.660
Chris mentioned particularly when looking at 
causal studies the studies can vary a lot in  

00:21:21.660 --> 00:21:25.620
how confident we can be that there are study 
that our findings really do reflect the impact  

00:21:26.220 --> 00:21:31.740
of the intervention as opposed to reflecting some 
other factor so if I look at a finding in a study  

00:21:31.740 --> 00:21:38.880
can I really believe it? I mean studies vary in 
there... in how they were conducted in ways that  

00:21:38.880 --> 00:21:43.080
can be very technical and can be difficult 
to sort out for a non-specialist audience.

00:21:44.880 --> 00:21:50.280
A third issue is applicability, which gets 
at, in colloquial terms, whether your mileage  

00:21:50.280 --> 00:21:56.400
may differ. That is, at an initial level, wanting 
to know or that's important to know is enough  

00:21:56.400 --> 00:22:01.080
about the intervention to tell if it seems like 
something that you could implement reasonably  

00:22:01.080 --> 00:22:06.600
well in your circumstances. And beyond that when 
looking at research findings you may want to  

00:22:06.600 --> 00:22:10.680
understand the time, place, and circumstances 
where an intervention has been studied  

00:22:11.280 --> 00:22:15.780
and whether those differ from yours in ways that 
might lead results that you get from implementing  

00:22:15.780 --> 00:22:21.360
the approach to differ from the results 
that were found in prior research. Next slide.

00:22:25.380 --> 00:22:29.760
So CLEAR not only brings together research on 
labor-related topics in a single clearinghouse  

00:22:29.760 --> 00:22:34.260
it then goes on to provide help with 
each of those three questions that  

00:22:34.260 --> 00:22:39.780
are key to making sense of the evidence 
base. You'll see the three icons on the  

00:22:39.780 --> 00:22:45.480
left of the slide here reappear throughout 
today's presentation as we walk through how  

00:22:45.480 --> 00:22:51.060
to find information for decision-making 
in CLEAR's evidence resources. Next slide.

00:22:55.500 --> 00:22:59.220
And we'll group those evidence resources 
into three types as we're walking through it.

00:22:59.220 --> 00:23:02.400
That'll be signaled by the 
colored boxes that you see here,

00:23:02.400 --> 00:23:06.900
which constitute a sort of navigation key.
The boxes will appear at the top right  

00:23:06.900 --> 00:23:11.640
of the subsequent slides as we cover 
each of those kinds of resources. Next.

00:23:15.000 --> 00:23:19.020
So to just walk through each of the three 
briefly, profile summaries provide details  

00:23:19.020 --> 00:23:23.700
on the individual studies that CLEAR has reviewed,
so they provide study-specific information  

00:23:25.320 --> 00:23:30.660
and usually studies are reviewed as part of 
a targeted search that's targeted by topic  

00:23:30.660 --> 00:23:38.040
or by time. And then the second box, an evidence 
review is a collection of research on a topic  

00:23:38.040 --> 00:23:43.200
so a collection of studies that may be 
defined by looking at a similar population  

00:23:43.200 --> 00:23:51.660
like low-income adults or older workers or their 
strategies to improve a particular outcome like  

00:23:51.660 --> 00:23:58.980
literacy, or re-employment, or equitable post-COVID recovery,
or potentially by intervention type so  

00:23:58.980 --> 00:24:04.020
CLEAR's interview has assessed research on 
apprenticeship and work-based training approaches

00:24:04.020 --> 00:24:12.780
and remote service delivery approaches of various 
kinds. Finally, the evidence syntheses that clear  

00:24:12.780 --> 00:24:18.660
creates are short reports that describe findings 
about what works from the body of evidence on a  

00:24:18.660 --> 00:24:24.240
topic and these syntheses also note gaps 
where more evidence is needed. Next slide.

00:24:28.860 --> 00:24:35.100
So when you come to CLEAR, you'll typically 
first hit the the front page here. You'll see at  

00:24:35.100 --> 00:24:41.640
the top there's a set of CLEAR navigation tabs in 
a navigation bar and if we go to the next slide  

00:24:41.640 --> 00:24:47.220
we'll see that we zoom in to make the evidence 
of the navigation bar a bit more readable for  

00:24:47.220 --> 00:24:54.120
you and the arrows highlight three elements in 
that bar. The first arrow points to topic areas,

00:24:54.120 --> 00:25:00.060
clicking on that will provide a drop down menu 
that you can browse of existing clear topic areas.

00:25:01.020 --> 00:25:05.940
The second arrow points to the search for studies 
page and that's a place that you'll often likely  

00:25:05.940 --> 00:25:10.740
want to start when you're looking for research 
evidence, and we're going to in subsequent slides  

00:25:10.740 --> 00:25:14.520
in this presentation going to walk through the 
search function at a couple different points.

00:25:15.540 --> 00:25:21.540
The third arrow points to the "About CLEAR"
element of the navigation menu. We aren't  

00:25:21.540 --> 00:25:24.720
going to walk through that directly in 
today's presentation but for those of  

00:25:24.720 --> 00:25:30.480
you who are curious about all of the details 
of CLEAR's procedures and how it rates studies  

00:25:30.480 --> 00:25:34.680
the kinds of things that Chris was talking 
about, you can find that documentation there  

00:25:34.680 --> 00:25:41.520
which you can peruse to your heart's content 
whenever you have some free time. Next slide.

00:25:44.640 --> 00:25:48.660
So Megan mentioned we're going to 
walk through an example of a hypothetical, 

00:25:48.660 --> 00:25:53.340
a hypothetical example of using CLEAR to 
inform a particular program question

00:25:56.160 --> 00:26:02.340
and in this case the question is "How can 
we find evidence on strategies to increase  

00:26:02.340 --> 00:26:09.360
trainee persistence?" So suppose in this 
case that we have a user named Teri who  

00:26:09.360 --> 00:26:13.680
is a community college administrator and 
they're administering a training program  

00:26:13.680 --> 00:26:19.200
or running a training program where they're 
having issues with students not completing  

00:26:19.200 --> 00:26:24.420
the program and want to know how can we improve 
low training program persistence and completion?  

00:26:26.040 --> 00:26:29.160
So in that case thinking okay we 
have some thoughts would it be useful  

00:26:29.160 --> 00:26:34.080
to know what's been tried elsewhere and in 
particular what's been studied before?

00:26:35.640 --> 00:26:40.740
Sp we're going to walk through how she might 
use CLEAR to identify research-based insights  

00:26:40.740 --> 00:26:45.360
on interventions that improve program 
persistence and completion. Next slide.

00:26:49.980 --> 00:26:54.780
So as I mentioned, she might start 
with the search for studies

00:26:55.740 --> 00:27:02.220
function so she would click on search for 
studies link on the navigation bar which  

00:27:02.220 --> 00:27:10.020
would bring her as we see in the next slide to a 
page that looks like this I realize that the  

00:27:10.020 --> 00:27:15.360
words are a little small but you can see 
the at the top you have a search by keyword  

00:27:16.200 --> 00:27:20.040
and then there are a number of ways that you 
can filter studies to try to get to the kind  

00:27:20.040 --> 00:27:24.120
of studies that are for particular interest.
So you could filter by the date that the  

00:27:24.120 --> 00:27:29.400
study was published, by the outcomes that 
were examined, by the target population,

00:27:29.400 --> 00:27:34.560
by the geographic setting, or by the evidence 
rating, which, as Chris mentioned, that's the  

00:27:35.100 --> 00:27:41.700
ratings of this credibility of the findings, so if 
you want to focus on only the most credible High  

00:27:41.700 --> 00:27:46.860
ratings or you can also look at Moderate or Low 
rated depending on your particular interest and  

00:27:46.860 --> 00:27:54.840
how much evidence actually happens to be available.
So in the next slide we'll see a hypothetical search  

00:27:54.840 --> 00:28:00.180
suppose that Teri comes in, searches by training 
completion because that's what she's interested in, 

00:28:01.740 --> 00:28:07.080
and you'll see that the search for studies 
will turn up not just relevant studies but  

00:28:07.080 --> 00:28:11.520
also research syntheses that summarize evidence 
from across the research literature on the topic  

00:28:12.180 --> 00:28:17.280
and in this case the first search result is 
a Community College Synthesis which seems  

00:28:17.280 --> 00:28:21.960
very relevant given where Teri is coming 
from as a community college administrator.

00:28:24.120 --> 00:28:29.280
You'll see that in the words there or in 
the description that the findings mention  

00:28:29.280 --> 00:28:34.200
educational outcomes, persistence outcomes, which 
is what we're interested in, so let's click on  

00:28:34.800 --> 00:28:37.980
the Community College Synthesis 
link there to see more

00:28:41.040 --> 00:28:45.120
and the next slide shows you what happens 
if you click on that link you'll see a  

00:28:45.120 --> 00:28:49.080
page like this there's a landing page 
for every research synthesis at CLEAR

00:28:49.620 --> 00:28:54.060
which provides a bit of information 
from the synthesis. It also provides  

00:28:54.060 --> 00:29:00.180
links to two PDFs in this case there's 
a synthesis and a supplement. Next slide.

00:29:03.540 --> 00:29:08.040
The synthesis provides an overview of findings 
from CLEAR's review of the research in the area  

00:29:08.040 --> 00:29:14.280
and then the supplement provides additional 
study-specific detail on the studies that  

00:29:14.280 --> 00:29:19.440
were viewed reviewed on the topic. Now let's 
click the link to the research synthesis  

00:29:20.400 --> 00:29:26.760
which will bring us to a PDF which at the top 
looks like this at the very beginning it talks  

00:29:26.760 --> 00:29:34.140
the the synthesis it'll describe the scope of 
what was covered in the review and then below  

00:29:34.140 --> 00:29:38.820
that we'll start listing the interventions that 
were examined by literature in this topic area.

00:29:41.640 --> 00:29:46.500
On the next slide, we can see a little bit 
of information from that introduction that  

00:29:46.500 --> 00:29:51.780
describes the scope. So you can see 
that it describes the years of  

00:29:51.780 --> 00:29:58.020
research that were covered, in this case 1994 
to 2019. And it just notes the fact that  

00:29:58.020 --> 00:30:03.000
this synthesis includes only High and Moderate 
rated studies so it excludes Low rated studies. 

00:30:04.380 --> 00:30:10.080
And syntheses will vary in cases where 
there's a smaller research literature, we  

00:30:10.080 --> 00:30:13.800
may also include Low rated studies. But when 
there's a large research literature, we have  

00:30:13.800 --> 00:30:17.940
the luxury of focusing on the more highly 
rated studies, the more credible studies.

00:30:20.520 --> 00:30:25.020
We'll see in the next slide that if you 
scroll down a research synthesis and  

00:30:25.020 --> 00:30:31.320
you can see a full table of the interventions 
that were examined in the topic area.

00:30:32.760 --> 00:30:36.600
Some of these are whole program models 
like career pathways and work-based training.

00:30:36.600 --> 00:30:42.240
That's beyond what Teri is looking for,
but if we're looking... interested in things  

00:30:42.240 --> 00:30:46.440
we can do to improve completion in existing 
training programs rather than implementing  

00:30:46.440 --> 00:30:51.060
new kinds of training programs, then the 
four at the top look potentially relevant.

00:30:54.420 --> 00:30:58.080
And if we scroll further down the 
table, we can see a summary of what  

00:30:58.080 --> 00:31:03.960
research is available and has found on 
each of those four topics. Next slide.

00:31:08.640 --> 00:31:12.660
So when evidence synthesis will include tables 
that look like this one that concisely summarize  

00:31:12.660 --> 00:31:18.900
what studies of an intervention have found each 
row or a particular... sorry each row is for  

00:31:18.900 --> 00:31:25.740
a particular intervention and the cells in a 
row count the study findings. Next slide, please.

00:31:28.740 --> 00:31:33.540
The findings are provided by whatever 
outcome categories the topic area focuses on. 

00:31:34.080 --> 00:31:40.020
In the case of a community college synthesis,
those are first, education and skill gains, second,  

00:31:40.020 --> 00:31:46.080
earnings and wages, and third, employment. 
Each vertical panel in the table presents findings  

00:31:46.080 --> 00:31:52.560
for a given outcome domain and you see the 
individual cells are color coded. The counts  

00:31:52.560 --> 00:31:56.460
in the green cells note how many studies of an 
intervention found evidence of favorable impacts  

00:31:56.460 --> 00:32:02.040
or outcomes in the outcome domain. Counts in the 
red cells note how many studies found unfavorable  

00:32:02.040 --> 00:32:08.280
impacts. Counts in gray show how many studies had 
null findings, that is there weren't any studies  

00:32:08.280 --> 00:32:12.240
that were statistically significant or weren't 
any findings that were statistically significant.

00:32:13.140 --> 00:32:16.920
And then the counts in the yellow show how 
many studies has mixed findings which means  

00:32:16.920 --> 00:32:20.040
there were some favorable and some 
unfavorable findings in that domain.

00:32:22.920 --> 00:32:28.200
We can see in the next slide that when we 
look at the education and skill gains panel  

00:32:29.100 --> 00:32:35.520
that the four interventions we're looking
at here have many studies that had  

00:32:35.520 --> 00:32:41.400
findings that had favorable 
impacts. And in the next slide, 

00:32:42.840 --> 00:32:48.300
if we look at accelerated learning 
in particular, we can that may pique  

00:32:48.300 --> 00:32:53.160
your interest to dig into more deeply. It does 
seem to have a large body of favorable findings. 

00:32:53.160 --> 00:32:58.680
In this case, 10 out of the 14 studies found 
favorable effects on educational skill gains.

00:33:02.040 --> 00:33:04.740
And so if we want to find a little 
bit more information on accelerated  

00:33:04.740 --> 00:33:08.820
learning and what the findings of these 
studies were the research synthesis will  

00:33:08.820 --> 00:33:13.260
provide a little more narrative information 
if we scroll down, so we go to the next slide,

00:33:16.140 --> 00:33:21.120
we can see that the research synthesis 
notes specifically that accelerated learning  

00:33:21.120 --> 00:33:24.840
interventions were found to have increased 
rates of course enrollment and completion  

00:33:25.380 --> 00:33:31.320
as well as having improved rates of degree or 
certificate completion. That sounds like the  

00:33:31.320 --> 00:33:35.880
kind of thing that Teri's looking for so that's 
promising. The synthesis also notes that studies  

00:33:36.840 --> 00:33:40.800
of accelerated learning 
interventions typically serve individuals in  

00:33:40.800 --> 00:33:45.480
developmental learning courses who traditionally 
have lower rates of academic persistence and  

00:33:45.480 --> 00:33:49.380
degree completion and that sounds like it may 
align with the kinds of students that Teri  

00:33:49.380 --> 00:33:56.220
has in mind to support. So that's all promising, 
but suppose we want to dig into specific studies  

00:33:56.220 --> 00:34:00.060
of accelerated learning interventions. 
If we want to do that we can go back to  

00:34:00.060 --> 00:34:05.760
the landing page for the Community College 
Synthesis, which we can see on the next slide

00:34:08.460 --> 00:34:14.100
is the second link right there so 
we click on that we'll be taken  

00:34:14.100 --> 00:34:18.420
to a document that looks like 
what we see in the next slide.

00:34:21.000 --> 00:34:27.480
Typically... the supplement 
will organize studies by the intervention  

00:34:27.480 --> 00:34:31.080
type so in this case we've gone down to 
the part of the supplement that has the  

00:34:31.080 --> 00:34:35.520
list studies of accelerated learning we 
can see here the first study in that list  

00:34:37.260 --> 00:34:43.800
and as you see here the supplement provides 
study-specific information including a full  

00:34:43.800 --> 00:34:49.500
citation, the findings by outcome domain, the 
causal evidence rating of the study, and also  

00:34:49.500 --> 00:34:53.760
a link to the full study profile so you can 
see additional information on that study.

00:34:57.720 --> 00:35:03.000
And at this point, I am going to turn 
things over to Rhaia to talk a little  

00:35:03.000 --> 00:35:05.340
more about finding individual studies of interest.

00:35:12.240 --> 00:35:14.460
Thank you, Andrew. Can we go 
to the next slide, please?

00:35:17.160 --> 00:35:19.980
You could get straight to relevant study 
profiles through the links in the supplement  

00:35:19.980 --> 00:35:24.240
as Andrew noted but I also want to go back 
to the search function available through  

00:35:24.240 --> 00:35:28.560
CLEAR to walk through some of the other 
features of find studies of interest. 

00:35:28.560 --> 00:35:32.460
And because clear review studies on an ongoing 
basis it's always worth checking whether new  

00:35:32.460 --> 00:35:37.260
studies have been reviewed that have not yet been 
incorporated into a synthesis. Next slide, please.

00:35:41.640 --> 00:35:46.680
Here let's search specifically for accelerated 
learning. Note that the images to the right is a  

00:35:46.680 --> 00:35:52.240
magnification of the circled part of the filtering 
menu to make it a little bit easier to read. Next slide, please.

00:35:52.280 --> 00:35:56.940
In addition to searching by the 
name of the intervention let's filter by the  

00:35:56.940 --> 00:36:01.800
outcome domain of interest, education gains, and 
include only High or Moderate rated studies to  

00:36:01.800 --> 00:36:07.740
focus on the most credible studies. Next slide, please. 
As you can see on the right side of the  

00:36:07.740 --> 00:36:12.300
screen, this pulls up 16 results. You can save 
them by using the export results function.  

00:36:12.960 --> 00:36:20.100
Next slide. Thanks, the export studies function 
produces a spreadsheet file with a row for each  

00:36:20.100 --> 00:36:24.360
item in the search result, including the sole 
citation, all of the characteristics that you  

00:36:24.360 --> 00:36:28.620
can filter on, and links to the full text of the 
study if they're available. Next slide, please.

00:36:31.200 --> 00:36:33.600
But let's go back to the search 
results and click on the first  

00:36:33.600 --> 00:36:37.620
result to see CLEAR's profile 
page for that study. Next slide.

00:36:39.720 --> 00:36:44.340
Here we can see the study's profile page the box 
at the top left summarizes the study's findings  

00:36:44.340 --> 00:36:49.620
by outcome domain. In this case, the study only 
examined outcomes within one outcome domain  

00:36:49.620 --> 00:36:53.880
defined by CLEAR. Findings are characterized 
as favorable which again means at least some  

00:36:53.880 --> 00:36:59.160
statistically significant favorable outcomes and 
no unfavorable outcomes and the study received a  

00:36:59.160 --> 00:37:04.380
moderate causal evidence rating. Again, that means 
CLEAR assesses that the studies findings likely  

00:37:04.380 --> 00:37:09.480
reflect the causal impact of the intervention 
although it may reflect some other factors as  

00:37:09.480 --> 00:37:14.040
well. The blue button provides a link to the full 
text if it's available to help you get additional  

00:37:14.040 --> 00:37:18.780
study details that you might want beyond what's 
available in the study profile. Next slide, please.

00:37:20.760 --> 00:37:25.080
Further down the profile, we find more details 
of the intervention the study that can help us  

00:37:25.080 --> 00:37:29.400
understand the applicability. The sections below 
highlight real information that gives you a sense  

00:37:29.400 --> 00:37:34.080
of the intervention without reading the full article. 
So again, for example, this intervention served  

00:37:34.080 --> 00:37:39.240
college students from summer 1999 to fall 2010 
and condensed developmental reading and writing  

00:37:39.240 --> 00:37:43.260
English courses from two semesters into one 
to create the accelerated reading program.

00:37:43.260 --> 00:37:47.400
So the treatment took eight credits worth 
of classes over the course of one semester  

00:37:47.400 --> 00:37:51.600
whereas the control group attend the classes 
as they were originally designed four credits  

00:37:51.600 --> 00:37:55.860
worth of classes in one semester. Outcomes were 
measured at one year after the intervention  

00:37:55.860 --> 00:38:01.560
started however you also learn the important 
caveat that the self-selected students...  

00:38:01.560 --> 00:38:06.720
the students self-selected to the accelerated program 
or the business as usual program. Next slide, please.

00:38:10.980 --> 00:38:15.060
Further down in just a few bullets you 
learn the study's key findings. Here we  

00:38:15.060 --> 00:38:18.060
learn that the students who were in the 
accelerated program were significantly  

00:38:18.060 --> 00:38:21.480
more likely to enroll and complete 
the college courses and increase  

00:38:21.480 --> 00:38:25.560
credit accumulation with higher level 
GPAs than those in the comparison group.

00:38:25.560 --> 00:38:29.640
However, the study profiles have additional 
layers of detail to help make sense of studies  

00:38:29.640 --> 00:38:34.500
findings. Here we are reminded that the treatment 
group self-selected into the accelerated program  

00:38:34.500 --> 00:38:39.540
meaning that the estimated impacts seen between 
the treatment and control may have at least in  

00:38:39.540 --> 00:38:44.460
part have something to do with that. In other words, 
students who self-selected into the accelerated  

00:38:44.460 --> 00:38:49.380
program may already be more confident in their 
skills or be more motivated to get through their  

00:38:49.380 --> 00:38:53.880
degree program faster meaning that the students 
who self-selected into the treatment might have  

00:38:53.880 --> 00:38:58.800
had higher enrollment and completion rates with 
or without the intervention. Next slide, please.

00:39:00.900 --> 00:39:07.380
So as we think about what Teri the Community 
College administrator might do, through her initial  

00:39:07.380 --> 00:39:11.640
search for CLEAR, she has found lots of research 
on multiple strategies from the synthesis report  

00:39:12.300 --> 00:39:16.680
she has learned about linked learning 
communities, paid performance incentives, and  

00:39:16.680 --> 00:39:21.240
the intervention we focus on in just now, 
the accelerated learning program. These  

00:39:21.240 --> 00:39:25.200
interventions have been evaluated in studies 
that CLEAR has reviewed and found to be credible  

00:39:25.200 --> 00:39:29.400
and those studies have frequently found those 
interventions to be effective increasing students  

00:39:29.400 --> 00:39:33.720
course completion rates. So each seems like a 
promising intervention for further exploration.

00:39:34.800 --> 00:39:38.640
So from here there are a few steps that Teri 
may want to take if she wants to move forward  

00:39:38.640 --> 00:39:43.920
with implementing an intervention such as the 
accelerated learning course. First, she could look  

00:39:43.920 --> 00:39:50.280
more closely at individual studies 
and other publicly available resources to learn  

00:39:50.280 --> 00:39:55.860
more details about what it might take to design 
and implement an accelerated learning course.

00:39:56.880 --> 00:40:00.720
For example, this may include learning 
what staff expertise is needed or if  

00:40:00.720 --> 00:40:04.440
there are any associated costs for 
implementation. She may also want to  

00:40:04.440 --> 00:40:08.220
look back at the original study to see how 
large the impacts are and think critically  

00:40:08.220 --> 00:40:12.240
about how they compare with the likely 
cost of implementing the intervention.

00:40:13.020 --> 00:40:17.280
Equipped with this information, Terry will be in a 
strong position to discuss with her colleagues  

00:40:17.280 --> 00:40:21.960
what she has found, the pros and cons of different 
options, and be able to support the program's  

00:40:21.960 --> 00:40:27.360
decision with what strategies or strategy to 
implement. And go ahead to the next slide, please.

00:40:29.760 --> 00:40:34.680
So again to recap, some further steps she 
can take is to review individual studies  

00:40:34.680 --> 00:40:39.060
and publicly available resources, gather more 
information on designing and implementing the  

00:40:39.060 --> 00:40:44.160
intervention, potentially read the full 
article from the CLEAR profile page and  

00:40:44.160 --> 00:40:48.180
discuss pros and cons with her colleagues.
And now I'd like to turn it back to Megan  

00:40:48.180 --> 00:40:51.480
and the panelists to discuss real 
life experiences using CLEAR. Megan?

00:40:52.680 --> 00:40:58.980
Great, thank you so much, Rhaia. And as we turn 
our attention to hearing from our panelists,

00:40:59.820 --> 00:41:04.980
Lisa and Ben, I want to just take a moment and 
remind folks to go ahead and use the chat for  

00:41:04.980 --> 00:41:11.220
any questions that you have and we will cover 
them in a few minutes once we hear more from  

00:41:11.220 --> 00:41:17.640
our panelists. So with that I would invite Lisa 
and Ben to go ahead and turn on your cameras.

00:41:18.780 --> 00:41:24.180
I'm going to start with Lisa and then turn 
to Ben to ask a couple of questions just about  

00:41:24.180 --> 00:41:32.400
how you use CLEAR or have used CLEAR in your work.
So, Lisa, thank you so much for being with us  

00:41:32.400 --> 00:41:37.860
today. I wonder if you might just share a little 
bit about how you've used CLEAR in  

00:41:37.860 --> 00:41:40.680
the past and perhaps give a few examples? 

00:41:41.920 --> 00:41:45.780
Alright. Thanks, Megan. 
Well it's so funny because I think  

00:41:45.780 --> 00:41:52.740
I am Teri in that example. That's pretty much 
very similar to my journey. So good afternoon  

00:41:52.740 --> 00:41:58.200
to everyone. I'm Lisa Salazar with the City 
of Los Angeles and in the introductions shared  

00:41:59.340 --> 00:42:05.940
that my connection to CLEAR really came 
through my work in the mayor's office quite  

00:42:05.940 --> 00:42:14.160
often we are asked for ideas on how best to 
serve folks with multiple barriers to employment  

00:42:14.160 --> 00:42:24.240
and of course you know having just 20+
years of experience in the field in direct  

00:42:24.240 --> 00:42:32.340
program, I have all my good ideas about what 
works and what doesn't work but you know in  

00:42:32.940 --> 00:42:40.740
in a mayor's office and when sitting in 
front of other elected officials, that's all nice  

00:42:40.740 --> 00:42:48.300
to know that I have my own perspective, but it's 
such a much stronger argument when I'm able to  

00:42:49.200 --> 00:42:55.620
cite actual examples of what 
is working in the field and coming  

00:42:55.620 --> 00:43:03.300
forward with research-based evidence. 
So that is really how I came to know CLEAR

00:43:04.680 --> 00:43:11.640
and it's just... it's just so 
easy to use so instead of me just saying  

00:43:11.640 --> 00:43:17.700
here's an approach to how we can better serve 
or connect folks experiencing homeless or involved  

00:43:17.700 --> 00:43:23.340
with the justice system or opportunity youth 
to employment opportunities, apprenticeships...  

00:43:24.660 --> 00:43:31.560
I would just go to CLEAR and find examples 
of what's working across the nation and borrow, 

00:43:31.560 --> 00:43:40.800
I'll say, those best practices and start to 
formulate sort of our own program design work on  

00:43:40.800 --> 00:43:47.580
that evidence that's out there and then also 
sharing with the mayor and our city council  

00:43:47.580 --> 00:43:55.080
members how these approaches have worked in other 
cities and having that data behind it.

00:43:55.080 --> 00:44:03.540
So that's how I've come to know CLEAR
and have been using CLEAR and are a big fan and  

00:44:03.540 --> 00:44:10.080
I have to say I have no background in research and 
evaluation. I am not a data scientist. I'm really a  

00:44:10.080 --> 00:44:16.320
workforce practitioner, and I have worked at the 
community level for a number of years so I'm  

00:44:16.320 --> 00:44:22.740
more of a program implementer I'm more of 
the person that CLEAR folks evaluate. So again,  

00:44:22.740 --> 00:44:25.200
Megan, thank you for the opportunity. 
It's great to be here.

00:44:25.880 --> 00:44:28.400
 Thanks so, much Lisa.
 I wonder if you could

00:44:28.440 --> 00:44:34.380
build on something you just said about how 
you're not a data scientist, right? So can you tell  

00:44:34.380 --> 00:44:39.480
me a little more about like from that perspective 
what do you particularly like about CLEAR?

00:44:42.360 --> 00:44:49.740
I would say two things I really like 
how easy it is to use. I started using CLEAR

00:44:50.640 --> 00:44:56.160
before there were webinars about how 
to use CLEAR or that I was aware of  

00:44:56.160 --> 00:45:01.320
webinars. It's very intuitive. It's very 
easy to just click around and filter.

00:45:02.640 --> 00:45:09.360
It's also... it's great in how the 
information is presented. It's not so dense  

00:45:10.680 --> 00:45:17.820
that it's, you know, it's very easy to understand 
especially for me the summaries because that  

00:45:18.420 --> 00:45:22.260
you know, I hate to admit, is about how 
much time I'm willing to spend to just  

00:45:22.260 --> 00:45:27.480
kind of understand what a particular study 
is about there's just looking at the summaries.

00:45:28.980 --> 00:45:32.940
The second thing I would say is 
that there's just so much information  

00:45:34.680 --> 00:45:39.540
in CLEAR. You know all 
of the filters not only by  

00:45:40.560 --> 00:45:48.360
the types of workers and their particular 
barriers but also the number of approaches  

00:45:49.740 --> 00:45:58.020
and supportive services that are in there. 
Before I knew about clear I would just Google  

00:45:58.020 --> 00:46:05.340
a particular topic and just find random 
reports and then have to read like 28-page  

00:46:05.340 --> 00:46:11.460
reports. With CLEAR, you can just like go in 
and just narrow down your search and pull up  

00:46:11.460 --> 00:46:18.360
everything that's in there and then 
you know and then I choose by the rating system  

00:46:18.360 --> 00:46:25.740
which ones to read so if it's got Low impact 
I'm probably not going to read it unless it's  

00:46:25.740 --> 00:46:31.080
kind of the only thing out there it's such like 
a niche study but you know those are a couple  

00:46:31.080 --> 00:46:34.400
things of what I really like 
in particular about CLEAR.  

00:46:34.400 --> 00:46:36.400
Thanks, that's really helpful. 

00:46:36.560 --> 00:46:42.660
I wonder if you've thought ahead at all about,
you know, how you might use it in the future?  

00:46:42.660 --> 00:46:47.820
You know, do you think you'll continue using 
it in the ways you've described? Are there any  

00:46:48.780 --> 00:46:53.560
you know new ways that you're thinking you 
might want to try to use it?   

00:46:53.560 --> 00:47:01.468
Yeah, sure. So I'd say three ways. 
Of most recent... in the city

00:47:01.468 --> 00:47:05.160
in the Workforce Development Area and also in  

00:47:05.160 --> 00:47:13.320
Youth Development, we are trying to attach research 
and evaluation to all of the grant-making that  

00:47:13.320 --> 00:47:20.400
we do out of the city. So again, not being a data 
scientist, how we are also looking at CLEAR is and  

00:47:20.400 --> 00:47:26.580
understanding the evaluation process, understanding 
the different types of evaluations that are 

00:47:26.580 --> 00:47:34.680
out there, what it takes to have a high impact on 
an evaluation. We've really just been learning  

00:47:35.640 --> 00:47:40.500
more about this field from reading the 
reports that are in there and understanding  

00:47:40.500 --> 00:47:46.980
what it takes to have a very strong 
report that would encourage and I think  

00:47:48.060 --> 00:47:54.780
help our elected officials make decisions about, 
you know, whether or not they should fund something  

00:47:54.780 --> 00:48:00.420
or continue to find something. The other was 
just this morning, I have a new staff on board  

00:48:01.740 --> 00:48:07.260
and I just said do you know about CLEAR?
And you know this is a very educated person  

00:48:08.160 --> 00:48:11.820
and she did not know about it and I said take 
a look at it and let me know what you think and  

00:48:11.820 --> 00:48:16.680
she was like wow this is great. I've bookmarked 
about seven tabs that I want to go back and read. 

00:48:17.760 --> 00:48:23.460
So I will continue to share CLEAR with my 
staff because you know for folks who work in  

00:48:23.460 --> 00:48:33.900
in a government setting, you know,
I could try to procure the help of a  

00:48:33.900 --> 00:48:41.700
consultant to help me put together an evaluation 
or find evidence-based practices but that  

00:48:41.700 --> 00:48:46.740
would take like six months, eight months for me 
to do that. I don't need to do that anymore I can  

00:48:46.740 --> 00:48:53.880
just go to CLEAR and CLEAR provides enough 
information for me to just grab onto and and  

00:48:53.880 --> 00:48:57.960
it's just great that way. I mean we're saving 
tons of time and tons of money by using CLEAR.

00:49:00.240 --> 00:49:03.080
That's really great to hear, Lisa.
And I love -- hearing you say this like this is I should say  

00:49:03.080 --> 00:49:05.656
And they didn't tell me to say this! 
Like this is...

00:49:05.880 --> 00:49:12.180
Yes, I didn't tell you to say it! But it's really 
great to hear how you and your staff are using it  

00:49:12.720 --> 00:49:18.960
and you know how you're thinking about using 
it going forward too. So, thank you so much for  

00:49:18.960 --> 00:49:23.760
those insights. If folks have questions for 
Lisa, feel free to put them in the chat and  

00:49:23.760 --> 00:49:29.160
we'll turn to Ben to hear how he's been using 
it, which I think is a little bit different. 

00:49:30.600 --> 00:49:35.580
Right. So Ben, I wonder if you could 
share a little more about how you've been  

00:49:35.580 --> 00:49:39.399
using CLEAR and provide a few examples for 
us?

00:49:39.588 --> 00:49:43.476
Sure. So, the Texas Workforce Commission  

00:49:43.800 --> 00:49:50.520
has been exploring and developing an approach 
to grant-making that incorporates an evidence  

00:49:50.520 --> 00:49:57.780
evaluation so that we are providing additional 
points in our applications to those programs  

00:49:57.780 --> 00:50:03.540
which are evidence-based and have evidence of 
effectiveness. When we were developing that  

00:50:03.540 --> 00:50:11.340
that grant-making process, we were really looking 
for a system to evaluate the evidence of proposals  

00:50:12.360 --> 00:50:18.480
given that, like I think most people, we don't 
have a dedicated research team or a dedicated

00:50:19.200 --> 00:50:24.300
analysis team that can go in and do these and 
evaluate all the different studies that are out  

00:50:24.300 --> 00:50:31.500
there and everybody's proposal and so on. 
And we found CLEAR along with a couple of other 

00:50:32.100 --> 00:50:41.760
federal databases or federal clearinghouses 
to be a really great way for us to find studies  

00:50:41.760 --> 00:50:47.880
that other people could use to explain what 
they're doing. So basically, we can ask applicants  

00:50:48.780 --> 00:50:54.060
please go and find a study that you are 
replicating, that your evidence is replicating, 

00:50:54.060 --> 00:51:00.000
and then we can use the CLEAR evaluation 
of that study to say that yes, your approach  

00:51:00.000 --> 00:51:04.200
has evidence base or not, and we can we 
go into some depth about how is your

00:51:04.200 --> 00:51:10.080
proposal like this study or different from 
the study. But we were very pleased with  

00:51:10.080 --> 00:51:14.940
CLEAR's approach because the primary goal and 
the primary thing that we were thinking about  

00:51:15.840 --> 00:51:21.240
during this process was well, obviously the 
primary thing we think about was finding good  

00:51:21.240 --> 00:51:27.180
evidence, the primary thing from a grant-making 
point of view is that we didn't want to scare  

00:51:27.180 --> 00:51:31.500
off any applicants. We wanted something that was 
very user friendly we wanted something that was  

00:51:31.500 --> 00:51:36.600
easy to understand and we wanted something 
that people could use and review and access  

00:51:37.560 --> 00:51:42.180
quickly so that it was not a significant 
additional burden on top of the grant-making  

00:51:42.180 --> 00:51:51.300
process which occasionally people have complained 
is tiring. So we are looking... we were looking for  

00:51:51.300 --> 00:51:56.640
something that was easy to understand for people 
that weren't data scientists. There's always been  

00:51:56.640 --> 00:52:01.920
you know meta studies and so on they could go out 
and find a lot of information about a specific  

00:52:01.920 --> 00:52:09.180
topic but the ability for someone like me who is 
also excuse me like Lisa not a data scientist

00:52:10.140 --> 00:52:15.720
to go out and quickly review and be able to say 
yes, this is a High quality study, this is not  

00:52:15.720 --> 00:52:22.260
a High quality study you know whatever we want 
to interpret was very convenient and very  

00:52:23.400 --> 00:52:28.740
user friendly and that was sort of what 
was carried out was we started to talk to our  

00:52:28.740 --> 00:52:35.280
grantees about how easy it was to use this 
system. So we were looking at CLEAR

00:52:36.060 --> 00:52:41.460
not strictly for internal use, obviously these 
are TWC-funded programs that we are trying to  

00:52:41.460 --> 00:52:45.660
increase our evidence base, but we were looking for 
something that external partners, some of them very  

00:52:45.660 --> 00:52:52.920
small, could still look at and quickly find results 
and to then use this as a way to encourage them to  

00:52:52.920 --> 00:52:59.520
bring evidence into their programming and their 
own internal processes, and that's I think the  

00:52:59.520 --> 00:53:04.560
other side of this is that yes, we want to increase 
our evidence base for our programs, I think, but  

00:53:04.560 --> 00:53:09.660
we also want to increase the understanding 
of evidence and the use of evidence in the  

00:53:09.660 --> 00:53:17.580
entire ecosystem in Texas that that 
we are finding ways to support the development of  

00:53:17.580 --> 00:53:23.640
evidence for programs. And our system doesn't just 
have the High and Moderate that CLEAR has, we also  

00:53:23.640 --> 00:53:31.680
have some lower stages that we developed, excuse me, 
based on our analysis of what our grantees were  

00:53:31.680 --> 00:53:37.740
doing so that we could genuinely say yes, you 
know, there are intermediate stages  

00:53:37.740 --> 00:53:42.360
between being a completely new program that has 
no evidence and has never been tested before  

00:53:42.360 --> 00:53:49.440
to a Moderate study that has gotten through the 
entire process of a complete comprehensive study.  

00:53:49.440 --> 00:53:55.620
But obviously the ultimate goal is always to 
move people upwards and we sort of think of our  

00:53:57.120 --> 00:54:01.080
scoring system as this sort of chain so 
that you know maybe this year you can only  

00:54:01.080 --> 00:54:06.540
get the experience and testimonial stage 
but you can move on to data collection,  

00:54:06.540 --> 00:54:12.720
you can move on to data analysis, you can move 
on to using an actual studied technique.

00:54:16.320 --> 00:54:23.146
That's really interesting, you know, and 
I think there's a lot to unpack there.  

00:54:23.665 --> 00:54:29.220
I love hearing how it's a little bit different from 
how Lisa's using it, right? I think it shows that  

00:54:29.220 --> 00:54:35.160
there's a variety of ways to use the resources 
and the references and the other materials that  

00:54:35.160 --> 00:54:42.600
we have in CLEAR. I'm wondering if you can talk 
about what you particularly like about CLEAR again  

00:54:42.600 --> 00:54:46.332
sort of as a non-data scientist.

00:54:46.568 --> 00:54:55.380
I think the two things in 2019 when I reviewed a lot of 
cleaning houses in 2019, I think one of the things that I really  

00:54:55.380 --> 00:55:03.420
liked about CLEAR was that it was very easy to 
quickly make a decision about whether a study had  

00:55:04.500 --> 00:55:09.540
what you wanted. You know as Lisa was saying 
previously you would Google a study and then  

00:55:09.540 --> 00:55:16.800
you had to read it you know you just quickly 
skim the uh synopsis and the methodology maybe  

00:55:16.800 --> 00:55:20.280
or you know something like that and try and get a 
sense of this is what I want but if you can't  

00:55:20.280 --> 00:55:24.480
tell for sure then you've got to wade through, 
you know, however long pages and again I'm not a  

00:55:24.480 --> 00:55:32.100
data scientist so sometimes the 
papers are a little bit daunting. But with this  

00:55:32.100 --> 00:55:36.600
with the ability to say, you know, here is what they 
were testing, here's you know what they were trying  

00:55:36.600 --> 00:55:42.720
to do, and then here's whether it was done you know 
to the High or the Moderate standard and just be  

00:55:42.720 --> 00:55:48.480
able to say yes, this is, they know they were
looking at homeless veterans, just let's  

00:55:48.480 --> 00:55:55.380
say, and this study had a Moderate effect. We can 
immediately say well that's in if we want to talk  

00:55:55.380 --> 00:56:00.420
about homeless veterans and it might not be, 
but it also gives us the opportunity to say  

00:56:00.420 --> 00:56:05.880
well maybe I'm not working with homeless veterans 
maybe I'm working with um people who are homeless  

00:56:05.880 --> 00:56:11.340
but have some other background but maybe there's 
some interesting information here or conversely  

00:56:11.340 --> 00:56:17.160
if I'm just looking at veterans maybe I want to I 
want to see you know this is addressing homeless  

00:56:17.160 --> 00:56:22.560
veterans but it's a well-done study so it may also 
give me some insight into the veterans' challenges.  

00:56:22.560 --> 00:56:30.000
And so it's that cross-referencing and the 
breadth of data but also just the fact that it's  

00:56:30.000 --> 00:56:36.180
presented in the these little bite-sized chunks 
that I can quickly, you know, interpret because  

00:56:36.180 --> 00:56:43.920
again as Lisa said, you know, we all have lots going 
on. We can't read 30 papers. And so we would  

00:56:43.920 --> 00:56:48.840
be able to do a quick study and say well here's 
eight studies that meet a Moderate or a High  

00:56:48.840 --> 00:56:54.000
standard and we can go through and pull those and 
then I can dedicate my time to reading those  

00:56:54.000 --> 00:57:00.000
rather than reading maybe seven or eight papers 
that that aren't particularly valuable.

00:57:00.000 --> 00:57:05.280
But I wouldn't know that until after the fact.
So I think that's the biggest thing for me.

00:57:06.960 --> 00:57:13.260
Yeah, you know, that's certainly something 
that CLEAR tries to do right to make it easier  

00:57:13.260 --> 00:57:20.880
to pick what you want when you need it.
So, you've been working on this evidence-based grant  

00:57:20.880 --> 00:57:27.960
making initiative for a while, right? Maybe about a 
year or more. How's it going so far? Do you have  

00:57:27.960 --> 00:57:32.755
impressions, things you're hearing from your 
grantees that you'd like to share?   

00:57:32.755 --> 00:57:41.520
Yeah, so we launched our first evidence-based grant 
making request for applications in 2021.  

00:57:42.420 --> 00:57:48.420
And we have since then launched a 
further five so we've launched six now.  

00:57:49.620 --> 00:57:55.740
And we continue to develop new 
ones but we did take the time to  

00:57:56.760 --> 00:58:07.020
conduct a webinar with the grantees for our 
first 2021 initial grant and we also took the  

00:58:07.020 --> 00:58:11.460
opportunity this was this is unique for us we 
took the opportunity because this was a grant  

00:58:11.460 --> 00:58:15.720
opportunity that had been offered in the 
past, we were able to reach out to some  

00:58:16.620 --> 00:58:22.980
potential eligible applicants who chose not to 
apply. And so we were able to hear both what  

00:58:22.980 --> 00:58:30.472
the people who applied thought about it and 
then also what... why people chose not to apply.

00:58:30.472 --> 00:58:38.040
We had decided in 2019, actually I think 
it was very late 2018, but we really buckled 

00:58:38.040 --> 00:58:45.360
down and started working in 2019, on designing this 
system and we were going to publish our very first...  

00:58:45.360 --> 00:58:52.080
the initial plan was to publish our very first
evidence-based grant-making project in March of  

00:58:52.080 --> 00:59:00.480
2020. Due to external reasons, we decided not to 
do that and postponed it a little while but 

00:59:01.560 --> 00:59:05.580
we were really pleased when we got back together 
with our grant he's at the at the one year mark.

00:59:05.580 --> 00:59:10.800
We spoke with them and got some information 
from them about how they were doing and how they  

00:59:10.800 --> 00:59:17.700
felt experienced that the application process 
and we were really pleased with how easy  

00:59:18.360 --> 00:59:24.660
they said I mean this is the actual 
grantees saying that I mean, yes, it was more work  

00:59:24.660 --> 00:59:30.840
than not doing an evidence evaluation, but it was 
easy to do. It was easy to understand how to get  

00:59:30.840 --> 00:59:36.300
the information that we needed, and that what you 
were asking for was in line with what the federal  

00:59:36.300 --> 00:59:42.420
government has been asking for, so this was not 
some surprise. And so we were very pleased  

00:59:42.420 --> 00:59:48.060
that we were able to to see this and that we heard 
from both community colleges and from small  

00:59:48.060 --> 00:59:54.840
non-profits saying that this was this was doable 
for our staff and then while we haven't done  

00:59:54.840 --> 01:00:00.840
that same kind of detailed webinar with other 
applications, we've continued to see quality  

01:00:00.840 --> 01:00:09.480
applications and not see a significant decrease 
in applications for our results or 

01:00:10.620 --> 01:00:16.560
requests for application, and that we are seeing 
a diversity of response. One of the things that we  

01:00:16.560 --> 01:00:21.060
set out and this is less related to CLEAR, but 
just to say that we know that most grantees aren't  

01:00:21.060 --> 01:00:26.940
necessarily going to have a High or a Moderate 
study that examines their intervention. but we  

01:00:26.940 --> 01:00:31.740
want to set that up as an ambitious goal that 
you know maybe someday everybody can be moving  

01:00:31.740 --> 01:00:39.540
in that direction. And so we've continued to 
see people at all points on our scale but what was  

01:00:39.540 --> 01:00:47.700
surprising to me and I'm going to be up front 
and say that I was the most gloomy person  

01:00:47.700 --> 01:00:52.380
when our development team, when we were talking 
about what we would see from our applicants,

01:00:53.640 --> 01:01:00.540
but we actually saw in our very first one that 
we had people who had Moderate and High levels  

01:01:00.540 --> 01:01:05.580
of evidence supporting their applications and 
we've continued to see Moderate and High levels of  

01:01:05.580 --> 01:01:12.240
evidence coming up. Now again, it's not the majority 
but we are really pleased with how much  

01:01:12.900 --> 01:01:16.080
having the ability to get that 
Moderate and High rating has really  

01:01:16.080 --> 01:01:19.200
supported some of our our grantees and applicants.

01:01:22.500 --> 01:01:26.820
That's really exciting. I, you know,
that sort of culture change,  

01:01:26.820 --> 01:01:33.000
that building that culture of sort of continuous 
improvement and you know evidence-building is  

01:01:33.600 --> 01:01:39.180
you know something that does take time.
I think it reminds me a lot of what we're seeing  

01:01:40.200 --> 01:01:45.360
in the re-employment services 
and eligibility assessments  

01:01:45.960 --> 01:01:52.440
world you know RESEA for those of you 
that are working and thinking in that space too.

01:01:53.100 --> 01:02:00.060
And you know I know we've got a variety 
of folks on the phone here that are

01:02:00.060 --> 01:02:04.320
thinking about some of those things. So you 
know there's definitely some crossover lessons  

01:02:05.040 --> 01:02:09.900
there that that could be interesting 
to explore more. I wonder if you might  

01:02:09.900 --> 01:02:14.220
share how you are thinking you'll 
like to use CLEAR in the future, Ben?

01:02:15.780 --> 01:02:22.740
So I think we're always looking at the first 
thing we want to do is find ways to expand our  

01:02:22.740 --> 01:02:31.860
projects you know we fund projects that 
are not necessarily as narrowly focused  

01:02:31.860 --> 01:02:36.540
on workforce in the sense of you're taking an 
unemployed person and moving them to employment  

01:02:36.540 --> 01:02:42.240
but there's a variety of other projects that we 
do and we are continuing to look at how we can use  

01:02:42.240 --> 01:02:48.060
CLEAR data to support those you know more precise 
interventions and how we can work with  

01:02:48.060 --> 01:02:55.620
that we're also moving in the direction of 
sort of more what Lisa was talking about  

01:02:55.620 --> 01:03:01.500
that that we want to start using CLEAR internally 
for our own things not just in the evaluation of  

01:03:01.500 --> 01:03:07.920
grants but in the original development of 
requests for application, that how can we  

01:03:09.720 --> 01:03:16.680
present our applications in a way that 
promotes solid design and solid development  

01:03:16.680 --> 01:03:24.720
of these programs and how can we, you know, not 
ask people to reinvent the wheel when we... 

01:03:24.720 --> 01:03:30.360
if we already know that there is a kind of a 
gold standard out there for how to solve one  

01:03:30.360 --> 01:03:35.700
specific aspect of it, we can simply, you know,
build that into our initial design concept.

01:03:36.960 --> 01:03:45.540
We're also very interested in the overall 
research summaries that CLEAR is  

01:03:45.540 --> 01:03:49.920
doing to sort of give us a broad spectrum to 
say well where is some place where maybe we  

01:03:49.920 --> 01:03:55.080
aren't currently doing research or currently doing 
grant-making but maybe there's enough research to  

01:03:55.080 --> 01:03:59.400
say well maybe we should be doing something in 
that these are all potential opportunities  

01:03:59.400 --> 01:04:05.580
that we have in front of us and, you know,
to just add to that, one of the other things  

01:04:05.580 --> 01:04:11.040
that we've used that isn't strictly technically 
using CLEAR, but one of the things we  

01:04:11.040 --> 01:04:17.820
liked about CLEAR was the very detailed and 
easily available standards that CLEAR was using  

01:04:19.260 --> 01:04:24.840
we've had the opportunity to 
use CLEAR standards when we have hired  

01:04:24.840 --> 01:04:30.720
external evaluators to say we want you to design 
a study that will meet at least the Moderate or  

01:04:30.720 --> 01:04:37.680
maybe the High level of CLEAR evidence, just 
so that we can for example as we look at 

01:04:37.680 --> 01:04:41.820
you know as we look at our own internal research 
and our and bringing in evaluators for our own  

01:04:41.820 --> 01:04:47.040
programs to know that we're getting the kind 
of quality evaluation that we want and they  

01:04:47.040 --> 01:04:51.900
could potentially be supporting long-term other 
people's evaluations and other people's research.

01:04:53.340 --> 01:04:58.860
So those are I think the key points 
for us looking at the future of CLEAR

01:04:58.860 --> 01:05:06.420
with us and again also finding ways to expand 
into other categories, how can we support youth  

01:05:06.420 --> 01:05:11.700
interventions, how can we support other ones that 
may not have that easy connection between  

01:05:12.780 --> 01:05:18.180
the intervention and the final outcome that 
we want. If you're intervening with a 14 year  

01:05:18.180 --> 01:05:22.740
old it's hard to say and then they got a job 
and we can say with confidence that there's  

01:05:22.740 --> 01:05:28.620
a connection there. So just finding ways to 
look at these much more complex studies than  

01:05:28.620 --> 01:05:33.960
Texas could ever do in the short term to see 
if we can find some guidance and support there.

01:05:35.760 --> 01:05:42.180
That's great, Ben. Thank you for sharing that.
Before we move to the next slide there was  

01:05:42.180 --> 01:05:47.280
one question that came in that I think is best 
to answer just right now while we have you  

01:05:47.940 --> 01:05:53.880
here and this is really this is for you, 
Ben, if you could tell us a bit more about  

01:05:53.880 --> 01:05:59.640
the kinds of programs that were the focus of 
your evidence-based grants. The question is  

01:05:59.640 --> 01:06:06.420
were they all service providing 
programs or were some research projects?

01:06:07.260 --> 01:06:17.220
So far they have all been service projects.
We've done... our pilot program was the  

01:06:17.220 --> 01:06:22.260
building construction trades RFA that 
we have done and it's basically an  

01:06:22.260 --> 01:06:28.740
intervention to help people go from under or 
unemployment into the skilled trades and the  

01:06:28.740 --> 01:06:33.780
construction industry and the different varieties 
of interventions and support services and and so  

01:06:33.780 --> 01:06:41.700
on that could be offered there. We've also done 
some studies on adult education services  

01:06:41.700 --> 01:06:50.340
and some additional ones for other 
service design programs like that.

01:06:52.020 --> 01:06:58.980
Thanks, that's really helpful. I want to thank 
both of you, Ben and Lisa, for joining us in this  

01:06:58.980 --> 01:07:05.100
panel. We're going to get to more questions in 
a few minutes and so if folks can keep entering  

01:07:05.100 --> 01:07:11.040
those in the chat, but if we could go to the next 
slide, I wanted to take a quick minute and share a  

01:07:11.040 --> 01:07:21.240
few resources about sort of where to go to get 
more information, right? So next slide, please.

01:07:21.240 --> 01:07:28.260
Now that you've heard a little bit about a couple 
of different ways to use CLEAR, right, both our  

01:07:28.260 --> 01:07:33.360
example from the study team where we walked 
through navigating CLEAR as well as some of  

01:07:33.360 --> 01:07:38.460
the experiences that Lisa and Ben have shared. If 
you want to get more into the nitty-gritty here  

01:07:38.460 --> 01:07:47.340
on the left-hand side we have a list of links 
to some of our reference materials, right, how we  

01:07:47.340 --> 01:07:55.380
do some of these things that we're talking about 
beyond the profile summaries and the syntheses  

01:07:55.380 --> 01:08:01.080
and things that we talked about earlier. I also 
wanted to remind everybody to save the date we  

01:08:01.080 --> 01:08:06.900
have another webinar coming up in December that 
we'll talk about using CLEAR to find strategies  

01:08:06.900 --> 01:08:15.240
that address equity. And while today's webinar 
was sort of more focused at the workforce  

01:08:16.260 --> 01:08:22.440
in the workforce perspective this webinar 
is going to really be trying to include both  

01:08:22.440 --> 01:08:32.700
workforce as well as compliance assistance, worker 
protection, and other areas that are in the labor  

01:08:32.700 --> 01:08:40.260
sphere that are of interest that are seeking to 
address equity. You can find any event recordings  

01:08:40.260 --> 01:08:45.840
that we have we are posting to the CEO events 
page and then also we're developing a page in  

01:08:45.840 --> 01:08:52.020
CLEAR where we can post these and other resources 
for you, and if you have a question at any time we  

01:08:52.020 --> 01:08:57.960
do invite you to just reach out and contact CLEAR.
So I have the link here it's also easily found on  

01:08:57.960 --> 01:09:03.420
our website. If you want to hear more about 
CEO, we've got a few ways here that you can  

01:09:03.420 --> 01:09:08.580
do that easily. You can click on the link here 
and look for future events that we're hosting  

01:09:08.580 --> 01:09:13.380
that are public. You can also sign up for our 
periodic newsletter, Building the Evidence Base,  

01:09:13.380 --> 01:09:18.960
and that will tell you a little bit about the 
latest work that CEO is doing. Next slide.

01:09:21.240 --> 01:09:26.820
So again, I wanted to just recap kind of what 
we went through today. We learned how to navigate  

01:09:26.820 --> 01:09:34.560
CLEAR to find information, we heard multiple 
examples of how some of their research and  

01:09:34.560 --> 01:09:40.560
evaluation-based evidence and CLEAR is or could be 
used to support policy and program decision-making,  

01:09:40.560 --> 01:09:48.120
and we talked about where to look to learn more.
So before we launch into our questions. we're going to  

01:09:48.120 --> 01:09:57.120
cover just a few polls so we can get your feedback 
about how you thought today's webinar went and  

01:09:57.120 --> 01:09:59.040
what you want to hear more 
about from us in the future.