Welcome. The Chief Evaluation Office is celebrating 10 years of the Clearinghouse for Labor Evaluation
and Research. We would like to welcome you to today's webinar, "Strategies to Find Evidence
that Addresses Equity in CLEAR." This webinar is the third and the final presentation of
CLEAR 10-year anniversary webinar series. All the presentations will be posted on CLEAR's website.
So CLEAR is funded through the Department of Labor's Chief Evaluation Office
and is supported by various independent contractors. Next slide.
To address any issues or submit content related questions please select MSG Webex A,
host and presenter, from the drop down menu in the chat panel on your right and enter
your questions in the chat box. Please note that this event is being recorded
so by continuing to participate you are consenting to be recorded, so you may want
to turn off your video as recording will be posted on CLEAR's website. Next slide.
So before we begin with today's presentation, I'm going to turn this over to Dr. Christina Yancey,
who is the DOL's Chief Evaluation Officer, to provide brief remarks. As DOL's Chief Evaluation
Officer, Dr. Yancey leads the Department's research and evaluation and evidence-building agenda to improve
programs, policies, and enforcement activities that's been DOL's mission. She also serves as a
Department scientific integrity official oversing the Department's policy to address threats to
scientific integrity. Christina?
Thank you so much, Chayun, and thank you for having me and welcome
everyone. We are so excited to host our final webinar of the series celebrating CLEAR's ten-year
anniversary and today we have a great topic we're going to focus on how to search strategically for
research evidence and CLEAR using equity as kind of our cross-cutting theme.
By the end of today's session, you will be... you will have learned different ways to find
information in CLEAR on policies and programs and strategies that address equity and to learn
what the current evidence, rigorous scientific evidence, says about supports and interventions
that promote or may promote equitable labor market outcomes. So the Department of Labor
overall is committed to improving well-being of all communities but especially those experiencing
or historically have been experiencing underserved marginalized or excluded experiences. To that end,
I would like to begin if I mentioning just a few of the things that the DOL's Chief
Evaluation Office is supporting to advance our knowledge about equity in the workforce.
First, the Chief Evaluation Officer, CEO, is working with several of the Department of Labor and other
federal agencies to ensure integration of an equity lens into design data collection analysis
of our new evaluation projects. Secondly, we're also supporting analysis of existing data assets
and resources to show how equity questions could be answered with existing administrative data and
supporting internal analyses of DOL's own data to understand our own work in new ways, and third, for
the past couple of years CEO has funded what were called a Summer Data Equity Challenge on
equity and underserved communities which is to help both analyze existing administrative
resources and data and also to explicitly support the pipeline of researchers that are
examining equity implications of DOL policies, protections, and programs. Now, while interest in
advancing equity has been steadily growing, the movement has gained significant momentum in recent
years with the passage of the Executive Order 13985, which is about advancing racial equity
and support for underserved communities through the federal governments and so we we note that
as a signal of the fact that there's... there's opportunities across different federal
agencies that are also reinforcing and supporting activities that we're discussing this afternoon
as well as other initiatives at the federal, state, and local levels. However, we all know that there
are challenges in being able to use research and evaluation in your everyday decision making
and CLEAR is here to help with addressing that challenge. CLEAR sifts through tens of thousands
of research studies to find and review the most credible ones that examine effective
and promising programs across a variety of labor-related areas. So our goal today is to demonstrate
how you might search that information on a very broad topic like equity that might be nested in
multiple places on CLEAR so that you're able to quickly and easily find the information that you
need and we are we will also highlight some of the findings on equity that you can find
in CLEAR. So I'm looking forward to a really robust discussion today as well to hear from
you all as well so that's why it's it's great that Chayun had pointed out that there's the
opportunity for you to engage with us as we spend the next hour or so together. So thank you again
for being here and I will turn it over to Chayun now. Chayun?
Thank you, Christina.
So, before we begin, I would like to introduce the presenters for today.
So my name is Chayun Ying. I'm a part of DOL's
Chief Evaluation Office's evaluation staff and a member of CEO's CLEAR team. In addition to CLEAR
project I manage a portfolio of various projects covering DOL's capacity assessment, restricted use
data program, and CEO summer fellows program, Dr. Kristein Abner is a dissemination
lead for the ICF CLEAR team. Dr. Abner supports on this project's focus on the workforce and
social programming emphasizing the intersection of research and practice and policy and has supported
CLEAR since 2016. Dr. Shelley Osborn is the Rapid Review Lead for the ICF CLEAR team.
Dr. Osborne is a survey research and program evaluation specialist, supporting the federal sector including multiple
projects within DOL's Chief Evolution Office. She has supported CLEAR since 2017. Next slide.
So first I would like to share a bit more about DOL's Chief Evaluation Office
for those who are not familiar with CEO.
CEO is an independent Department level advising office since 2010. We coordinate, manage, and implement
DOL's evaluation programs, and we work closely with colleagues from about 15 DOL agencies to
help address their research questions and support their learning and continuous improvement efforts
by conducting evaluation studies, data analysis, and evaluation technical assistance. Most of our
work are done through third-party contracts and we currently have about 60 active projects for
working with the DOL agencies. If you want to learn more about CEO or our current studies or
view recently completed studies please check out the links at the bottom of the screen. Next slide.
So I would like to ask you a couple questions
on where you're from and your experience with CLEAR.
Our poll questions are here in the chat on your right so please click
please respond in the chat. The first poll is what organization are you from?
Are you from the federal government, state, local, tribal, or other government intermediaries,
DOL grantee, research university, employer or business, or other?
And then the second question is what is your experience with CLEAR?
Novice. I have never used CLEAR and I'm interested in learning more. Or are you an occasional user?
I have used CLEAR before but could use a refresher. Or are you a regular user? I use CLEAR whenever I
need evidence on what works to inform my work and I feel comfortable navigating the website
and if not sure yet. So it looks like we have people from various organizations and
many of you are CLEAR Novice. Okay, next slide.
So I'll turn it over to Kristin Abner.
Thank you, Chayun. Now that we know a little bit about our audience, I want to go through the goals of today's webinar.
So today, using equity as an example, our team will really demonstrate how to strategically search the
CLEAR website to find information on a subject or population of interest. So we'll share information
about what CLEAR has found on equity in workforce and employment programs but our core purpose
today is to really demonstrate strategies to find information in the Clearinghouse. Next slide, please.
So we'll accomplish our goals... oh can you... that's good, thank you... we'll accomplish these goals by
providing first a summary of key concepts from our last webinar in this series which was how
to use CLEAR for decision-making. We'll then show and discuss multiple ways to find information in
CLEAR using equity as our example and this will take us through various parts of the Clearinghouse
and will give us an opportunity to demonstrate for you all the multiple ways that information is
synthesized and presented in CLEAR. We'll summarize some findings across CLEARs products and then
we'll close with a recap of the many resources in CLEAR before we open the webinar up for Q&A.
Next slide, please/
So, as Dr. Yancey mentioned this is... oh can you go back? Thank you. As Dr. Yancey mentioned, this is the
third webinar in a series that are celebrating the 10-year anniversary of CLEAR. So our first
webinar focused on CLEAR 101 and the second webinar focused on using CLEAR for decision
making and I want to share a brief recap of that webinar before we move forward and just to note
that the recording, PowerPoint presentation, and transcript of the question and answers for
the webinars will be available forthcoming on CLEAR's website, so I will encourage
you to view those for more information when they're available. Next slide, please.
So recapping the second webinar, the webinar began with a hypothetical question
and a community college administrator wondering how
persistence and completion could be improved for their college training class. Next slide.
Our administrator learned that evidence on CLEAR comes in the form of evidence reviews, evidence
syntheses, and profile summaries. Our speakers explained that the resource... explain the resources
and walked the audience through the Clearinghouse to find evidence related to the policy question.
We'll be discussing CLEAR's Rapid Reviews on new emerging research forthcoming in this presentation
so I'll focus on the other three for the purpose of this recap. So evidence reviews are
collections of research on a topic and these may be defined by a population, so that might
include things like low-income adults veterans or older work, an outcome of interest, which could be
literacy, re-employment, or equitable post-COVID recovery, or an intervention type, so these might
include apprenticeship or work-based training, remote post-secondary education, or remote service
delivery strategies. So our evidence indices are short narratives that summarize the findings on
what works in a given topic and note gaps where evidence is needed. Our profile summaries provide
details on individual studies that CLEAR has reviewed. So usually studies are reviewed as
part of a search targeted by a topic or a time period, occasionally a profile summary will cover
a flash review of a single study that falls under an existing evidence review. So our profiles are
really short snippets of a larger study and you'll notice on the CLEAR website that the profiles will
link to that larger study if you want to go there for more information. Next slide, please.
So in our hypothetical situation our college administrator found strategy across CLEAR's
products for linked learning communities, paid performance incentives, and accelerated learning.
And each of these strategies had favorable impacts, were from credible studies,
and addressed the Community College population. So with this information in hand, they were in a
better position to discuss new program or program changes to improve persistence and completion.
Next slide, please.
So today, our webinar is on strategies for searching CLEAR using equity as an example
so we're using equity as our example for several reasons. As Dr. Yancey mentioned
equity is a priority for the Department of Labor and the Chief Evaluation Office is
working across the department and other federal agencies on multiple projects
related to equity. Equity also crosses different types of evidence from use in
CLEAR and therefore allows us to approach a search using multiple strategies. Lastly, equity
well it doesn't have its own evidence review, you can search using keywords, target populations,
and other strategies that will demonstrate throughout this webinar. Next slide, please.
So before we dive in we want to kind of set the stage here distinguishing equity from equality.
So this image on the screen is from the Robert Wood Johnson Foundation. The left hand slide shows a
one-size-fits-all curb, that everyone needs to negotiate across the street. On the right
though the curve and the signs are adapted to accommodate barrier, circumstances, and conditions.
Equity in the context of labor market outcomes refers to the fairness and justice in employment,
education and training and earnings. It's distinct from equality in that equity accounts for the fact
that not all populations start from the same place and interventions that promote
equity adjust to counteract imbalances related to bias, discrimination, and other systemic barriers.
So now I'm going to turn the mic over to Dr. Shelley Osborn
to walk us through our strategic search. Shelly?
All right thank you so much, Kristen. So next slide, please.
So there are multiple places to look in CLEAR for equity or anything else that you might be interested in
And as Kristin summarized, there are multiple types of revie ws in CLEAR.
So for our equity example, we're going to start with new and trending research.
So CEO conducted a rapid review to identify strategies
and evidence to promote equitable recovery from COVID-19. As shown here from the home
page, clear.dol.gov, you would navigate to New and Trending Research and use the drop
down menu to select Trending Research. This will lead you to the rapid reviews. Next slide, please.
So in CLEAR you'll find a list of the rapid reviews, we list them here, of which there
are nine currently each with a short description summarizes the publications reviewed. In addition
to equitable recovery from COVID-19, rapid reviews address worker health
and safety, distance learning, remote compliance assistance and enforcement, just to name a few.
For anything that looks promising, hyperlinks open the document. Each rapid review follows the same
general format. A guiding research question begins the document. This is followed by an introduction
of the subject and the objective of the rapid review. In general, these types of reviews identify
and summarize existing literature. The rapid reviews are organized around strategies that
have shown to be or may be promising. The reviews conclude by describing some gaps in the research.
Each rapid review also contains a supplement with summaries of each publication that's included in
the document. So each summary includes links to the article, report, or publication, so that
you can go to the primary source material. Some of those links will take you to the
full text, while others may be accessible only through a library account. Next slide, please.
So another place to look for research in CLEAR are the systematic evidence reviews. So the focus
of these systematic evidence reviews are causal studies, some include descriptive studies that use
statistical techniques to another quantitative approaches but do not attempt to assess the
causal impact of a program or intervention, some policies also include implementation studies.
We list here just some of what is covered in terms of populations outcomes and interventions.
As noted in our recap of the second webinar, systematic evidence reviews can be found and
CLEAR through the study profiles of the individual studies and for many topics
and in-depth synthesis of the evidence is also in CLEAR. We're going to turn our
attention now to some of the systematic evidence reviews that touch on equity. Next slide, please.
Our first is employer compliance. Here we show what is covered by the systematic review
and who is covered by the rules, policies, and enforcement activities. This is a good
time to note that CLEAR is always expanding. So while there are currently 22 studies,
systematic searches for new research occur regularly and as those systematic evidence reviews are
updated, CLEAR will contain additional studies. So you can check back often to see what's new
in CLEAR in these for systematic evidence reviews or rapid reviews. Next slide, please.
So disability employment policy is another example of a systematic evidence review. You'll note that
unlike our last example, employer compliance, this evidence review includes a synthesis report,
which we talked a little about. So the syntheses look at the research evidence across studies, highlights gaps
in the literature, and then also suggests areas in which further research is needed. Next slide, please.
So our last example is older workers. Like disability employment policy, this topic includes
a synthesis report. You'll note that the research extends back to 2005, which is more recent than the
previous two examples. Decisions on what years to include and what studies to include are decided
by DOL's CEO office and subject matter experts. These are then documented in a study protocol
which can also be found in CLEAR. So pausing here a moment provides a nice segue about how to search
in CLEAR. If you are interested in older workers, you now know that there are individual studies
and even a synthesis report you can review but what if, like equity, there isn't a drop-down menu
that has what you need? How can you search in CLEAR to find that information? Next slide, please.
So CLEAR is currently expanding how studies are tagged to follow the 2020 census taxonomy for race
and ethnicity. Like other statistical agencies, the Census follows standards on race and ethnicities
set by the U.S Office of Management and Budget in 1997. These standards guide how the federal
government collects and presents data on race and ethnicity. As we've talked about, equity is
not its own section in CLEAR, and in fact the CLEAR products and content that include equity
issues can be found across the full spectrum of the Clearinghouse, so we want to take a few
minutes to focus on different strategies to find evidence and information in CLEAR.
The first thing to note is that search in CLEAR is similar to how you would search for
any information on the web. You often start with a search term that leads to another search term
that leads to still another search term. So because authors refer to similar concepts in
different words you may have to experiment with a few different keywords before you
find the information that you're looking for. In this example, I first search for in turn BIPOC,
Black, Indigenous, and People of Color, and then equity and then equitable. So while each term
returned an increasing number of studies, I still wasn't finding exactly what I was looking for.
So that may be because equity is a fairly recent term, as is BIPOC, and the term again needs to
be in the publication and therefore the study summary to be captured by the search function.
So thinking about that, I began to experiment with different search terms as listed here.
So the same process could be used to search for other racial or ethnic groups, examples of
which are shown. You can also search for specific geographic areas and a keyword such as a state if
you were interested in, say, just programs that or studies that were conducted in Wisconsin,
or regions, if you were just interested in studies conducted say in the Southwest. Next slide, please.
So on the last slide, you can see, excuse me, if you go back to the last slide for one moment...
So here you can see that I searched for Black, and then you could also search
for publication years, so that's how you can limit a search to just articles that were
published since 2010 if you wanted something more recent. Thank you, next slide, please.
So you can also search by other parameters that are shown here on the right. So in this example,
I mean we find my results to the outcome that I was interested in, limited my results to the US, and
then restricted my research to studies that assess the causal impact or programs or interventions.
This is a good time to note also that most of the studies in CLEAR are based in the
United States with a few exceptions such as those focused on child labor outcomes. Next slide, please.
So as another example, I initially searched for studies in CLEAR using the term Latinx
as my only search term and I did not find many results. So, as with BIPOC, Latinx is a newer term
and again the term must be in the original publication and CLEAR study summary or study
profile so that it's captured by the search. So I searched instead using a drop down menu
for any of these words and included other terms that researchers may have used other than Latinx.
So the search returned actually dozens of studies. After reading through a few pages of results,
I decided to refine my search using the filter options. So the study shown here was just one
of the studies I found after I refined my search to those with the high causal evidence rating.
As we've mentioned, the high causal evidence ratings are given to studies in which we can be confident
that the effects were caused by the intervention examined. As you may know, CLEAR rates the strength
of the causal evidence in studies as high, moderate, or low. So from here I can search CLEAR using the
study author's name just to see if they have anything else that's included on CLEAR that
was reviewed and then you'll see tags at the bottom so I could also click on one or more
of those tags and that will take me to places in CLEAR that have related studies. Next slide, please.
So up to now we've been searching using keywords another way to search on CLEAR is by starting
with one or more of the available filters. So there are multiple layers to these filters,
for example on the right, other options you can select under target population, and then within
each of these are additional filters. We show some of these on the next slide. The next slide, please.
So this shows just two of the additional levels, so what is underneath age and then employment barriers.
Note that there's an option to select older workers when you refine by the target population
and then further refine by age. So doing so is another way in this case to access the
systematic evidence review for older workers, which you now know exists in CLEAR. In addition to the
individual studies, currently 43, as we discussed on a prior slide, there is an older worker synthesis
also if you click on the older workers in CLEAR there is a report titled Behavioral Finance Synthesis.
Many of the studies in this report look at how individuals often make different
decisions about saving for retirement than classic economic theory would predict. Next slide, please.
So let's suppose that we used one or more of a combination of these
search strategies to find evidence in CLEAR regarding equity issues.
What would we find? So we'll turn now to some highlights across CLEAR.
Next slide, please.
So as equity is our example today, we're going to start by discussing the rapid review on
equitable recovery from the COVID-19 pandemic. This rapid review was completed earlier this year
and we continue to look for publications that provide new insight and information.
This review summarized information from dozens of publications, some of which were causal studies,
some that were descriptive studies, and others that were literature reviews, policy briefs, and other reports.
Key concepts, such as equity in this case, are defined in the rapid review.
This rapid review focuses on Black, Indigenous, and People of Color and women,
while recognizing that people may also experience marginalization based on their
sexual orientation, disability status, immigration status, religion, income level, among other factors.
BIPOC individuals and women were disproportionately affected by the pandemic
which is one reason the rapid review focused on these populations. As noted, CLEAR is always
expanding so future updates to the rapid review may include other populations. Next slide, please.
There are four strategies identified in the literature for the rapid review synthesis.
First, education and training supports, with an equity focus, deliberately recruit, retain,
and support participants from marginalized groups. Employment and other supportive services include
strategies like telework, quality child care, and mental health services. A key Federal response
to support small businesses during COVID-19 was the Paycheck Protection Program or PPP,
a component of the Coronavirus Aid, Relief, and Economic Security Act, or the CARES Act, that was
made available in April 2020. Place-based interventions are programs to improve economic
and social conditions in specific geographic areas by providing resources to government, non-profit,
or private sector entities instead of providing resources directly to individuals. Examples of such
programs are Community Development Block Grants, the New Market Tax Credit, HUD Empowerment Zones,
Opportunity Zones, and also Promise Neighborhood. So over the next four slides we're going to show
the main takeaway findings under each of these strategies. While detailed findings
are outside the scope of our webinar, we will provide one or two examples of research as we
move through the summary. We encourage you to review the rapid review. Next slide, please.
So here we show the takeaway findings for education and training supports.
The third bullet notes that directly addressing equity can improve access to
online Career and Technical Education, or CTE, programs, but how exactly can this be done?
So one publication in this section recommended revising coursework that
requires substantial computing power, internet bandwidth, or high digital literacy skills.
Another recommended assistive technologies such as text-to-speech functionalities, captioning,
alternative text, and visual aids, so that students with vision and or hearing impairments or limited
English proficiency could access online CTE courses or programs. Next slide, please.
So listed on this slide are the main takeaway findings included in the rapid review under
employment supports and other supportive services. So the second bullet notes the
importance of expanding telework and flexible work options. Studies in the section noted that
working from home differs by race and ethnicity among occupations that can be done outside of
the office and that Latino and Black workers are overrepresented in jobs that do not offer telework.
For example one study found that 16% of Hispanic workers and 20% of Black workers compared
to 30% of white workers were offered the opportunity to telework. Next slide, please.
So here we have some findings for supports for small businesses and we want to talk a
little about the second bullet. So publications reviewed for this section of the rapid review
discussed many of the difficulties with the government's Paycheck Protection Program, or PPP.
For example the nation's smallest businesses were less aware of the PPP and less likely to apply.
One of the studies discussed in this section was a case study of pandemic relief for Latino-owned
businesses in the Washington DC area. It found that non-governmental and community-based organizations
raised awareness about the available relief options. So therefore building the capacity of
these community-based financial institutions could increase accessibility to Federal relief for those
who have difficulty accessing it, including micro businesses, which are defined as those with 10 or
fewer, excuse me, with less than 10 employees, and then businesses in communities of color.
Next slide, please.
So the last strategy that's discussed in the rapid review for equitable recovery is place-based interventions.
So recall that place-based interventions are when resources are directed to government, nonprofit, or private sector entities.
Activities vary widely and include providing block grants, offering tax incentives to businesses, and
investing in local physical infrastructure, or providing Workforce training. Regarding the second
bullet, one of the publications reviewed in this section noted that place-based programs rarely
include components that explicitly target local conditions caused by racially biased policies.
For example, opportunity zones are designed to attract outside investment. They do not specifically target
wages, creating jobs, or supporting local ownership of business.
Next slide, please.
So we turn our attention now to another rapid review in CLEAR, one that's focused
on remote post-secondary education. This rapid review addresses the research question: what is
the evidence on technology-based or distance learning that can inform efforts to provide
post-secondary education remotely?
So equity considerations is a section in this rapid review.
Among some of the findings emerging in this review, researchers noted that remote CTE compared
to in-person CTE attracted more working professionals, students who were currently
employed, students outside of that school's district, single parents, and part-time students.
So the publication's authors noted that these dimensions of difference should be kept in
the forefront when designing education and training courses and programs. Next slide, please.
We highlight here some findings that address equity from three more rapid reviews.
So the first is from return to work. The guiding research question here was in response to the COVID-19 pandemic.
What strategies can be effective for helping unemployed individuals return to work?
The second is from the worker health and safety rapid review. That addressed the research question:
What strategies can promote worker health and safety during the COVID-19 pandemic?
And the remote service delivery rapid review address the research question:
What is the evidence on remote service delivery programs and strategies to
inform DOL's efforts to provide remote employment services. Next slide, please.
So we want to return for a moment to the rapid review for equitable recovery.
So there's a section, the fifth in the rapid review, that discusses ways to measure equitable recovery that have
been or could be used to assess recovery among women, BIPOC, and other marginalized population.
Historically, employment and earnings have been used to measure labor market recovery.
As important and illuminating as these can be, some experts posit that they provide an incomplete
picture of economic recovery from an inclusive or equitable perspective. So listed here on the slide
are the main ideas included on the rapid review for different ways to measure equitable recovery
outside of, again, labor market earnings. And this could be applicable for multiple topic areas and
for equity more generally. So I'll pause here for just a little bit so you can read through them.
Okay, next slide, please.
Lastly, we list some research gaps that were identified across several of the
rapid reviews that address equity. The first research tap comes from the CLEAR rapid review
on remote secondary, remote post-secondary education, which we've talked a little about.
The second is from the worker, excuse me, the rapid review on worker health and safety during COVID-19.
again, which we've mentioned, and the third is the rapid review that presented evidence
on targeted or remote compliance assistance and enforcement strategies that could assist
employers in conforming to new and existing health and safety and work requirements under COVID-19.
The last is from the equitable recovery rapid review which we've talked quite a bit about.
We focused on the rapid reviews since these synthesize existing evidence. We encourage
you to explore CLEAR using the strategies we've outlined today to find research for the
populations and outcomes that you're interested in. I will now turn things back over to Chayun.
Thank you Shelley. Next slide.
So I would like to share additional resources before
we dive into questions and answers part of the presentation. Next slide.
So here we have the links to some of CLEAR resources. We already had some
questions about whether this presentation or the slides will be posted on CLEAR's website?
You can check out CLEAR's website. All three webinars... the recordings will be
posted we'll also have a questions and answer transcript and the PowerPoint slides there.
You can... if you want to hear more from CEO, you can join us for future CEO events by
clicking through the links, and you can sign up for CEO's newsletter. Next slide.
And there are additional equity resources as Christina mentioned, Department of Labor takes
seriously the serving of underserved communities... of underserved,
marginalized, or excluded communities. So please check out DOL's Equity Action Plan to find out more
as well the latest papers and data sets from CEO's Summer Data Challenge on Equity and
Underserved Communities is available on CEO's website, so please check it out. Next slide.
So before moving on to discussions and questions, let's take a quick poll on the quality of today's presentation.
First, how would you rate the quality of speakers content of today's seminar?
Very high quality, high quality neutral, low quality, or very low quality.
Okay next question.
Overall, how would you rate the relevancy of today's topic of your work?
Is it very relevant, relevant, somewhat relevant, or irrelevant?
And Brittni, whenever you're ready you can advance the next question.
The last question: what specific areas would you suggest CLEAR feature or investigate further?
And you can select multiple options. More research on equitable recovery,
more evidence on remote education training, more on implementation research,
more on the employment services and eligibility assessment program, or other. And you can specify in the chat.
And Brittni, you can close the chat once the analysis is complete.
Okay so the poll has ended. Next slide.
Okay so I'll open up the floor for any questions or discussions...
Okay so I have a question.
Will CLEAR consider adding a tag or a way to filter for additional terms such as equity?
Is there a process to submit recommendations?
And Kristin and Shelley, could you guys take this?
Definitely. I think the CLEAR as we noted in the presentation CLEAR's collection is
growing constantly and so as we are moving and adding new things to the collection we
think about how that... how we might tag and filter for additional topics. So it's definitely under
consideration. If folks have recommendations, on the CLEAR website you can contact us through
the form on the website which will come to the team at CEO and we'd be happy to take those.
Chayun, do you have anything to add?
No, thank you.
I have another question. What are the sources in general of the research on CLEAR?
Is it only DOL grantees?
So CLEAR conducts, when we do our systematic evidence reviews, we conduct a
very wide reaching search to gather all of the causal evidence in a given topic area.
And one of the things that you'll learn from the previous webinars is that CLEAR also
includes descriptive and implementation studies, but we cast a really wide net, so you'll
get dissertations that will be included. It's not just limited to DOL publications,
DOL grantees, so we do... we capture great literature from organizations that may post
reports on their websites but we look at the academic literature, of course,
journal articles, dissertations, like I mentioned, gray literature,
other federal agencies and departments would be included if it would fit the criteria for that search.
Thank you. So, someone asked how are tags, study type, etc. created for these studies?
So if you go on the CLEAR website, we have... one of the goals of CLEAR is to be transparent in how
we tag and filter our resources and so we have a tag and filter guidelines and they were created
really thinking about the kind of the broad scope of literature around employment and labor related programs.
So those are how the tags and filters are initially created, so you can go on the
website and see how we tag and filter resources and we, like I said, we really we think
about how we can always improve our tag and filter process to help users find what they need.
Thank you. Does CEO plan any future research specifically focused on equity in workforce program?
Does Shelley or Kristin... do you want to take that?
I can't speak for DOL, but I'll speak for CLEAR. We're currently, as Shelley mentioned,
we're updating evidence reviews currently at the disability employment evidence review is
currently being updated a lot of the other evidence reviews on CLEAR are being updated.
And you know... I'll let Shelley, do you want to talk a little bit about our rapid reviews and how
those sort of address new and upcoming topics?
Certainly and one thing I wanted to go back to about the search terms that we use
in the rapid reviews we do list, I didn't mention it during the presentation, but in addition to the strategies,
(coughs) excuse me, and the gaps in research
and all of the study summaries, there is also a section that explains what search terms and
keywords we searched for. So for folks that are really interested in kind of how that search
was done, they can look for those.
And then, the rapid reviews in general, every, and again this is a fairly complicated process that I won't go into great detail, but,
every few weeks, we are sort of looking for, again, using a variety of different keywords in a variety of
different searches, searching on Google Scholar for newly released information about the topics
for those rapid reviews. So it was, you know, sort of one sentence in the presentation about how
those are continually being updated, but just to stress even more sort of every couple weeks
we are looking for additional information and as Kristin mentioned those cover dissertations,
subject matter expertise, blog posts, newspaper articles the search is really really broad.
And then we sort of winnow that down as we find things that are most relevant for that topic area.
And I can add a little more. As Kristin and Shelley walked us through today,
some of the topic areas and then the rapid review synthesis wouldn't state equity, like it wouldn't
spell out "equity", but then a lot of the rapid reviews also cover some sections on equity or
you know specific target population or programs. So you might want to check
different reports, different topic areas across CLEAR, because it's a cross topic subject.
And then, you know, and DOL while you can check out the publicly available
evidence building plan too, the Equity Action Plan that that was linked in the PowerPoint slide.
And then I had another question.
Sorry, I lost the question.
Chayun I have one about the searching,
so let me talk a little bit more about that. The question was about sort of if the searches search existing synthesis reports or sort of a larger group of studies?
So when you search in CLEAR, and Kristin you can add to this of course too,
when you search in CLEAR, you're generally searching, if you're searching by keywords or if you're filtering or refining by the the ways that we mentioned you can do,
you're generally finding causal evidence studies that have been reviewed and those are
going to go a very very thorough and systematic review and that's where we list, you know, the
type of study that it is, whether it's causal or descriptive or other, you know, what methods they
used profiles are created... so generally when you search in CLEAR, those are what you're searching for,
which is why we, you know, sort of discussed different ways like the new and trending research
that you can find that synthesize in the case of the rapid review for equitable recovery from
COVID-19 there were 91 publications that were reviewed and sort of included in that larger
synthesis document. So when you're searching in CLEAR specifically you're usually searching
just individual studies, but if it's a particular topic area like for older workers there may also
be a synthesis report that you would find when you search.
That's a nice segue to the next question.
One of the audience members asked how long does it take to make a synthesis report?
Now that is an excellent question, and I did not, you know, we didn't track our time.
It takes a long time. It takes a really long time. So again for the
equitable recovery one there were 91 studies that were included in the document.
Multiple times that many were reviewed for sort of background information
reviewed and then ultimately not included, perhaps because they didn't really describe well enough a strategy that could
be implemented or used. So a lot of iterations and a lot of decisions and a lot of time goes
into which ones are included, how they're included, under which topic area they best fit,
which strategy, you know, etc. So quite a bit of time and awful consideration goes
into the rapid reviews and also again the individual study summaries and reviews.
But I think we can also add that we are targeting six months after the completion
of the evidence review... sometimes shorter, sometimes longer, so it depends
on the evidence and the topic area, like how much evidence is there? How many reports we're
synthesizing? And also in like rapid review and the synthesis we periodically make updates as well.
And then next question. Someone asked about small business... sorry.
Let me check...
Okay, okay there you go.
So is there anything that would particularly be useful in assisting small businesses?
Yeah I'll start so the equitable recovery rapid review has a section in supports for
small businesses and again when we were creating that rap... that
particular rapid review, most of what was available at the time was about the PPP or
the Paycheck Protection Program, but that section does mention other strategies
again that either have been shown to be or are hypothesized to be helpful. So that's one
place I know that you can search in CLEAR for information for small businesses.
We also have like the topic areas on entrepreneurship or self-employment
evidence topic area, so that may be of interest to small businesses. We have synthesis on those as
well so you can check out these topics, and Kristin, were you gonna say something else?
I was going to add that you can use, you know, for a topic like I want information on small
business, the information shared on this webinar can really be be applied in terms of searching for
information on a particular topic. So you can browse, if you go to the CLEAR website, you can
browse through our topic areas and kind of scroll through and see if there's relevant information to you.
You know, if you go to search for studies and search for keyword "small business"
and then you want to winnow down using the filters, you can do that like, you know,
I'm interested in, you know, I'm in a keyword search small business but then I'm really interested
in outcomes related to compliance. So you can keyword search and then filter down as well
to get kind of the information you need. Or, you know, I'm interested in small business maybe
in rural areas so there's the ability to filter down to find what you need
in the search for studies section of the website as well.
And Shelley, you briefly mentioned about why you focus on women and BIPOC in the equitable
recovery review and someone wants to know, you know, like why you didn't mention
like older workers or people with disabilities or any other, you know, population.
Yeah, in this case, in that particular case, we were really driven by what we were finding, and because
Black, Indigenous, and People of Color and women were particularly disproportionately impacted
that, as I mentioned, that's one of the reasons that we focused on them. There also happened to be
a great deal more information about those groups than there were other groups, such as LGBTQ+
individuals or older workers and low-income like sort of other vulnerable populations in
general, so that's the primary reason is that was the bulk of the research that we were finding were
with those groups, but as we've mentioned, we are continually searching for, and it's possible that
there may be sort of enough evidence and enough strategies for particular populations or groups
that when that review gets updated we can include additional groups.
Thank you. Maybe Kristin, you can take this. If someone wants to make a
recommendation for publication to include in CLAER, what's the process of sharing?
So CLEAR has the ability to contact us on the website if you have a suggestion for
a publication inclusion you can email us through the contact CLEAR portion of the website and
what we would do is our team would review it against our causal evidence guidelines and
our policies and procedures to make sure it fits with the collection and and go from there so...
We do include, as briefly mentioned, kind of what we call flash reviews, so if something
comes up and it fits within kind of the scope of CLEAR we will review it if it's causal through
our causal evidence guidelines and assess the strength of the quality of the evidence.
Thank you, and I think there's a question about the timeline of when the PowerPoint slides will be
posted online and we are aiming for early next year. So you should definitely check back CLEAR.
Okay any other questions?
Okay I think there may be a question coming in.
Okay is there a way to export the list of findings from a search? Like titles and DOI?
So, within topic area there is a button to export the results which would go into a CSV file.
I do not believe it includes DOI, but it would include the titles and how the profiles
are tagged. I don't believe... oh yeah, you can also export under search for studies as well.
So...
clicking on it now, so yeah so that'll export it into a CSV for you and you can filter by topic.
I am... I don't believe it has DOI, but it should have the other citation information.
And I'll add for the rapid reviews the DOI is included, so as I mentioned that the
supplement that accompanies every rapid review has basically the authors, the title, the type
of study, and then it lists the the website link which also often is the DOI and then
it includes a summary as well, and again those hyperlinks will at least generally take you
to the study abstract with perhaps a little bit more information, and then sometimes the full
text is available because it's not behind any kind of a paywall, and then sometimes it is so
you may have to access it, you know, request it through like an interlibrary loan or log
in if you have sort of library access to EBSCO or one of the other larger search engines.
Yeah, so if you go to like a topic area if you're, I don't know if you're on the CLEAR website but
there's the orange button it says export results and you can download the...
Yeah, download the results...
And also as a reminder, you know the goal of CLEAR is to make research accessible to
researchers and you know other users. So we do the summaries for you.
So we summarize our studies, and you don't need a library membership if you use CLEAR so...
Okay I don't think we have any other question...
Let's wait for a couple minutes and then...
Okay, Brittni, next slide.
Okay so if you haven't... if you weren't able to ask us a question or if you just browse through a
CLEAR and then come up with a question, feel free to reach out to us. You can contact us through
DOL CEO's email address or the CLEAR contact us link in the PowerPoint slide. So thank you again for
joining and this concludes today's webinar. The materials for this webinar will be the recording
PPT and transcript will be available online early next year so feel free to check it out.
Thank you.
Thank you, everyone