Project GATE - Design Report, Employment & Training Administration (ETA)

Project GATE - Design Report

April 23, 2003
Prepared by:
IMPAQ International, LLC

PROJECT GATE
DESIGN

CONTENTS

Introduction
Summary of GATE Service Strategy
Summary of GATE Evaluation Strategy

A. Introduction

Many individuals have the motivation and skills to develop small businesses, but lack business expertise and/or access to financing. Project GATE (Growing America Through Entrepreneurship), sponsored by the U.S. Department of Labor, Employment and Training Administration (DOL/ETA), and the Small Business Administration (SBA), has been designed to assist these individuals. Project GATE provides training, technical assistance and assistance in applying for small business loans to individuals who are interested in starting and/or growing a small business. The GATE demonstration project will be implemented at seven sites in three states: Maine, Pennsylvania, and Minnesota. Special emphasis will be placed on reaching immigrants, rural populations, and low-income urban populations.

The goals of Project GATE are:

To increase awareness of microenterprise services through community outreach;
To offer microenterprise training and technical assistance to individuals seeking help in starting their own small businesses;
To evaluate whether this microenterprise initiative is viable, effective, and cost-effective in promoting self-employment, generating new businesses, and promoting community economic development.

The effectiveness of Project Gate will be evaluated using an experimental design. This design, widely viewed as the "gold standard" approach to evaluating programs, will involve randomly assigning eligible applicants to Project GATE to either a program group or a control group. People in the program group will receive GATE services. People in the control group will not be able to receive GATE services during the study, although they can receive other microenterprise services available in the community. All applicants will have an equal chance of being selected for the program—personal factors or the applicant’s business idea will play no role in whether a participant is selected. The effectiveness of the program will be determined by comparing the outcomes of members of the program and control groups.

B. Summary of GATE Service Strategy

The Project GATE service strategy is depicted in Figure 1. Participants in Project GATE will progress through the program in the following stages:

Community Outreach. A multi-pronged community outreach effort will publicize Project GATE. This outreach campaign will use print ads, posters, brochures, flyers, kiosks, and a Project GATE website, and will be customized to target each of the seven project sites using a multilingual and multicultural approach.
Registration. People who are interested in learning more about Project GATE will be able to register for an orientation session by postcard, toll-free telephone number, web-based kiosk, or the Project GATE website.
Orientation. Project GATE will invite registrants to attend an orientation session at a One-Stop Center, where a Small Business Development Center (SBDC) staff member will describe the project in more detail.
Application. Following orientation, those individuals who remain interested in receiving GATE services and are eligible for the program may submit an application form.
Random Assignment. Applicants who submit a complete application will be randomly assigned to either the Program group or the Control group.
Assessment. Participants in the Program group will undergo assessment at an SBDC office and will be assigned to an entrepreneurship training organization (i.e., service provider) based on the participant's needs.
Service Provision. The service provider will offer classroom training sessions on microenterprise skills, and individual counseling including assistance in obtaining SBA loans.

Data will be collected about the participants throughout this process, starting at Registration. Ancillary data about the services provided, loans made, and small businesses started will also be collected. Follow-up surveys will be administered to sample members (Treatment and Control group members) at two points (i.e., 6 months and 15 months after random assignment). The data collected, together with the data from GATE Participant Tracking System (PTS), will be used for the evaluation.

Figure 1
Project GATE Service Strategy
The project GATE service strategy

Community Outreach

To create a brand name which the public can associate with Project GATE, we will develop a logo, tagline, and overall marketing message for Project GATE. The overall marketing message will guide the content, tone, and “look and feel” of all the materials developed specifically for each market.

A general Project GATE website will be designed to convey information about the project to anyone who visits the website from a computer connected to the Internet. This website will have content written with a target audience of potential participants in mind. It will also include an online Registration Form for people to indicate interest in the program and request to be scheduled for an orientation.

Although there are seven Project GATE sites, there are not necessarily seven markets. For example, within the Minneapolis-St. Paul site, there is a Hmong population, as well as a Somali population, which may be interested in Project GATE services. To reach these distinct populations, it may be necessary to develop separate community outreach materials for these groups. On the other hand, at least two of the three rural sites in Maine (Portland and Lewiston) may be close enough geographically and have sufficiently similar populations that certain marketing materials may be shared across these two sites.

For each market, community outreach materials will be developed, describing the Project GATE services and inviting people to register. These materials will include brochures, print ads, flyers, and posters. We are also considering public service radio or TV ads in each of the target markets. The outreach materials will be disseminated at the offices of key community, government, and faith-based organizations.

In addition, kiosks will be set up at selected locations in each project site, such as at One-Stop Centers. Each kiosk is essentially a computer mounted inside a cabinet, with the screen accessible to passersby and a keyboard available for data input. Each kiosk will have a touch screen which people can use to navigate through a simple series of informational pages to learn about Project GATE. In addition, the kiosk will be able to display a Registration Form which people can use to indicate their interest in receiving project services.

The kiosks will be designed to be compliant with the Americans With Disabilities Act. For example, the kiosk cabinets will be designed at an appropriate height and with enough clearance in front of the kiosk and across the width of the kiosk so that people in wheelchairs will be able to access the screen and keyboard. For people with visual impairment, large type font will be used, and a toll-free phone number will be available to potential GATE participants to learn what the GATE program is about and how to register.
Registration

A person who has learned about Project GATE can indicate interest in finding out more about the project or receiving project services by filling out a Registration Form. The Registration Form will include the person’s name, address, telephone number, and email address.

The Registration Form will be available in several different formats. First, each brochure disseminated at a project site will contain the Registration Form on the back of a postcard that is addressed to IMPAQ. Second, the general GATE website will contain an online Registration Form which anyone connected to the Internet can fill out. Third, IMPAQ will set up a toll-free telephone number which people can call to provide their contact information.

Whenever we receive contact information from a new registrant, we will enter that information into a database which we call the Registrant Database. This information will be used in the next step, when registrants are invited to an orientation session.

Project GATE will develop a Participant Tracking System (PTS). The PTS will be a database system designed to track individuals as they move through the GATE program. Specifically, the PTS Database will contain: participants’ demographic and background information, whether they are assigned to Treatment or Control group, what services and SBA loans they received, and information about the businesses they may have started.
Orientation

After we receive contact information from a person interested in Project GATE, we will attempt to contact that person, both by phone and by mail, to invite him or her to attend an orientation session held at a conveniently located One-Stop Center.¹ Although it is possible that we will receive some incomplete or inaccurate contact information, we will make every effort to contact each registrant. Once we have successfully invited the person to an orientation session, we will record this fact in the Registration Database.

Orientation sessions will be held at One-Stop Centers. At these sessions, SBDC staff will describe Project GATE’s goals and services offered, and answer any questions that attendees may have. The Orientation will present a realistic view of self-employment – both the advantages and disadvantages of becoming self-employed. In addition, the presentation will stress the important role that Project GATE plays in research that evaluates microenterprise efforts, and indicate that project services will be offered on a lottery basis, with approximately 50% chance of receiving services. Also, we will ask orientation attendees to provide some background information on a short form.
Application

At the end of the Orientation session, attendees who are still interested in participating in Project GATE will be asked to fill out an Application Form. This form will ask for SSN, some background information, and a short description of the business idea that the person would like to pursue. The applicant will submit the application form to the SBDC, One-Stop Center or mail the form to IMPAQ.

Once the completed Application Form is received at an SBDC or One-Stop Center, the staff person will go online to a specific Internet address and type the data from the paper Application Form into an online Application Form. That online form will be connected to the PTS Database so that the information can be captured. At this stage, tracking of Project GATE participants begins. If the applicant mails the paper Application Form to IMPAQ, then IMPAQ staff will type the information into the PTS Database.
Random Assignment

Once data from the Application Form enters the PTS database, IMPAQ staff will clean the data. This process involves several steps:
- Making sure the data are complete – that is, there are no missing fields.
- Checking that the data do not duplicate data already in the PTS Database.
- Checking that the data do not contradict existing data in the PTS Database.
After cleaning the data, IMPAQ staff will also make sure that the business idea is appropriate (that is, not illegal) and meets the other eligibility criteria. Anyone who meets these two criteria will be randomly assigned to either the Treatment group or the Control group. The PTS Database will capture which group each applicant was assigned to.

Once an applicant is randomly assigned, the person will be notified of the result. Those assigned to the Treatment group will be asked to contact their local SBDC office to arrange an Assessment interview.
Assessment

A person who is assigned to the Treatment group will meet with a staff person at an SBDC office for an individual assessment of needs. At this time, the staff person will review the participant’s microenterprise background, and describe the various training and technical assistance services offered by service providers in the area. Together, the staff person and the participant will decide which Project GATE services and provider would be most beneficial to the participant. The SBDC staff person will then contact the appropriate service provider to schedule a time for the participant to begin services.

After the service providers are selected, the staff person will go online to a specific Project GATE web address and fill out an online Assessment Form. This form will capture the participant’s SSN and the name of the service provider which will provide the training services. This information will be captured by the PTS Database.
Service Provision

After the Assessment interview, the participant will begin a series of training sessions and individual counseling sessions at a service provider. IMPAQ will negotiate with service providers at each site to implement training services and individual counseling services that are consistent across all the Project GATE sites. To ensure consistency of services in all sites, we will require each service provider to provide a minimum set of GATE services. The objective is to ensure that Project GATE participants have access to essentially the same treatment, regardless of location. All service providers will be asked to periodically enter information about the receipt of services (attendance at workshops, amount of counseling provided, types of technical assistance provided, referrals etc.) into the PTS database.

C. Summary of GATE Evaluation Strategy

Project GATE will be rigorously evaluated. The evaluation will address whether the program can be replicated on a larger scale, whether it works, and whether it is cost effective. Data for the evaluation will be collected via the PTS, two follow up surveys, administrative data, and periodic visits to the sites by researchers.

Participant Tracking System. A Participant Tracking System (PTS) will be developed for Project GATE. This will be a database designed to track individuals as they move through the GATE program. Information from the application forms and forms periodically completed by the SBDCs and other service providers will be maintained on the PTS.
Follow Up Surveys. Two telephone surveys will be administered to members of both the treatment and control 6 months and 15 months after they apply for the program. These surveys will ask about the receipt of microenterprise services, starting a business (including application and receipt of SBA and other loans), experiences starting a business, and experiences in jobs working for other people. The first survey will mostly collect information about the receipt of services, while the second survey will collect more information about business and labor market outcomes.
UI Administrative Data. We will collect data on quarterly earnings and receipt of UI benefits for all members of the program and control groups. These data will be collected for a period beginning a year or two prior to their application to GATE and extending for a year or two afterwards.
Site Visits. Researchers will visit each site at least three times during the demonstration. During these visits, we will both provide technical assistance on the implementation of the demonstration and collect information for the evaluation. Researchers will interview staff at one-stop centers and service providers, observe training workshops and counseling sessions, collect cost data, and conduct in-depth case histories and focus groups.

The evaluation will involve three components: a process analysis, an impact analysis, and a benefit-cost analysis.

Process Evaluation

The process evaluation will identify what it takes for Project GATE to be successful, so that others who wish to replicate the program in the future will be able to draw on lessons learned from this demonstration. Six research questions will guide the process evaluation:
- How was Project GATE implemented in each of the seven sites?
- What challenges did organizations implementing the program face?
- Who participates in the program and what program components did they use?
- Under what type of socio-economic, demographic, and service environments was Project GATE implemented?
- What barriers to starting businesses did participants face and how did the program address these barriers?
- What effects did the new businesses started by GATE participants have on their communities?
Impact Analysis

The impact analysis is designed to evaluate whether the program is effective in improving outcomes for program participants and the community. As we will conduct random assignment, differences between the outcomes of program and control group members can confidently be attributed to Project GATE. Key outcomes of interest will include:
- Receipt of microenterprise training, technical assistance, and other services;
- Completion of a business plan;
- Receipt of SBA Microloans and other loans;
- Whether a business was started and the size and financial success of the business;
- Employment, both in self-employment and working for someone else;
- Receipt of unemployment insurance and other public assistance; and
- Satisfaction with employment.
The evaluation will address whether participation in GATE improved these outcomes overall, and also whether its effectiveness varies by characteristics of the participant, whether they live in an urban or rural area, how the program is implemented, and characteristics of the local economic or service environment.
Benefit-Cost Analysis

Even if the program is effective, its impacts may not be large enough to justify the resources spent on the program. To address its cost-effectiveness, we will measure all the benefits of the program and compare them with its costs.

In some cases we may consider holding orientation sessions at community centers such as the Hmong Mutual Assistance Association.