Evaluating a Funneled Entry Program – as applied to procurement training for women-owned SMEs

Under a funnel approach, a program implementor casts a wide net and offers a basic level of services to many people, and then extends more intensive and expensive services to progressively smaller subsets. This was the approach used in the IFC’s Sourcing to Equal program in Colombia, designed to build the capacity of women-owned-or-led SMEs to bid for and win procurement contracts from large corporations. The funnel here has three stages: 1. A 2-hour online webinar: this provides a general overview of corporate procurement requirements, processes, and opportunities – and can be delivered at large scale to many firms (offered to 1,105 firms in our setting, with 612 taking it up) 2. Bootcamps: two half-day virtual sessions and one 2-day in-person session. These go into more depth, providing case studies, legal considerations, how to build a B2B value proposition, working capital solutions, negotiation skills, advice from buyers, putting together a pitch and proposal, etc. These are more expensive and intensive, but still delivered in a group setting, so not as much as the final stage. 399 firms attended the bootcamps. The combined cost of the bootcamp and masterclass was $628 per participant, with most of this for the bootcamp. 3. Mentoring: the most intensive phase, consisting of three group sessions, three individual sessions (10 hours). Here mentors developed customized workplans for the needs of each firm and helped them prepare for and bid for contracts, e.g., by designing measures to show a buyer they are capable, working on how to structure client engagements, and helping through the vendor registration and onboarding. 120 firms received mentoring, at a cost of $819 per participant. This type of funnel approach is being used in other business support programs such as this one in Malawi, and also applies in many other settings: e.g. education programs that provide basic support to everyone, and then more intensive support to a subset; or job search assistance and counselling in which everyone gets a basic level, and then more help is given to those revealed to need more. However, these programs raise several challenges for impact evaluation. First, because of the desire/need to draw many units into the funnel, you need a large sample which may make it hard to use an oversubscription design – in Colombia 1,105 was the total number of firms we could get after extensive outreach and advertising. Second, normally in impact evaluations we aim to minimize low take-up, given its implications for statistical power. But in funnel programs a reduction in take-up is a feature, not a bug – the funnel is used to reveal information (e.g. about who shows up, who demonstrates promise) that then determines where to focus more resources. Evaluating funnel programs In a new working paper with Rachel Cassidy and Kalyan Neelamraju, we evaluate the Sourcing to Equal program in Colombia, and use this to show how to address these issues and make progress on learning more from funnel programs. Phased-in RCT First, to deal with the issue of how to still conduct an experiment and have a control group when our entire sample of firms had to pass through the funnel, we used a phased-in RCT, randomly assigning firms to get the program in a first, second, or third wave. In each wave firms went through their respective funnel. We then conducted a midline survey in which wave 1 is treated (for 6 months), and waves 2 and 3 serve as control, and an endline survey in which waves 1 and 2 have been treated (for 12 and 6 months), and wave 3 as the control. The downside is this limits measuring longer-term impacts. An upside is that because we observe which firms get through to different phases of the funnel in later waves, under an assumption of temporal stability in funnel pass-through, we can then e.g., compare treated firms who get to the mentoring stage with control firms who subsequently do. Figure 1: Using a phased-In RCT, the funnel was applied in waves Complier-Weighted Treatment Effects The standard approach to evaluating this program would then be to compare average outcomes for those in the earlier waves to those in the later waves, to give the intention-to-treat (ITT) effect. However, here only 55% of firms even attend the masterclass after being invited, 36% to the bootcamps, and 11% the mentoring. The ITT gives the average effect for all firms of being offered the program and having a chance of making it through, which doesn’t tell us that much about the firms that go through the full program. One could use a LATE, which seems appropriate for looking at the effect of getting to at least the masterclass, but the exclusion restrictions become less plausible for looking further down the funnel. Instead, we estimate complier-weighted treatment effects (CWTE), following Joffe and Brensinger (2003) and Coussens and Spiess (2021). These estimates put more weight on the units that are predicted to be more likely to reach the particular stage of the funnel. For example, suppose that firm size perfectly predicted attending the masterclasses, and that no firms with fewer than 10 workers attended. Then smaller firms only add noise and contribute nothing to the treatment effect, and we could look at the average effect for bigger firms. In practice, prediction is of course more complicated and less deterministic, but a similar idea applies. Estimation takes place in two steps: Step 1: Use baseline variables to predict compliance with different stages of the funnel. Leave-one-out estimators are used here, so that an observation is not being used to predict its own status. If you have many baseline covariates, Coussens and Spiess provide a machine-learning approach, but we do not have very many, and so just use probits. It is hard to predict that well which firms will make it through (which is itself a reason for using the funnel rather than just ex-ante selection). However, a few variables like previous procurement knowledge, having submitted bids before, and saying in the survey they have more interest in attending to predict getting through to the bootcamps and mentoring. Step 2: We then use these predicted likelihoods as weights in a treatment regression – so that e.g. the CWTE for bootcamp compliers puts more weight on firms with characteristics that predict more likelihood of making it to the bootcamp phase. What did we learn, and how does this compare to the ITT? We find that the program increases procurement knowledge and readiness, leads firms to be more likely to register in corporate portals, to submit a corporate bid, and to win a corporate bid. The CWTE and randomized roll-out comparisons of compliers suggest that these impacts are larger for the firms that got to the bootcamps and mentoring stages. · The ITT on registering in a corporate portal in the previous 6 months is 10.3 percentage points, relative to a control mean of 52%. The CWTE is 12.9 p.p. for those predicted to get to the bootcamp stage, and 17.2 p.p. for those predicted to get all the way to mentoring. · The ITT for winning a bid in the previous 6 months is 4.8 p.p. relative to a control mean of 30%. The CWTE is 5.0% for those predicted to get to the bootcamp, and 7.3% for those predicted to get through to mentoring. The CWTE therefore show larger gains from the program for the types of firms that are more likely to have participated in more of it. We had also hoped that the use of CWTE might improve precision, but the significance levels are the same as for our ITTs. This is where there is a partial tension for implementation and evaluation: CWTEs will be more precise when compliance can be easily predicted – but if you could predict it really easy, there would be less need for a funneling process. The greatest value may then be when compliance is ex ante difficult to predict, but ex post correlates strongly with baseline characteristics.