fbpx
Loading

Disrupting Human Trafficking

Independent evaluation of STOP THE TRAFFIK’s work and model

December 2019

Introduction

Over six months in 2019, STOP THE TRAFFIK worked with Robin Brady and Dr Ruth Van Dyke to conduct an extensive evaluation of their work and model. The goal was to carry out a review in a substantial level of detail, to determine with conviction how effective their interventions are in disrupting the business of human trafficking.

We are delighted to share the results with you.

Since its inception, STOP THE TRAFFIK has been focused on creating a world where people are not bought and sold. The organisation started as a campaign in 2006 to bring an end to human trafficking worldwide, coinciding with the bicentenary of the Abolition of the Slave Trade Act in 1807.

After significant activist support globally, it presented 1.5m signatures to the UN in 2008. In 2015, it developed an intelligence-led prevention approach supported by partners such as IBM: informing its focus at a local, national and global level. STOP THE TRAFFIK believes that it delivers this approach using a model that it calls ‘collect-analyse-share’.

STOP THE TRAFFIK works to create change in the following change areas:

  • Prevention of human trafficking
  • Knowledge of signs and how to respond appropriately
  • Behaviour when coming across a potential trafficking situation or in order to prevent one
  • Approach of the anti-trafficking sector, promoting collaboration, data use and sharing

Methodology

From the evaluation design process to data collection and reflection, the evaluation process was led by the evaluation participants and the STOP THE TRAFFIK team engaged fully with the evaluation process. STOP THE TRAFFIK and the evaluation team agreed that a mixed method approach would be most suitable for this evaluation. Read more about the mixed methods used, the evaluation criteria and questions, the data collection and analysis tools used and the participatory process below.

Scroll left and right to explore
report-method
  • Mixed methods

    STOP THE TRAFFIK and the evaluation team agreed that a mixed method approach would be most suitable for this evaluation. The evaluation process was framed with a contribution analysis approach; using aspects of process tracing to identify and assess the voracity of available evidence and outcome harvesting to undertake deeper dives into specific case studies. Raynor’s framework for assessing networks was applied to understand better the network and partnership working collaborative approach.

    Contribution analysis (Mayne, 2008; Centre for Evaluation Innovation, 2017) is particularly useful when considering processes and results from complex programming across a number of interventions and intervention sites. Contribution Analysis also provides a platform to investigate both intended and unintended results (both positive and negative) in a rigorous format.

    Process tracing (Collier, 2011; Punton & Welle, 2015) is often used to consider less tangible programme results and programme management performance and is also used to understand the impacts of activities in both the immediate and longer-term.

    Outcome harvesting (Wilson-Grau & Britt, 2012) is specifically designed to capture unintended results where the causes of change are not linear or clear, identifying the impact first and working backwards to the intervention to understand how the change has come about. Outcome Harvesting results in a case study-like narrative that will be useful not only to demonstrate the impact this programme is having.

    Raynor's framework(Raynor, 2013) can be applied to any level of partnership or network interaction and uses a simplified approach to determine how effective the partnership or network, the members of that partnership or network and the results or outcomes of that partnership or network have been.

  • Criteria and evaluation questions

    Using a participatory approach, including two workshop sessions with the STT team, the evaluators and STT identified a set of criteria, each with related key evaluation questions. These criteria and key evaluation questions informed the focus of our investigation. In addition, indicators were developed for each evaluation question to enable the inquiry to assess how well the evidence available supported the evaluation questions and therefore the criteria for the evaluation.

    Following on from the evaluation design workshop, an overall hypothesis of STT’s work was developed for the purposes of this evaluation. This hypothesis view was based on the existing Theory of Change, STT’s model and the Rich Picturing exercise during the evaluation design workshop. This exercise allowed certain causal chains to be identified for further investigation during the evaluation.

  • Data collection and analysis tools

    To deliver these methods, some familiar tools were used:

    • 141 Internal and external document reviews
    • Key informant interviews with internal and external stakeholders
    • Email-based or online investigation to follow up on the Outcome Harvesting questionnaires

    Data analysis tools

    The completed evaluation framework has identified areas where the hypothesis is robust and areas where it is less robust. This has also provided an opportunity to highlight to STT where there is scope for further learning and areas that could be included in a learning agenda.

    Evidence collected was assessed for how well it supports the evaluation criteria and questions agreed with STT using Contribution Analysis’ categorisation for assessing the contribution story. Evidence was also considered for type and for test of robustness under Process Tracing, using Bayesian analysis.

    We have reviewed a total of 168 pieces of evidence to assess whether the hypothesis is valid. The majority of the evidence does allow us to say with a high degree of certainty that the hypothesis, for the most part, is valid. However, the uniqueness of the evidence available and collected needs to improve if the whole hypothesis is to hold. This is not an issue of STT having not delivered the interventions, but rather having not presented or maintained the evidence behind their achievements, something we know the organisation is already working on.

  • On the role of participation

    From the evaluation design process to data collection and reflection, the evaluation process was led by the evaluation participants and the STT team engaged fully with the evaluation process. The scope and focus of the evaluation as well as both the initial top line findings and the recommendations were developed in a participatory manner.

    In the top line findings workshop and the validation workshop Appreciative Inquiry method was used. The workshop participants were facilitated to develop and agree sets of actionable recommendations, based on the key themes and patterns that emerge during the evaluation and the workshop. The overall evaluation approach encouraged participants to consider practical answers to the questions: “Now what?” “What are our options?” “When and how can we act?”.


Strategic framework

STOP THE TRAFFIK’s strategic framework was adjusted during this evaluation. In its core is STOP THE TRAFFIK’s recognisable operational model which is used to deliver a range of interventions and programmes with and across a wide range of stakeholders and partners to achieve its twin goals.

Click a circle to find out more
Scroll left and right to explore
Strategic Framework
Change Type
Stakeholders
Interventions / Projects
Tip: All findings and successes that match selected filters will be displayed. To return targetted results, use fewer filters.

Conclusions

What appears to have made STOP THE TRAFFIK so successful, and is seen as added value by its stakeholders, is its ability to engage with stakeholders and partners in a manner relevant to the partner’s needs. It is rare for NGOs to be able to speak to businesses in a way that the businesses can understand and appreciate, whilst being able to have a similar relationship with law enforcement agencies.

STOP THE TRAFFIK is highly valued for its collaborative approach and its global reach. Partners and other stakeholders believe that the organisation regularly punches above its weight and has an influence wider than its own scale. At the same time, the organisation is valued for its independence and ability to bring diverse groups together to affect change at a local level too. It is clear that STOP THE TRAFFIK’s work at a local and systemic (campaigns, corporates, FSOs, etc.) level reflect the same collaborative approach to working with stakeholders and partners.

Where STOP THE TRAFFIK is less successful is in understanding its own work and influence internally. The organisation appears to be unaware of the degree of its success and potential and the apparent siloed way of working is prohibiting better flexible working and more responsive approaches to the needs of stakeholders.

STOP THE TRAFFIK’s reference to its model as ‘collect-analyse-share’ is slightly misleading. We have found that the actual model being used and appreciated by stakeholders and partners is ‘partnership-trust-collect-analyse/share-(use)’. Partnership development and building trustworthiness is crucial to the collaborative approach that STOP THE TRAFFIK promotes. A collaborative approach is also much more than a transactional event, as the current model description suggests. In addition, ensuring that partners and stakeholders are well placed to make use of the data that is shared with them and can actually use it, is essential to the overall success of this approach.

While our investigation found that the majority of the hypothesis developed with STOP THE TRAFFIK at the start of the process was valid, there are areas where the hypothesis appears to not take account of what has actually been achieved or is being achieved. As such we have developed a revised hypothesis that outlines these additional outcomes.

The hypothesis shown above as a wheel diagram can also be described as a strategic framework in which STOP THE TRAFFIK has at its heart a recognisable operating model. The organisation uses this model to deliver a range of interventions and programmes with and across a wide range of stakeholders and partners. In effect it makes no difference whether STOP THE TRAFFIK is delivering policy change interventions to an Independent Review panel or training 700 taxi drivers, the same model is in play and if appropriately measured and tracked could move the organisation closer towards its twin goals.

Finally, it is reasonable to say that STOP THE TRAFFIK is a unique NGO with a specific operating approach that its stakeholders find valuable and relevant. With appropriate strategic development, STOP THE TRAFFIK could continue to punch above its weight and deliver significant results against modern slavery and human trafficking globally.

Recommendations

  • Monitoring, evaluation and learning (MEL)

    STOP THE TRAFFIK has already started to address its MEL systems and processes and some of these recommendations are already under consideration. We would like to see the organisation apply its MEL process more consistently, develop output-level targets for activities and use them alongside outcome-level targets, and follow up longer term data collection to see whether change sticks. Consider conducting a value for money assessment of STOP THE TRAFFIK’s key achievements, using an approach such as social return on investment (SROI) to understand the value of the social change generated by STOP THE TRAFFIK’s work.

    STOP THE TRAFFIK RESPONSE:

    This evaluation highlights the organisation’s developed evaluative thinking approach and existing MEL practices. A plan to develop more robust monitoring and more accessible tools to be used and embedded had been put in place.

  • Strategic planning

    Structured planning for periods of three to five years, informed by MEL findings and including strategic milestones, might help STOP THE TRAFFIK to clarify what it expects to achieve in these smaller time periods that will help it move towards its goal. This planning is already underway as STOP THE TRAFFIK had prepared a 2019-2022 strategy, and the organisation should integrate findings and recommendations from this evaluation into it.

    STOP THE TRAFFIK RESPONSE:

    During the evaluation period we had worked on our 2019-2022 strategy, which had now been published.

  • Policy influencing

    STOP THE TRAFFIK policy influencing work has had a significant impact globally. Resources should be invested in developing policy work within the organisation to, at the very least, monitor and react with targeted policy change work at an international, national or industry-wide level as needed.

    STOP THE TRAFFIK RESPONSE:

    We have now added a sixth strategic objective to influence local, national and global systems to embrace an intelligence-led approach to disrupt trafficking. We will be developing an accompanying operational plan to implement, too.

  • Integrated working approach

    STOP THE TRAFFIK has a recognised brand. It is viewed as impartial and collaborative in its work to prevent and disrupt modern slavery at local, national and international levels. Its credibility is in part due to its feet on the ground, embedded in local communities and businesses and then extending its reach internationally. We recommend that STOP THE TRAFFIK retains the existing practice teams while breaking down the silos between teams through improved internal collaboration and communication. The organisation should work on developing a management and project management approach that draws on good practices in the Agile and Matrix traditions in order to retain that urgency about its work as it normalises systems and processes and matures as an organisation.

    STOP THE TRAFFIK RESPONSE:

    One of the reasons for this evaluation was a desire to see how we create impact that is bigger than the sum of our parts, and we worked with the evaluation team to adapt project management approaches to support us working in a more integrated manner.

  • Sharing good practice

    STOP THE TRAFFIK had developed practices in a number of activities that are successful and effective. These need to be recorded and should be shared internally and externally. Internal sharing of good practice will ensure that as the organisation grows, it can retain its organisational intelligence and ensure that activities are done in the most effective way. External sharing would enable external stakeholders or those with an interest in preventing modern slavery to see what works, under what conditions and for whom. It will also raise STOP THE TRAFFIK’s profile as a thought leader in the sector. Over time, this could be evolved into a learning agenda for the organisation, focused on its strategic framework or long-term vision.

    STOP THE TRAFFIK RESPONSE:

    We have been sharing information about our work in events and conferences, especially in the past year or so. We will be developing a plan to share more good practice as part of our Communications Strategy.

Join our global maling list

More news on how you can get involved and how we're disrupting human trafficking all over the world.

About

This evaluation was commissioned by STOP THE TRAFFIK and funded by Comic Relief. The evaluation team included Robin Brady and Dr Ruth Van Dyke. It was co-designed by the evaluation team and the organisation, using a participatory approach and focused on STOP THE TRAFFIK’s four change areas, how they contribute to its wider impact and how their interdependence is influenced by that of STOP THE TRAFFIK’s team structure.


Back to top
Our website uses cookies. By using it you agree to this. Read our Privacy & Cookies Policy for more information.