Introduction
Over six months in 2019, STOP THE TRAFFIK worked with Robin Brady and Dr Ruth Van Dyke to conduct an extensive evaluation of their work and model. The goal was to carry out a review in a substantial level of detail, to determine with conviction how effective their interventions are in disrupting the business of human trafficking.
We are delighted to share the results with you.
Since its inception, STOP THE TRAFFIK has been focused on creating a world where people are not bought and sold. The organisation started as a campaign in 2006 to bring an end to human trafficking worldwide, coinciding with the bicentenary of the Abolition of the Slave Trade Act in 1807.
After significant activist support globally, it presented 1.5m signatures to the UN in 2008. In 2015, it developed an intelligence-led prevention approach supported by partners such as IBM: informing its focus at a local, national and global level. STOP THE TRAFFIK believes that it delivers this approach using a model that it calls ‘collect-analyse-share’.
STOP THE TRAFFIK works to create change in the following change areas:
-
Prevention of human trafficking
-
Knowledge of signs and how to respond appropriately
-
Behaviour when coming across a potential trafficking situation or in order to prevent one
-
Approach of the anti-trafficking sector, promoting collaboration, data use and sharing
Methodology
From the evaluation design process to data collection and reflection, the evaluation process was led by the evaluation participants and the STOP THE TRAFFIK team engaged fully with the evaluation process. STOP THE TRAFFIK and the evaluation team agreed that a mixed method approach would be most suitable for this evaluation. Read more about the mixed methods used, the evaluation criteria and questions, the data collection and analysis tools used and the participatory process below.
-
Mixed methods
STOP THE TRAFFIK and the evaluation team agreed that a mixed method approach would be most suitable for this evaluation. The evaluation process was framed with a contribution analysis approach; using aspects of process tracing to identify and assess the voracity of available evidence and outcome harvesting to undertake deeper dives into specific case studies. Raynor’s framework for assessing networks was applied to understand better the network and partnership working collaborative approach.
Contribution analysis (Mayne, 2008; Centre for Evaluation Innovation, 2017) is particularly useful when considering processes and results from complex programming across a number of interventions and intervention sites. Contribution Analysis also provides a platform to investigate both intended and unintended results (both positive and negative) in a rigorous format.
Process tracing (Collier, 2011; Punton & Welle, 2015) is often used to consider less tangible programme results and programme management performance and is also used to understand the impacts of activities in both the immediate and longer-term.
Outcome harvesting (Wilson-Grau & Britt, 2012) is specifically designed to capture unintended results where the causes of change are not linear or clear, identifying the impact first and working backwards to the intervention to understand how the change has come about. Outcome Harvesting results in a case study-like narrative that will be useful not only to demonstrate the impact this programme is having.
Raynor's framework(Raynor, 2013) can be applied to any level of partnership or network interaction and uses a simplified approach to determine how effective the partnership or network, the members of that partnership or network and the results or outcomes of that partnership or network have been.
-
Criteria and evaluation questions
Using a participatory approach, including two workshop sessions with the STT team, the evaluators and STT identified a set of criteria, each with related key evaluation questions. These criteria and key evaluation questions informed the focus of our investigation. In addition, indicators were developed for each evaluation question to enable the inquiry to assess how well the evidence available supported the evaluation questions and therefore the criteria for the evaluation.
Following on from the evaluation design workshop, an overall hypothesis of STT’s work was developed for the purposes of this evaluation. This hypothesis view was based on the existing Theory of Change, STT’s model and the Rich Picturing exercise during the evaluation design workshop. This exercise allowed certain causal chains to be identified for further investigation during the evaluation.
-
Data collection and analysis tools
To deliver these methods, some familiar tools were used:
- 141 Internal and external document reviews
- Key informant interviews with internal and external stakeholders
- Email-based or online investigation to follow up on the Outcome Harvesting questionnaires
Data analysis tools
The completed evaluation framework has identified areas where the hypothesis is robust and areas where it is less robust. This has also provided an opportunity to highlight to STT where there is scope for further learning and areas that could be included in a learning agenda.
Evidence collected was assessed for how well it supports the evaluation criteria and questions agreed with STT using Contribution Analysis’ categorisation for assessing the contribution story. Evidence was also considered for type and for test of robustness under Process Tracing, using Bayesian analysis.
We have reviewed a total of 168 pieces of evidence to assess whether the hypothesis is valid. The majority of the evidence does allow us to say with a high degree of certainty that the hypothesis, for the most part, is valid. However, the uniqueness of the evidence available and collected needs to improve if the whole hypothesis is to hold. This is not an issue of STT having not delivered the interventions, but rather having not presented or maintained the evidence behind their achievements, something we know the organisation is already working on.
-
On the role of participation
From the evaluation design process to data collection and reflection, the evaluation process was led by the evaluation participants and the STT team engaged fully with the evaluation process. The scope and focus of the evaluation as well as both the initial top line findings and the recommendations were developed in a participatory manner.
In the top line findings workshop and the validation workshop Appreciative Inquiry method was used. The workshop participants were facilitated to develop and agree sets of actionable recommendations, based on the key themes and patterns that emerge during the evaluation and the workshop. The overall evaluation approach encouraged participants to consider practical answers to the questions: “Now what?” “What are our options?” “When and how can we act?”.
Strategic framework
STOP THE TRAFFIK’s strategic framework was adjusted during this evaluation. In its core is STOP THE TRAFFIK’s recognisable operational model which is used to deliver a range of interventions and programmes with and across a wide range of stakeholders and partners to achieve its twin goals.
Findings
This section considers STOP THE TRAFFIK’s results derived from its activities through the lens of the four change areas it is trying to influence.
Click the double arrow icon to explore how a finding relates to the strategic framework.
1 Increased perception of risk
There are no findings in this section that match the filters you have chosen.
There is evidence that STOP THE TRAFFIK has delivered work that has contributed towards an improved understanding of the risk of human trafficking. This improved understanding was observed at all levels and across diverse stakeholders within community partnerships, business and financial services partnerships, and its campaigning and policy influencing work. A central part of the work undertaken by STOP THE TRAFFIK relates to improving intelligence related to the risk of human trafficking and modern slavery.
STOP THE TRAFFIK has delivered social media campaigns with local partners in diverse settings that have been successful in raising awareness of the risk of human trafficking. These campaigns have been geographically targeted, for example: Nigeria-Libya-Italy; Kenya and Uganda; Croydon; Fenlands and Lincolnshire-Lithuania. These have been collaborative with STOP THE TRAFFIK reaching out to various partners in the different locations. Local partners also benefitted from increased interactions with target communities and groups, such as increased calls to helplines, increased attendance at drop-in centres, etc.
In its work with corporates, STOP THE TRAFFIK has again used the relationship development approach to drive success in its work. Its work with businesses is also intended to raise awareness of the risk of modern slavery and human trafficking. This is done through training and a supply chain audit process related to the sector, in order to help them respond more effectively.
At a community level, STOP THE TRAFFIK has delivered programming across four areas in the UK that has included developing partnerships and networks in Manchester, East Sussex, Colchester and London. For example, this has included setting up the Business Forum in Greater Manchester to support businesses responding to the Modern Slavery Act section 54, and wider collaboration and intelligence-sharing across public, private and third sectors.
In increasing the perception of the risk of human trafficking, STOP THE TRAFFIK has prioritised relationship development as a key approach to designing and delivering interventions that have delivered immediate results. This approach proved to be crucial in ensuring initiatives were well designed, relevant and effective.
2 Gained knowledge of signs and how to respond appropriately
There are no findings in this section that match the filters you have chosen.
Under this second area of change that STOP THE TRAFFIK has been working towards, we have found evidence that it does share knowledge of the signs of human trafficking and information and knowledge about how to respond appropriately to potential human trafficking situations. STOP THE TRAFFIK does this through training initiatives and awareness raising events and activities.
To enable its community and business stakeholders and the public to respond and thus to prevent human trafficking, STOP THE TRAFFIK has acted to share knowledge of signs of modern slavery and human trafficking, and information about how to respond to it, that is tailored to the stakeholder it is addressing. Thousands of people have participated in STOP THE TRAFFIK facilitated activities across communities and amongst its business partners in the past three years.
A key message is that modern slavery is hidden in plain sight. Knowledge of signs to look for and how to respond appropriately are key elements of interventions in order to enable the public to respond where they have concerns. Participants in STOP THE TRAFFIK facilitated training or other activities reported an increased likelihood of knowing what to do when presented with a potential human trafficking situation.
Awareness raising activities were also targeted at professionals. Some focused on single professionals like prison staff, taxi drivers, Modern Slavery Single Points of Contact, business directors, operational managers and frontline staff in key industries such as hospitality and banking, while others entailed multiple agencies. Professionals learned about the risk of modern slavery and human trafficking, indicators to help them spot it, their responsibilities as first responders or according to their organisation’s policy, and appropriate actions to take.
By working in partnership with a range of stakeholders, STOP THE TRAFFIK has developed and shared knowledge of the signs of human trafficking and information and knowledge about how to respond appropriately to potential human trafficking situations. It has done so using a range of intervention types and approaches that have been appropriate, relevant and of value to its stakeholders and audiences.
3 Adopted preventative behaviour regarding a potential trafficking situation
There are no findings in this section that match the filters you have chosen.
A wide range of behaviour changes have been observed resulting from STOP THE TRAFFIK’s work with its stakeholders. In most cases we have seen reports from participants of intended changes and use of knowledge. It is significant that we have been able to record a broad range of behaviour change.
There is some early evidence that people who have received training from STOP THE TRAFFIK are using it to report on potential trafficking situations and those who have been targeted by STOP THE TRAFFIK’s social media campaigns have responded positively to the campaign and reached out to the local partners. This is a difficult area of change to measure as the organisation collects and has access to limited training-based feedback and generic social media reporting, which describes intended, but not actual behaviour change.
Much of STOP THE TRAFFIK’s work to deliver changed behaviour is done at a systemic level in a collaborative manner with partners across sectors.
Other changes in behaviour reported by partners included: encouraging the public to download the STOP APP and report incidences, raising awareness of the risk of human trafficking and encouraging individuals to change their behaviour, undertaking referrals to the National Referral Mechanism or completing Duty to Notify forms, reporting concerns to a specified email or telephone number or referring potential victims to the police force.
Behaviour change has also been reported by STOP THE TRAFFIK’s corporate partners, especially with regard to changes in reporting and monitoring processes and supply chain audits. System and process weaknesses were often highlighted during training sessions with corporate partners, with potential changes to systems and processes also being discussed. Corporate partners also reported staff being more aware of the potential for modern slavery and human trafficking and taking steps to respond to situations that they thought could represent a potential case of trafficking.
Organisations that have partnered with STOP THE TRAFFIK on social media campaigns have reported an increase in demands on their services as a result of the campaigns.
One element of STOP THE TRAFFIK’s Community Practice remit is to enhance the support and protection of victims. Although this does not feature as part of the prevention-focused evaluative framework, much of this work delivered results during the evaluation’s time period and it is apparent that there have been a variety of successes in relation to this objective, including providing systemic changes in organisational practice.
Another expectation of STOP THE TRAFFIK’s Community Practice programme was not only awareness-raising training of frontline professionals but also an increase in referrals of potential victims of modern slavery. This has been observed through formalised channels developed within local authorities and informal channels, such as from the taxi drivers trained by the organisation in Colchester.
In summary, there has been a wide range of different behaviour changes resulting from STOP THE TRAFFIK’s work. Behaviour change has been observed in most stakeholders and at both individual and organisational levels.
4 Approach of the anti-trafficking sector
There are no findings in this section that match the filters you have chosen.
STOP THE TRAFFIK’s unique approach of collaborative partnerships is successful and valued by its partners. These collaborative partnerships have resulted in shared data for analysis and circulation amongst its partners.
STOP THE TRAFFIK’s operational model is heavily focused on collaborative partnerships that result in shared data for analysis and circulation amongst its partners. Partners are very positive about working with STOP THE TRAFFIK and report that it is unique amongst the NGOs that they work with in being able to work and speak with the partner’s stakeholders using language that they understand.
STOP THE TRAFFIK has been most successful at working with other organisations at both a local and global level. The work of its Community Practice programme has a key focus of bringing diverse stakeholders together in what sometimes end up as being fully-fledged networks. Outside of this work delivered at a local level, the organisaiton has differentiated itself by being prevention-focused rather than victim-focused, which has led it to work with a wide range of partners outside of the sector, rather than staying exclusively within the traditional anti-trafficking sector. This broader reach and partnership working approach has enabled STOP THE TRAFFIK to extend its influence beyond the anti-trafficking sector.
Where STOP THE TRAFFIK has been able to affect the development of a data-sharing network at a global level, is in its relationship with IBM and the development of the Traffik Analysis (TA) Hub. The TA Hub is in many respects the physical embodiment of the goal that STOP THE TRAFFIK had set for its collect-analyse-share model: a network of organisations and agencies across sectors, all sharing their data and information in such a way that the data can be jointly analysed to indicate possible areas of human trafficking or modern slavery activity or the potential for such activity, and appropriate action can be taken. As stakeholders build up trust in the TA Hub, this membership and data sharing-based network has the potential to thrive and deliver the kind of network envisaged by STOP THE TRAFFIK.
5 Other achievements not included in the hypothesis
There are no findings in this section that match the filters you have chosen.
It is STOP THE TRAFFIK’s involvement with the policy change process that led first to the UK’s Modern Slavery Act and then subsequently the inclusion of section 54 of the Act that is perhaps the single most underestimated impact amongst all of the organisation’s work. STOP THE TRAFFIK worked with Peter Talibart to analyse data and develop a coherent argument that has been used to influence (currently) three legislative processes across three continents: the UK, Australia and Canada. Common to all of them is the requirement of company directors (or persons controlling the company business) to sign off on an annual statement or report about modern slavery in their supply chains. This is crucial for two reasons. Firstly, because it raises awareness of modern slavery in corporate supply chains, which itself is an important result and can contribute towards the reduction of modern slavery feeding into the legitimate economy. Secondly, in most jurisdictions, company directors signing off on corporate statements of one form or another carries legal and compliance implications and associated penalties for getting it wrong. The strengthening of the UK law and the alignment of other jurisdictions with the corporate reporting requirement makes tackling modern slavery at a global level more likely and highlights the influence of STOP THE TRAFFIK’s analysis and the policy submissions made by one of its partners on three separate jurisdictions.
Successes and challenges about STOP THE TRAFFIK’s approach
Click the double arrow icon to explore how successes and challenges relate to the strategic framework.
There are no results in this section that match the filters you have chosen.
What has been clear during this evaluation is the high degree of relevance that STOP THE TRAFFIK’s partners have found in working with the organisation in a collaborative manner. There is strong evidence that stakeholders report that both the process of working with STOP THE TRAFFIK and the results achieved through the interventions are both useful, used and appropriate to the stakeholders’ requirements. There is also strong evidence that sustainability in the results generated by STOP THE TRAFFIK’s interventions is being developed. There is significant evidence that stakeholders have integrated changes into their systems and processes or core way of working, and that these changes are viewed as permanent. Some have had to develop new systems or processes as a result of the work done with STOP THE TRAFFIK, while others have demonstrated changed institutional behaviour with respect to their own partners and supply chains. Local partnerships have also delivered opportunities to effectively disrupt modern slavery and human trafficking. STOP THE TRAFFIK has been identified as a trusted partner and opportunities have been created to develop more sustainable and effective responses based on this learning and collaborative practice.
STOP THE TRAFFIK’s social media campaign work has been very effective, achieving most of the pre-identified goals for each campaign. These campaigns are also well recorded and reported on, with concise analysis conducted post-campaign to inform future design and development. Partners have reported that they believe their partnerships with STOP THE TRAFFIK are very effective. Local delivery partners are an important element of the social media campaign model. Partnerships developed for social media campaigns are often time-limited and can be perceived as being more beneficial to STOP THE TRAFFIK than to the local partner. In these instances, the trust-building between organisations critical to campaign success can be challenging as the organisation is seen as being literally far away from the targeted communities.
What is less clear is how STOP THE TRAFFIK is measuring effectiveness from its end. While a robust and appropriate Monitoring, Evaluation and Learning (MEL) approach has been put in place, the MEL tools are inconsistently applied, making it difficult to identify effective delivery of plans across the four change areas the organisation is working towards. Effective change can be difficult to measure and identify in the moment, especially where the operational model is one of ongoing partnerships rather than defined projects with clear exit dates. This is why the Outcome Mapping and Outcome Harvesting approach used in STOP THE TRAFFIK’s MEL approach is most appropriate. Use of the outcome journals that have been designed should be encouraged, so that during its regular review meetings, the organisation can clearly see the changes taking place in its boundary partners, and can track these changes over time towards its stated goals.
One MEL challenge also relates to the difficulty of measuring the impact of STOP THE TRAFFIK’s work on prevention and on behaviour change. For example, to understand if behaviour changed as a result of awareness raising aimed at frontline professionals, the organisation had started to survey professionals 3-6 months after the training to see if and how training changed their and/or colleagues’ behaviour. This is similar for understanding the longer term effectiveness of its social media campaigns, which should have similar medium- to long-term follow up.
Another challenge to measuring effectiveness is the absence of regular milestones to be achieved between today and when the goal stated in the Theory of Change might be achieved. More structured planning for periods of three to five years might help STOP THE TRAFFIK to clarify what it expects to achieve in these smaller time periods that will help it move towards its goal. Such planning can be developed in a sufficiently flexible manner so as to allow the organisation to take advantage of unforeseen opportunities and still remain on track towards each set of milestones. Moreover, a review of milestones would fit in with the learning environment being fostered within the MEL processes.
What is clear however, is that the most effective work that STOP THE TRAFFIK had engaged in, in order to disrupt modern slavery and human trafficking at a global level has been the analysis and evidence submitted in support of the UK Modern Slavery Act as set out in the section above. This legislative framework has helped drive not only its own work, but the work of many of its partners and, where appropriate, their supply chains.
Conclusions
What appears to have made STOP THE TRAFFIK so successful, and is seen as added value by its stakeholders, is its ability to engage with stakeholders and partners in a manner relevant to the partner’s needs. It is rare for NGOs to be able to speak to businesses in a way that the businesses can understand and appreciate, whilst being able to have a similar relationship with law enforcement agencies.
STOP THE TRAFFIK is highly valued for its collaborative approach and its global reach. Partners and other stakeholders believe that the organisation regularly punches above its weight and has an influence wider than its own scale. At the same time, the organisation is valued for its independence and ability to bring diverse groups together to affect change at a local level too. It is clear that STOP THE TRAFFIK’s work at a local and systemic (campaigns, corporates, FSOs, etc.) level reflect the same collaborative approach to working with stakeholders and partners.
Where STOP THE TRAFFIK is less successful is in understanding its own work and influence internally. The organisation appears to be unaware of the degree of its success and potential and the apparent siloed way of working is prohibiting better flexible working and more responsive approaches to the needs of stakeholders.
STOP THE TRAFFIK’s reference to its model as ‘collect-analyse-share’ is slightly misleading. We have found that the actual model being used and appreciated by stakeholders and partners is ‘partnership-trust-collect-analyse/share-(use)’. Partnership development and building trustworthiness is crucial to the collaborative approach that STOP THE TRAFFIK promotes. A collaborative approach is also much more than a transactional event, as the current model description suggests. In addition, ensuring that partners and stakeholders are well placed to make use of the data that is shared with them and can actually use it, is essential to the overall success of this approach.
While our investigation found that the majority of the hypothesis developed with STOP THE TRAFFIK at the start of the process was valid, there are areas where the hypothesis appears to not take account of what has actually been achieved or is being achieved. As such we have developed a revised hypothesis that outlines these additional outcomes.
The hypothesis shown above as a wheel diagram can also be described as a strategic framework in which STOP THE TRAFFIK has at its heart a recognisable operating model. The organisation uses this model to deliver a range of interventions and programmes with and across a wide range of stakeholders and partners. In effect it makes no difference whether STOP THE TRAFFIK is delivering policy change interventions to an Independent Review panel or training 700 taxi drivers, the same model is in play and if appropriately measured and tracked could move the organisation closer towards its twin goals.
Finally, it is reasonable to say that STOP THE TRAFFIK is a unique NGO with a specific operating approach that its stakeholders find valuable and relevant. With appropriate strategic development, STOP THE TRAFFIK could continue to punch above its weight and deliver significant results against modern slavery and human trafficking globally.
Recommendations
-
Monitoring, evaluation and learning (MEL)
STOP THE TRAFFIK has already started to address its MEL systems and processes and some of these recommendations are already under consideration. We would like to see the organisation apply its MEL process more consistently, develop output-level targets for activities and use them alongside outcome-level targets, and follow up longer term data collection to see whether change sticks. Consider conducting a value for money assessment of STOP THE TRAFFIK’s key achievements, using an approach such as social return on investment (SROI) to understand the value of the social change generated by STOP THE TRAFFIK’s work.
STOP THE TRAFFIK RESPONSE:
This evaluation highlights the organisation’s developed evaluative thinking approach and existing MEL practices. A plan to develop more robust monitoring and more accessible tools to be used and embedded had been put in place.
-
Strategic planning
Structured planning for periods of three to five years, informed by MEL findings and including strategic milestones, might help STOP THE TRAFFIK to clarify what it expects to achieve in these smaller time periods that will help it move towards its goal. This planning is already underway as STOP THE TRAFFIK had prepared a 2019-2022 strategy, and the organisation should integrate findings and recommendations from this evaluation into it.
STOP THE TRAFFIK RESPONSE:
During the evaluation period we had worked on our 2019-2022 strategy, which had now been published.
-
Policy influencing
STOP THE TRAFFIK policy influencing work has had a significant impact globally. Resources should be invested in developing policy work within the organisation to, at the very least, monitor and react with targeted policy change work at an international, national or industry-wide level as needed.
STOP THE TRAFFIK RESPONSE:
We have now added a sixth strategic objective to influence local, national and global systems to embrace an intelligence-led approach to disrupt trafficking. We will be developing an accompanying operational plan to implement, too.
-
Integrated working approach
STOP THE TRAFFIK has a recognised brand. It is viewed as impartial and collaborative in its work to prevent and disrupt modern slavery at local, national and international levels. Its credibility is in part due to its feet on the ground, embedded in local communities and businesses and then extending its reach internationally. We recommend that STOP THE TRAFFIK retains the existing practice teams while breaking down the silos between teams through improved internal collaboration and communication. The organisation should work on developing a management and project management approach that draws on good practices in the Agile and Matrix traditions in order to retain that urgency about its work as it normalises systems and processes and matures as an organisation.
STOP THE TRAFFIK RESPONSE:
One of the reasons for this evaluation was a desire to see how we create impact that is bigger than the sum of our parts, and we worked with the evaluation team to adapt project management approaches to support us working in a more integrated manner.
-
Sharing good practice
STOP THE TRAFFIK had developed practices in a number of activities that are successful and effective. These need to be recorded and should be shared internally and externally. Internal sharing of good practice will ensure that as the organisation grows, it can retain its organisational intelligence and ensure that activities are done in the most effective way. External sharing would enable external stakeholders or those with an interest in preventing modern slavery to see what works, under what conditions and for whom. It will also raise STOP THE TRAFFIK’s profile as a thought leader in the sector†. Over time, this could be evolved into a learning agenda for the organisation, focused on its strategic framework or long-term vision.
STOP THE TRAFFIK RESPONSE:
We have been sharing information about our work in events and conferences, especially in the past year or so. We will be developing a plan to share more good practice as part of our Communications Strategy.
Join our global maling list
More news on how you can get involved and how we're disrupting human trafficking all over the world.
SubscribeAbout
This evaluation was commissioned by STOP THE TRAFFIK and funded by Comic Relief. The evaluation team included Robin Brady and Dr Ruth Van Dyke. It was co-designed by the evaluation team and the organisation, using a participatory approach and focused on STOP THE TRAFFIK’s four change areas, how they contribute to its wider impact and how their interdependence is influenced by that of STOP THE TRAFFIK’s team structure.
STOP THE TRAFFIK
info@stopthetraffik.org
Robin Brady
www.robinstrading.co.uk
Dr Ruth Van Dyke
ruth.vandyke@stmarys.ac.uk