By Meri Ghorkhmazyan, Senior Director of Monitoring, Evaluation and Learning at Mercy Corps
I’ve spent my career building and leading monitoring and evaluation systems and programs across some of the world’s most impactful nonprofits. I am passionate about using data to inform and improve program design. And I’m keenly attuned to the role that digital tools can play in making the most of scarce resources - time and money. After nearly two decades in the Monitoring, Evaluation, and Learning (MEL) space, I joined Mercy Corps. In the last 2 years, we worked through a collaborative process to select and implement a set of best-in-class digital tools across Mercy Corps’ global offices to support both M&E and program teams. I believe this process can serve as something of a blueprint for other organizations needing a similarly robust set of tools to support stakeholders globally. Today, I’m sharing our process in the hopes that it will help fellow MEL, M&E, and evaluation professionals.
I joined Mercy Corps in October 2020 as the Senior Director of Monitoring, Evaluation, and Learning (MEL). Mercy Corps is a global humanitarian organization with over 5,400 team members working across 40+ countries affected by crisis, disaster, poverty, and climate change to meet urgent needs and develop long-term solutions for lasting change. And I immediately discovered a challenge: Our global program teams were using many different digital tools, the selection of which was guided by prior experience or what they were familiar with. A survey showed that some of our programs were on average using 7 to 8 different tools for data collection, analysis, and visualization. For example, in some cases teams were using up to 7 different platforms for data collection across multiple programs in one country.
This was causing problems on a number of levels.
So, we got to work designing and launching a thoughtful process to solve these problems and streamline the way MEL/mobile technologies were selected, procured, and used within programs in the organization. The entire process took nearly 8 months start to finish. Here’s how we approached it:
You need to have the support and buy-in from the organization and its senior leadership to be successful at rolling out organization-wide guidance and agreements with digital solutions. At Mercy Corps, this meant first establishing strong synergies and agreements between the technology and data teams to prioritize this issue and work together to identify solutions (such as M&E and Learning, IT, and Data for Impact teams). Then, reaching out to the M&E and Program Directors and managers at country and regional levels for deeper engagement and learning, together with building support from the organization’s senior leadership.
It also meant getting the right people on the team who had the expertise and management support to roll up their sleeves and do the work. In November 2020, I started building a task force of M&E, ICT4D and IT team members to work collaboratively, assess the true scale of the misalignment across various platforms, and identify solutions for streamlining.
The task force started by clearly articulating a terms of reference, the deliverables it aimed to achieve and what success would look like, as well as how the results achieved by the task force would support the organization, and the deliverables of each engaged team. My team also did the legwork to ensure we could understand and communicate the “what’s in it for me” for each of the task force team members. For the MEL team, one of the biggest priorities to achieve that year was providing clear guidance on how mobile technologies were selected and used to enable the country programs to achieve one of the standards required by the organization’s MEL Policy (The Mercy Corps MEL Policy requires that all programs develop a clear MEL Tech plan articulating how tech is used in programs). For ICT4D and IT teams it was important to see a strong alignment and potential for interoperability across the technologies used by country programs, because it would enable us to move data easily. The program teams were excited about having clear guidance from the organization on the technologies they should use, since very often those were procured at the start up phase, and required additional LOE to research and identify appropriate solutions.
When the task force started Mercy Corps had already conducted 2 MEL tech surveys with the country programs (one in 2018 and one in 2020), mapping the various technologies used at different phases of the data lifecycle, starting with data collection, analysis, visualization, mapping, and communication. We started with a thorough review of the survey results. This helped us understand the baseline with questions around:
One interesting finding in the survey data was that when we compared the technologies used and the needs for training, we noticed that training was required for almost all technologies that teams procured. We wanted to know what guided the selection of those technologies if training on them was in such high demand. Then, we dug deeper to better understand the problem. Between December 2020 and Feb 2021, the task force conducted in-depth interviews and focus groups with country teams across 12 countries to interview staff who selected mobile technologies, those who used it, those who signed off on the purchase requests, as well as those who requested training, used the platforms effectively, or struggled to do so. We selected countries that had ample successful experiences and those who reported challenges. We created case studies and eventually developed a white paper summarizing the findings - both the challenges and outlining 6 potential solutions. This whitepaper helped us engage the organization in a discussion around effective ways of selecting and using technology platforms, and offered guidelines on how to improve our capacity to do it well. One of the recommendations that came out: enterprise purchasing models were necessary for the most frequently used technologies to decrease the time spent on procurement.
We also discovered in this phase that teams were making decisions on what tools to use based on limited analysis and consultation - without any guidelines - and often chose the tools they happened to know. A big surprise to my team came when we learned country offices actually wanted guidance from HQ. We had wrongly assumed our input would not be welcomed.
This process end to end took 200 hours across chairing the program, coordinating meetings, writing and editing the whitepaper and holding discussions and interviews. Each task force member had the support of their management to contribute LOE to the project, which was essential. This sounds like a lot of work, and it is. But it's essential to invest the time to do this right, because the opportunity cost was much higher.
With the problems identified and solutions in hand, one of the most urgent tasks was figuring out how to offer a better procurement experience at a cheaper cost per service for data collection platforms, since those were in high demand. In August 2021, we started with engaging our finance teams, compliance, and procurement teams to identify a process that would allow us to procure and charge the use of those technologies to programs. Once those were agreed upon, Mercy Corps announced a global tender requesting proposals from vendors that offered comprehensive data collection services as outlined in the RFP.
You can see Mercy Corps’ RFP for a Comprehensive Data Collection and Reporting Platform here.
Our Compliance team was able to dedicate a team member to guide me through the procurement process, who was extremely knowledgeable and was able to support me throughout the process. We also had several supportive members of our finance, IT, and other teams that participated in the procurement, and helped me to complete the process. Everyone engaged understood the benefits and prioritized this task. We started with laying out the requirements of the platform in the RFP, using the data on the needs, functions, and performance of the required platform, that we had collected through our case studies of the 12 countries I mentioned above.
Data security, ease of use and available training, as well as a partner who would be willing to walk along patiently with us, help us think through efficiencies, and have our interests in mind as well, were among the strong requirements that came together with the technical requirements laid out in the RFP.
We knew we were driving change internally within the organization (since this would be the first time we were procuring a platform that was fully funded by programs, not the core budget of the organization), and needed a partner who understood the complexities of doing so and was able to work with us throughout the process.
The procurement process took us nearly 5 months (Nov 2021 - end April 2022) and was successfully completed identifying CommCare by Dimagi as the partner for Data Collection and Case Management.
Once you’ve selected your tools, you need to establish a management process around them, document them, communicate with your teams and offer training to support teams to get the most value from them.
As we started rolling out Mercy Corps’ MEL policy, we also rolled out the MEL tech Suite laying out the technologies to support our MEL functions:
Through the case studies of the 12 countries, and also feedback received during the roll out (May 2021 - October 2022) we knew that training on those platforms was in high demand. In September 2021, my team started working on a MEL tech training with a consultant company, Bixal, composed of three major components:
Our partner, Bixal, identified a series of third-party courses that met the requirements laid out by us, and we procured those courses for the identified participants. These included courses from Dimagi Academy trainings, Data Camp, LinkedIn, Udemy and others. Upon completion of the third party courses, the participants then engaged in a cohort-based learning, by tackling each of the technology platforms moving from Level 1 to Level 2, where complexity increases. At the cohort-based learning we also offered additional content related to data privacy and security, as well as other essential skills on data handling. The value of tools comes from teams knowing how to build with them, so this training can amplify the value your organization gets.
At Mercy Corps, our training is offered to MEL, IT, and T4D teams, prioritizing program MEL and T4D team members who have a wider reach and are able to replicate training to our local partners or other team members. Developing and conducting a fully virtual MEL technologies training program like this requires resources and commitment, as well as various sets of expertise (E.g., e-learning curriculum development expertise, deep knowledge of each technical platform, and hands-on facilitation expertise using virtual platforms). In May 2022, we covered Ona, CommCare, PowerBi and TolaData (Mercy Corps’ own platform for program indicators), and will be offering another set of trainings for MAXQDA, STATA and QGIS in a few months.
This four-step process will look different at every organization, and you will need to consider the unique aspects of your organization to tailor it for your needs.
To help guide you, we’ve extracted some of Mercy Corps’ keys to success.
I hope this guide and set of tips is helpful as you consider the unique data and technology needs of your teams.