Expert Insights: Monitoring, Evaluation, and Learning for LL

  • Screen Shot 2015-02-13 at 3.36.30 PMOur expert Catherine Borgman-Arboleda is a pioneer in participatory project monitoring and evaluation (PM&E) and especially in MEL: Monitoring, Evaluation, and Learning. She’s a co-founder of Action Evaluation Collaborative and has 15 years of experience in NGO and non-profit evaluation, planning, and training.  Catherine has led evaluations for major foundations, international organizations and NGOs, as well as smaller grantmakers, organizations and collaboratives.

    Catherine Borgman-Arboleda kindly agreed to share her thoughts on monitoring and evaluation as a learning process – a concept central to both business and policy-oriented Living Labs (see, e.g., the resources under the LL tab):

    “Participatory data collection and research methods and tools [such as the Living Lab model] are vehicles, entry points, means to another end, which is really about shifting how we do development. This shift encompasses an awareness of power, its relationship to knowledge and the ability to have expertise redefined by stepping back for other world views.

    When we talk about this work, we need to be aware of who drives it; who takes leadership, who has ownership, sees value, and ultimately feels empowered and has agency because of, and through this process. This is ultimately what will determine success.

    In the spirit of openness and learning, underlying premises of the Tipping Point, it’s important to step back and think about what we are learning. Here are just a few of my personal takeaways and there are certainly others…

    • Timelines need to come from the pace, rhythms, and priorities of the practitioners. This does not lessen the ability to be accountable, but instead requires a process to develop a common understanding of what it means to be accountable, and to whom.
    • The building of ownership is painstaking. It requires a process that allows people’s innate creativity, and ability to think critically, to rise to the challenge, and be the driving forces.   This letting go requires a significant amount of programmatic flexibility and willingness to sit back, re-think about what constitutes knowledge, watch, and trust.
    • Reflective pauses, and turning the mirror on ourselves and our process, is essential. The process must be, by its very nature, iterative. There must be space to acknowledge that something isn’t working, and come up with a new ideas, plan, or approach.
    • Understanding how learning drives change is important. Change can happen to us, or we can be more instrumental in how that change takes place. For that to happen participants need the kind of spaces and time that allows learning and the active challenging of existing power relations that is an inherent part of any learning with others that allows one to use new knowledge and apply it to existing situations.
    • And thus, knowledge and for whom? Perhaps this is the most challenging. The question of credibility and the politics of evidence are points for heated discussion throughout international development and evaluation circles. This discussion continues to consume significant amounts of time and energy; to try and produce knowledge that meets an external set of highly subjective measures. Given that the educational sector generally operates with fairly limited resources, and choices must be made, one set of questions that are perhaps useful to consider are “Who needs to be convinced?”, “why?”, and “What are the costs then to knowledge production and needs?”

    Find our more in Borgman-Arboleda’s blog here.

    And, a bit more about participatory monitoring and evaluation:

Advertisements

The Showcase: Live Blog

Screen Shot 2015-02-05 at 9.00.02 AM

Friday, February 6th, 2015:

The GESCI-AKE “The Sound of the City” Showcase

This is the live blog of the event.

12:45 (UTC+03:00): Let the Showcase begin:

Jerome Morrisey, CEO, GESCI: The more local, the more global! The Living Lab means a continuous process, from upscaling to employment.

Screen Shot 2015-02-06 at 4.53.53 AMThe representative of the ForMin Finland: ICT4D has been an important area  of co-operation. Finland is a strong promoter of ICT development and innovation. Discusses the AKE timeline. Thank you GESCI-AKE for the fantastic programme and congratulations, participants!

Elaine Hurt: Thanks to Duncan and Alan as team leaders who have fundamentally shaped the AKE programme!Screen Shot 2015-02-06 at 5.03.52 AM

13:00 Practitioners are setting up their artifacts for showcasing. Elaine: Please view, ask questions, interact!

Audience is exploring different individual artifacts…

Screen Shot 2015-02-06 at 5.20.24 AMScreen Shot 2015-02-06 at 5.15.17 AMScreen Shot 2015-02-06 at 5.12.42 AM

Screen Shot 2015-02-06 at 5.07.59 AM

13:30 Showcasing the final collaborative products: First, work of the 2012 cohort – stories affected by the wars, especially about the suffering of children.Screen Shot 2015-02-06 at 5.30.41 AM

Screen Shot 2015-02-06 at 5.27.54 AM

Screen Shot 2015-02-06 at 5.29.52 AM

Screen Shot 2015-02-06 at 5.33.43 AM

13:40 Showcasing The Sound of the City (2014) – focus on Nairobi and its urban culture! As a symbol: matatu!

Music video: “This is what the industry needs – full-blown production. All the music, including lyrics, by the practitioners.”Screen Shot 2015-02-06 at 5.36.52 AMScreen Shot 2015-02-06 at 5.36.43 AM

Screen Shot 2015-02-06 at 5.35.45 AM

Screen Shot 2015-02-06 at 5.40.43 AM

13:45 TSoTC game – “3.5 months – a video game – a great achievement!” The game is about experiencing matatu. Everything is by the students, including the sound design. “A massive opportunity in gaming – there are few games that are Africa-oriented, this is a start for something big!”

Screen Shot 2015-02-06 at 5.45.54 AM

Screen Shot 2015-02-06 at 5.44.46 AM

Screen Shot 2015-02-06 at 5.43.02 AM

13:50 Collaboration with Aalto University on LL Research (see the video here). Please share your thoughts here below as comments!

Screen Shot 2015-02-06 at 5.49.16 AM

14:05 Q&A Session – facilitated by Mary Hooker

Screen Shot 2015-02-06 at 6.03.07 AM

“What an exciting day!

This is a collaboration between AKE and Aalto. We need a research model than we can capture our experience. Often we think research is something up there. But research should be for everyone, in what we can all to participate in. Connections to government and policy, to other citizens. An opportunity to us: GESCI team, practitioners, business, policy-makers, end-users…  Now we need to hear from  YOU. We are going to take your ideas and take them to the policy forum. You are doing EDU-tainment, not only entertainment. Generally, training is removed from the real world. How can we combine business and edu needs.

Four areas: leadership, content (local content that is not that much available – as Duncan noted), innovation, sustainability.

What do you think of the results?”

Some insights:

  • Sustainability as a key challenge: innovation/creativity & business skills needed.
  • Technical training not enough – although the education tends to focus on that.
  • Sustainability = innovation and local focus = content. African-oriented. But the consumers are not there because it’s a global market.
  • Policy context: The Kenyan context – policy – sustainability. How can policies support the local market?
  • We need more African stories – an AKE trademark, emphasized in the programme.
  • Who are the consumers? We need to know more about them. Are we thinking about the global market (also)?
    • Can a local product also be international (e.g., as in Nollywood).
  • We don’t yet have enough skills to satisfy the market – so that is the future – the local market. My vision for AKE: we need to build the talent-base, support from the government.
  • Cutting-edge difference.
  • Setting up start-ups – own companies.
  • Policy decisions re: national culture segments need to be made. Ex: publishing in local languages, even without policy support.
    • 2004: culture
    • Emerging policy for national music
    • Mary: matching policy with good practice, research
  • Authenticity = a basic feature of originality. Technology = possibly connecting local – global markets.
  • LL model to other fields?
  • Example of Royal Media Services: Started to collect African content when no one else did; created sustainability.
  • Mary: Summing up: merger of business/edu LL especially considering the positive policy climate for local content! Is this what we are saying?
  • We need to accept the realities of social media platforms and the realities of distribution – how are the business models of the new media landscape?
  • Universal understanding/product of a local/national culture for a variety of audiences.
  • Music a good example – speaks to everyone.
  • Sustainability in terms of professions and messages!
  • Opportunities are there.
  • AKE – we want to bring this product to a bigger level. Is this a viable model to take forward?
  • The full circle: Innovation is a means to essential communication, enabling new social values, and globalization is bringing out the local innovation and need for local culture.
  • Helena: We need to carefully examine the market, demand-side (both skills and the end-users). Next step – the region.
  • Mary: CONTINUE THE DIALOGUE!

    This ended in the crucial question: What is the market?

  • (Biz living labs include users=members of the market, part of the innovation process).

14:50 – CONGRATULATIONS – THE NEW GRADUATES AND THE ENTIRE GESCI-AKE TEAM!

Jerome: “The high quality of graduates: maturity, creativity, commitment, richness of talent. We are so proud of this group.”Screen Shot 2015-02-06 at 7.02.22 AM

Screen Shot 2015-02-06 at 6.59.17 AM Screen Shot 2015-02-06 at 6.59.10 AM

Screen Shot 2015-02-06 at 7.00.05 AM

A Video Greeting for The Sound of the City

The script:

Hello everyone! I am Vesa Saarinen. I’m a sociologist, currently affiliated with the Media Lab Helsinki of the Aalto University, here in Finland. Through this video, I’m sending you my greetings from snowy and icy Helsinki, Finland’s capital.

I hope you’ve had a great day, and I’m sure that your hard work in “The Sound of the City” has produced some great results. I’m really looking forward to seeing your work. And so do the creative professionals who have promised to give their thoughts on artifacts you’ve produced. 

– –

As you probably know, there has been also a “research component” included in “The Sound of the City” project. On this video, I’ll tell you a bit about this “research component”, and present some of the very first results of the research.

This “research component” is done in co-operation with Aalto University’s Media Lab Helsinki. Aalto University is a multidiscliplinary university, combining art, technology, science and design. Inside the Aalto University, Media Lab Helsinki’s mission is to explore, discover and comprehend the new digital technologies. There are three master’s degree programmes in the Media Lab: new media design and production, sound in new media and games. And there are lots of interesting research and arts projects going on, from sound and animation, to games and arts. For more information, go to medialab.aalto.fi or aalto.fi/en.

The “research component” of the “Sound of the City” is mainly done by two researchers, myself and dr Minna Aslama Horowicz. Minna, who is also Finnish like myself, is living in New York City. She is an expert in Living Labs research and policy analysis, and professional in bringing different stakeholders together in collaborative projects. My perspective combines new media studies with my sociological background.

In addition to myself and Minna, several external professionals have assisted the research component, for example, by answering to our questions, by pointing us to interesting material, by writing blog posts, or by helping us by commenting your artifacts. We sincerely thank all of them.

(Living Labs)

“The Sound of the City” is an example of a “Living Labs” project. A “Living Lab” is a research and innovation concept. Their core idea is that innovation happens simultaneously, hand in hand with research. In other words, “Living Labs” are about creating and learning by innovation, so we can come up with better results. Living Labs are an example of what is called open innovation – innovation not only by experts, but by everyone.

You can learn more about “Living Labs” and research related to them from our research blog, located at thesoundofthecity.wordpress.com.

(Living Labs in Creative Collaborative Processes)

Living Labs have become a key tool in creative industries as well as in policy-making because they have radically mixed the roles of those who innovate. They are about the co-creation, exploration, experimentation and evaluation of innovative ideas. They involve user or client communities, not only as observed subjects but as co-creators.

Living Labs are most often used in a regional context, for example in developing policy solutions within in a city. In those cases, Living Labs integrate research and innovation processes within a triangular citizen – government – industry partnership.

Living Labs can also be used within companies to come up with new ideas and products. In that situation, end-users  and different experts of product research and development, technology, marketing and sales, all work together throughout the product design and testing process.

Living Labs are a relatively new concept in education – you are some of the pioneers!

(Researching “The Sound of the City”)

In the research component of “The Sound of the City”, there have been three broader research goals.

First goal is to get information from “The Sound of the City” project and its success, and compare this information to broader research related to Living Labs. How does the research done within “The Sound of the City” reflect to the previous Living Labs research?

Second goal is to collect information and best practises for creating a replicable model for future education-related Living Labs projects. How should the Living Labs projects be organized in an learning environment? What are the differences to, for example, Living Labs projects done in a corporation? What are the “do’s and dont’s”?

And the third goal is to reflect the research process in itself – the possibilities and challenges in doing Living Labs research in three different continents.

In this video, we’ll focus to the first two goals.

(Resarch Methods)

To answer the research goals, we’ve used several research methods.

First, in December, we did a survey questionnaire for you to answer. In that survey, we asked for your experiences, best practices and thoughts on working in creative processes. There’s a printed follow-up survey as well, which we hope you’ll answer today.

Second, we’ve interviewed, or will interview, some participants and stakeholders directly related to “The Sound of the City”. We’ve also interviewed some professionals, to broaden the perspective gained from the first interviews. These interviews still continue, so if you’d like to participate, do contact us!

Third, we’ve tried collaborative and participatory blogging at our research blog at thesoundofthecity.wordpress.com. We’ve tried to build a platform for collaboration and discussion, and written several blog posts related to “The Sound of the City” or Living Labs in general. Do take a look!

And fourth, previous research done in Living Labs projects have given valuable information on the known possibilities and challenges in building Living Labs models around the world.

– –

(Some Research Results)

While the research component is not yet completed, there are some initial results that we find interesting. Here we present them through the keywords of innovation, leadership, content and sustainability, and focus especially to “The Sound of the City”.

First keyword – innovation.

  1. Innovation, as in the mission of upscaling.

All interviewees, supported by the survey results, have noted that innovation as such is not the only goal of Living Labs, or “Sound of the City”. Innovation needs to have social value for end-users, and this should be emphasized in the training. Innovation is seen as a mean, not as a goal. 

  1. Innovation, in curriculum design.

Some participants noted that co-design of the curriculum with the students, prior to the course, would help to tailor the content to the specific needs of the young professionals. Co-creation of the syllabus would follow the general Living Labs model, as presented earlier, where all stakeholders work together from the very beginning.

Second keyword – leadership. 

  1. Both interviews and the survey indicated that you find leadership to be crucial. The vision of the training needs to extend beyond the single project or training that is going on, and the leader needs to keep that vision alive. This is generally the core challenge in Living Labs: how to incorporate various expertise in creative collaboration processes? How to encourage different skills and knowledge while keeping the focus on the whole? One of your thoughts is the clarity of the roles and the project at hand: if everyone knows what is their task, and are taking steps to the same direction, the project is generally better. However: it’s easier said than done.

Third keyword – content, especially in “The Sound of the City”.

  1. Three dimensions of training. Most participants seem to call for three things: 1) tech skills upscaling, 2) creative opportunities, inspiration and support and 3) business development and entrepreneurship training.
  1. Balance between collaboration and individual work. You tell that it’s important to have both collaborative and individual projects. However, collaborative projects across clusters seem to take quite a bit of time, and peer-to-peer learning needs to have space in the training curriculum. Equally, individual mentoring is considered very valuable. In general, this is the observation from many Living Lab experiments: individual insights and development can be unnecessarily overlooked. Therefore, some extra attention needs to be given to individual professional development.
  1. Content: the balance between highly structured work and free brainstorming should be given some thought. The free brainstorming time is needed, and is generally the first stage in a Living Labs innovation model.

Fourth and final keyword is sustainability.

  1. Sustainability and especially follow-up: It is important to build some kind of a continuum to the curriculum, to support the participants beyond the intensive training. There is need to keep the already-built collaborations alive, and help the practitioners network and help one another. Support for start-up development, follow-up gatherings with info session, informal meetings with drinks and snacks – these are some of such possibilities.

A Living Labs model is process-based, and often focused on an end-goal such as a product or an innovation. An educational Living Labs may need to differ in this respect, in order to provide support beyond the so-called “end” of a project.

 

(Next Steps)

 

While “The Sound of the City” is nearing its closure, the research component continues, by deepening the results presented here, and by building a broader educational Living Labs model.

Some questions remaining to be answered are, for example,

How do the educational Living Labs projects differ from the more business-oriented Living Labs in practice? Your interviews and survey answers have given great research material for answering this question, but we should dig deeper to this question.

How should the educational Living Labs’ be practically organized, in comparison to the more business-oriented Living Labs? Especially the question of follow-up processes should be given some thought. And for instance, in your case, how to incorporate a goal of social impact and value, that you expressed, into the entire project from the start.

And this is related to the main question: how could we combine participants’ different unique personal, educational and professional dreams and skills even better to a one, working and innovative Living Labs environment?

If you wish to contribute to the research, we’d be happy to interview you, or to answer your possible questions. Or if you would like to write a blog post on your thoughts, we’d be happy to publish it in The Sound of the City research component blog, at thesoundofthecity.wordpress.com.

– –

Me and Minna, the Finnish research team of “The Sound of the City” research component, would sincerely like to congratulate and thank you for your participation. We have been privileged and happy that we’ve been able to participate to this wonderful project. We can’t wait to dig deeper to the research material you’ve been kind enough to provide.

If you want to know more about the research, about the results or Living Labs in general, go to our blog at thesoundofthecity.wordpress.com. From that blog, you’ll also find our contact information, if you like to discuss further, or to give feedback. 

But, once more, with a feeling:

thank you — and congratulations!