Managing cultural heritage crowdsourcing projects

The prospect of managing a project with hundreds or thousands of participants, especially with a specific purpose, can seem daunting. a few basic strategies for project management can make a world of difference. In this section, we provide an overview of project management and how it translates to the specific challenges of managing cultural heritage crowdsourcing projects.

Crowdsourcing projects often change through their course, requiring adjustments while the project is already underway. At the outset, consider what might cause problems or raise new challenges using tools to analyze strengths and weaknesses like those discussed in the chapter on “Designing cultural heritage crowdsourcing projects.” For example, a project often proves to be far more, or less, popular than imagined. Having to reset expectations and support a substantially different number of contributors than intended — in either direction — can be stressful!

In this way, project management is likely to involve iterating on the project design, though often without the support, time, and space that a design phase enjoys. One way of managing these kinds of changes is to update some of your goals, checkpoints, deliverables, and deadlines. Capturing outcomes is discussed in the “Designing cultural heritage crowdsourcing projects” chapter for details.

A common issue in crowdsourcing projects is anticipating how participation impacts timelines. Due to the difficulty of guessing how popular a crowdsourcing project will become, timelines often require adjustment during the project. Crowds grow and shrink; one phase might be fast while the next might be very slow. Timelines will have to be adjusted, team size may need adapting, and so on. The key element we try to maintain is to never surprise anyone: by effectively communicating both the initial scope and the progress and variability of your project, you will save yourselves a lot of headaches. To put this another way: while you may not know how many participants will take part in a given phase, you will know in advance that you may not know how many participants you will have! It is okay to plan for uncertainty.

In addition to the “Designing cultural heritage crowdsourcing projects” chapter, the “Connecting with communities” chapter will be a valuable tool in handling these particular complexities.

Ongoing project management

Both in and outside of cultural heritage projects, project managers may be the people who wear more hats than anyone else. The role of the project manager is complex and multifaceted, drawing on diverse skills, from high-level leadership to detail-oriented tracking and analysis. The project manager may work almost alone or have an entire internal team to manage. They may manage the online community, the social media presence, or the expectations of stakeholders. The project manager may be in charge of tracking the goals of the project; reporting, evaluating, and monitoring the metrics used to assess the project, and of course analyzing the project according to those metrics. On smaller projects, many of the other roles described in the “Designing cultural heritage crowdsourcing projects” chapter may also be fulfilled by the project manager, while larger projects may hire professional project managers with specific skills and credentials.

In our experience, the most valuable skill in project management is organization. Understanding where everything is can enable a project manager to leave well enough alone (and avoid driving into micromanagement), or to dive in headfirst when there are warning signs. A well-organized project manager can help to avoid surprises, or at least ensure that your team is prepared when they do arise.

Onboarding, transitioning team members, and documentation

Crowdsourcing projects often operate with only a few key team members and several peripheral members of the project team who may contribute as needed, depending on the phase of the project. Just as participants may experience project fatigue, team members may need to cycle in or out of a project at various points. As we have seen during the COVID-19 pandemic, institutional staff may unexpectedly need to go on leave for days or even weeks. Rather than creating a work vacuum, changes in staffing — whether planned or unplanned — can be handled gracefully with some preparation.

Strong documentation will ease the process of onboarding new team members, transitioning them out of roles, and dealing with unplanned absences. Depending on the nature and size of your project, or institution, you may do this in a series of simple documents, or a git repository, or an entire project wiki. Making this documentation available to the public — including, but not necessarily limited to, your participants — can help other projects learn from your example.

In addition to documentation or a training manual, onboarding a new team member may include participating in the project as a volunteer first or shadowing the current role they are stepping into to observe how it works on a day-to-day basis, particularly at public events. While documentation of processes and technical decisions is essential, documentation should also reflect your project values and include behavioral expectations, such as treating participants with respect or reaching out to underrepresented or underserved communities. For additional information, see the “Identifying, aligning, and enacting values in your project” chapter.

Perspective: finding the right people: DigVentures’ Culture Deck — Brendon Wilkins, Co-founder at DigVentures

In scaling DigVentures1 to meet the need and demand from our crowd to get increasingly involved, the biggest challenge we faced was finding the right people to join our professional team — a quality we call being “DV-ish.” Technical and digital archaeological skills are important, as is having a track record of experience to draw upon. But skills can be taught and experience gained – the absolute joy and enthusiasm for working with the crowd, however, is a quality we have found remarkably difficult to find in our professional sector. To help with this we created a Culture Deck of DV-ish qualities to help onboard new staff and transition team members into community-focussed roles. This is made up of 20 or so slides that illustrate and distil our values as an organisation, with open questions designed to stimulate discussion around how these values could be applied to real world situations. We also have the kind of documentation you might also expect from a social enterprise, such as operations manuals, organisation charts, policies, and project designs. But in bringing new people into DigVentures, the Culture Deck is what has worked best for us in managing the uncertainty of a growing team.

Communication and cultivating relationships

Crowdsourcing projects, especially in the cultural heritage field, are powered by a sense of connection, unique intimacy with both the project as a whole and with the content, so enabling and encouraging that should be a top priority. Communication of any project changes, inflection points, and so on, is a great step in that direction.

Clear communication with your participants in advance of changes to your team is essential. This especially includes when a team member is departing a project and may have created personal relationships with participants. If possible, the departing team member might write a farewell blog post, email, or other communication reflecting on their time with the project and introducing the new team member. Introductory emails can also be sent when a new team member joins the project. Projects may begin with a principal investigator or project manager and onboard additional internal staff only once ready to begin working with external participants.

In addition to making sure that tasks are completed, the project manager may be tasked with creating or managing the connections between an internal project team and any external collaborators, including seeking out and establishing relationships with partner organizations. It is important to balance any partnership commitments with the actual requirements of the project and to be careful not to over-extend your team. A Memorandum of Understanding (MOU) can help to formalize relationships and set clear expectations between partners as to project goals and objectives, including who will carry out what tasks, what are the timelines for tasks to be completed, etc. The project manager or other internal staff may seek ways to tie in a crowdsourcing project with other activities or events at or beyond their institution. Additional information on this aspect can be found in the “Connecting with communities” chapter.

Perspective: building coalitions at the Smithsonian Institution and Library of Congress — Meghan Ferriter, Library of Congress

Bringing people together to accomplish complex goals requires the kind of planning and care we have described throughout this book. I have had the unique opportunity to build new crowdsourcing programs at two large cultural heritage organizations. However, the skills I gathered in supporting my institutional colleagues transfer to most environments, and even to supporting participants. One pillar of this work is communication. In my experience, surfacing shared needs and opportunities for collaboration across an organization and aligned with objectives is made possible through regular and ongoing engagement with stakeholders and leadership. Here are some ways I connected relationship building with communication as I supported my colleagues at the Smithsonian Institution: individual consultation meetings to understand their goals and project needs; training sessions using documentation about the Smithsonian Transcription Center’s administrative functionality; quarterly large group meetings; bespoke written project progress updates; monthly comprehensive reports by email; campaign and challenge meetings and, of course, plenty of individual emails and phone calls to troubleshoot and improve projects.2 That practice prepared me to bring Library of Congress colleagues together to build a new program of engagement, as well as a new crowdsourcing platform at the Library of Congress in 2017. I added open houses, talks, and consultation to my approaches and used the opportunities to reflect on previous successes and lessons learned. I brought that forward when applying my experiences in designing tasks and templates to the challenge at hand (whether for data needs or showcasing collections), convening challenges, or supporting an evolving user experience. Throughout this time at these organizations, I aimed to make communication regular, useful, and enjoyable whenever possible. Sharing knowledge I gained from deep immersion in this work provided an evidence-based platform on which I could not only build stronger relationships with my colleagues but also empower them to take on this practice, too.

Iterating and improving

The project manager should, with the help of project team members, establish phases of iteration for the project. Recall the SMART (Specific, Measurable, Achievable, Realistic, and Time-bound) goals from the “Designing cultural heritage crowdsourcing projects” chapter as you define project phases. These phases may be time-based, such as coming together to review feedback from participants monthly or quarterly, or goal-based, such as a percentage completion of the crowdsourced task or a certain number of participants.

We recommend establishing multiple routes for participant feedback, especially as a way to de-escalate tensions that may arise and allow participants to communicate any frustration they may be feeling around the project. While the project team may receive a lot of feedback informally via conversation or email exchange, online forms or other avenues can also be helpful for completely remote participants.

Participant feedback can then be distilled and communicated to relevant project team members, such as a web developer or data management specialist. Participants are not a monolith, and different audiences will have different suggestions for improvement of the crowdsourcing platform or other aspects of the project. High school students working on a project for class may have different requests than power-users who are accustomed to working across a range of crowdsourcing projects. Feedback from one participant may even conflict with feedback from another. It is essential to consider participant feedback alongside project goals and the availability of your team to determine what changes should be made and when. Remember to consider accessibility when implementing feedback — it is vital to prioritize improvements to accessibility over other preferences.3 These may be decisions for the project manager, or a shared decision made by the entire project team, depending on the norms of your institution.

Perspective: responding to participant feedback on the Zooniverse platform — Samantha Blickhan, Zooniverse Humanities Lead

We get a lot of very useful feedback from participants across the various projects on the Zooniverse platform. Some of our community members have been around for over a decade, and their feedback is crucial to the health of the platform. However, as a small, grant-funded team, we aren’t always able to address issues right away. A key part of how we communicate with participants about feedback is to clearly articulate whether or not we will be able to make their requested changes or incorporate feedback, and to include an estimated timeline if we will. We have found in the cases of the former that a direct explanation of why a requested feature or update cannot be addressed immediately is often sufficient explanation, but it’s up to our team to make sure we are circling back on these participant requests and keeping people in the loop. For example, if participant feedback from a beta test leads to a direct change in a project or tool, we do our best to communicate the source of the change as part of the update announcement (e.g., “Based on your feedback, we made the following changes…”). In this way, we keep participants informed about how direct feedback impacts the platform, what scale of change is possible at a given time, as well as what pathways exist for requesting bigger changes (e.g., opening a GitHub issue or submitting a request via our ticketing system).

Touchpoints and ongoing maintenance

In our experience, using project design as the core of tracking and progress documents has been a helpful starting point. The design process will generally include many of the important elements of your project that would benefit from regular check-ins as well as ongoing maintenance. Evaluation and quality control measures can be another source of inspiration of what to track. You can read more about evaluation and quality control in the “Evaluating your crowdsourcing project” and “Working with crowdsourced data” chapters.

In general, tracking documents are useful, but it is vital that they remain active and maintained. A messy spreadsheet where you go to dump a single line of text that is never read again is not helpful. Tracking documents that fall by the wayside should be identified early and labeled as such. It is ok to let a document fall into disuse, but you should communicate to your team that it is something no longer actively being maintained, to avoid clutter and confusion. Some suggestions of project management tracking tools are available in the “Designing cultural heritage crowdsourcing projects” chapter.

Cultural heritage artifacts are often irregular and unique by nature, and that irregularity may extend to important organizational elements. What is in a collection, how it is organized or structured, and the actual structural organization of the parent institution will all deeply influence how your project can approach and complete its work. These organizational structures must all be carefully considered and addressed. We frequently find unexpected variations in organization or content that are not what, or where, we thought they would be. In a similar vein, cultural heritage artifacts may have spent some time in less-than-ideal circumstances, which can lead to damage or disorganization. In both of these cases, it is important to empower and encourage your participants to draw attention to these issues, as well as ensure that they have a pathway to provide this type of feedback.

Frequently asked questions

Institutional staff members external to the crowdsourcing project may have questions for those who are directly involved. Some common questions from colleagues are listed below along with sample responses. Depending on the level of familiarity the person doing the asking has with crowdsourcing, these questions can take a skeptical view of the process, and in light of this we also offer examples for reframing these questions more positively.

1. How can we trust the accuracy of the results? Crowdsourcing has been shown to produce fairly accurate and reliable results. Reframing: Even if the results are not 100% accurate, will they bring more access or value to a collection than we currently provide?

2. Why can’t staff do this task? Staff have many job responsibilities already. Staff time is limited. Outside perspectives are valuable. Reframing: Staff are welcome to participate! Welcome your institutional colleagues as knowledgeable participants.

3. How will the results be used? Who owns them? Aren’t you just asking for free labor? This may vary by project, but results should be used to benefit both the public and the institution. Results should be made publicly available and openly accessible. Reframing: How might the results be used? We may not know all the potential uses of the results of our crowdsourcing project. It  is wonderful to think of potential uses and make sure they are accounted for in the technology choices.

4. Are we inviting vandalism? Most participants will undertake the task in good faith. If there are incidents of vandalism, we have policies in place to address them (see the section on Governance in the “Connecting with communities” chapter). Reframing: All public interactions contain some risk, even lending a  library book. Risk offers an opportunity to plan carefully.

5. Will this replace me or my job? Aren’t you just asking for free labor? This is often-unexpressed anxiety that can underlie many of the questions from staff. Crowdsourcing can never replace staff, but we all need to be open to changing how we work as technology evolves. Reframing: What could you do if you were able to use crowdsourcing to assist with some tasks? What new communities could you engage meaningfully with a collection through crowdsourcing? Even conversations that begin from a defensive place are an opportunity to get a colleague on board, as long as you approach them honestly and openly.

Project completion and resolution

In this section, we look at the important and often overlooked process of closing out your project, documenting the results, and preparing for the possibility of future work. While we often think of these tasks as occurring after a project is complete, many of these steps actually start before it officially ends. Some of these issues will be covered more completely in chapters such as “Evaluating your crowdsourcing project.”

While all good things come to an end, the nature of crowdsourcing projects in the cultural heritage world makes shuttering them particularly complicated. Dedicated participants will often resist losing the work and social interaction that has become important to them. Cultural institutions can also grow attached to the positive response to a crowdsourcing project and maintain it far past the planned sunset date. For some large institutions with endlessly massive collections such as the US National Archives and Records Administration,4 the Smithsonian Transcription Center,5 and the Library of Congress’ By the People,6 there is often no planned end to a project, as they can continue to bring in new records or documents for participants to work on. Even in cases like these, the project life cycle may describe individual phases of the larger institutional effort.

Having a clear project end date–or at least a clear end to distinct phases of the project — will allow your team to plan for important elements of the work, including changes in technology, project goals, institutional alignment, and relationships with partners and participants. Closing your project begins with the initial project design and continues through the entirety of your project lifecycle. a strong start to your project with clearly articulated values, goals, and objectives (as described above and in the “Designing cultural heritage crowdsourcing projects” chapter) will make ending the project much easier.

Tracking, documentation, and evaluation

At the end of your project lifecycle, you will want to be able to demonstrate the successes of your project. Fortunately, the groundwork for describing and documenting the impact of your project exists in the planning process, particularly in defining your goals and outcomes. Tracking metrics that indicate progress towards your outcomes along the way allows your team to show achievement as well as the growth of the project over time. Most projects track basic metrics including website analytics, sources, number of participants, and number of completed pages, items, or tasks. Funders and institutional boards frequently require regular reporting on project metrics. Your team can check and document these measures regularly to consistently track how your project is progressing. You can read more about project metrics in the chapter on “Evaluating your crowdsourcing project.”

Information collected often contains Personally Identifiable Information (PII) about participants. PII is information that would allow individuals to be identified or contacted online (including email addresses). This information needs to be protected or appropriately destroyed at the end of your project. Data protection laws and practices differ from country to country and should be consulted directly.

In addition to these quantitative measures, tracking qualitative information about the project goes beyond how much is being done to provide some sense of what the activity means. Interactions with participants, partners, and other staff via discussion boards, email, phone calls, and in-person conversations give insight into the experience of the project as it happens. Most cultural institutions value the quality of the experience — engagement, sense of contribution, learning — just as much as the number of records produced or pages transcribed. This quality cannot be seen through quantitative metrics alone.

Documenting the thoughts of contributors when they occur saves time at the end of the project when individual comments and quotes might be lost in the overwhelming number of forum posts and messages. Many projects blog about these discussions as they happen7 while others simply keep a file (sometimes still a paper file) of comments they wish to remember. At the US Holocaust Memorial Museum, the project manager for the History Unfolded project sends regular staff and stakeholders “happy-grams” that document (usually positive) comments or reports from partners and participants. However they are tracked, these details flesh out the quantitative statistics of the project to tell a more complete story.

Anecdotal reports of participants’ experiences can help your understanding of the project, but cannot replace formal evaluation. When possible, a summative evaluation can help provide a formal report on the impact of your project. Ideally, the evaluation process starts long before the project comes to a close and builds on the carefully defined goals already established. While summative evaluation is associated with the end of a project, some forms of evaluation will need to occur while participants are still active, before your project has ended. Some foundations require an outside evaluator for the summative evaluation and, even when not required, external evaluation helps increase trust in the reliability of results. Evaluations represent additional time and expense even when conducted in-house, and are best included in early planning. Please see the “Evaluating your crowdsourcing project” chapter for more information on this process as a whole.

Technology and maintenance

Technology can change more quickly than your project and vice versa. Do not assume that the software you employ will continue to function throughout the entirety of your project lifecycle. Changes in security and operating systems, updates to licensed software, and other factors can alter technical functionality mid-project. Bespoke systems or customizations of existing software can be retired, but they may no longer function when you return to them if enough time has passed. Good documentation, including a thorough accounting of dependencies, can mitigate this problem. Any custom code created for your project should be stored in a repository of some kind, both for preservation and in service to the larger community.

The end of your project will offer your team the opportunity to review the effectiveness of the tools you use, to make final backups and migrate data, and make recommendations for changes, adaptations, or application of tools for other projects. Continued support of outdated software can consume institutional resources that could otherwise be used for new projects and priorities. Closing the project (or at least closing out the development phase) can help to free up funding and developer time. Ideally, your project planning process should account for ending support to project software.

Please see the “Aligning tasks, platforms and goals” chapter for information on software, and the “Working with crowdsourced data” chapter for more on planning for the longevity of your data.

Partners and participants

Perhaps most importantly, you will need to plan for the impact on partners and participants of your project ending. These groups might have very little agency in the decision to end your project while they are still highly invested in the work. Ending a project gracefully ensures that the relationships and goodwill built throughout your project do not evaporate with the final contribution.

Project teams can help prepare participants for a project closing by establishing open and transparent communication from the start. Clearly articulating important decision points as well as the anticipated lifespan of your project can help to avoid the appearance of sudden changes. Regular recognition of important milestones for your project can communicate appreciation for contributions as well as approaching changes. Teachers among your participants will want enough time to plan for replacement classroom activities. Other participants may want the opportunity to find new projects to work on. In a perfect world, your institution would announce a new crowdsourced project at the same time it retires the old one. If not, perhaps your colleagues at other institutions will welcome your project participants into their community. If you know of existing similar projects, do not hesitate to make your participants aware of these opportunities, or invite project team members to signpost their project availability via your project message boards, e-newsletters, or any other communication space you may use.

Conducting a retrospective

An end-of-project retrospective offers team members and stakeholders the chance to honestly and openly discuss and record the highs and lows of your project.8 Cultural heritage crowdsourcing projects particularly need to surface the voices of participants and partners in the retrospective critique. Some project managers might use fairly structured techniques for this process. For example, those using agile methodologies may already be doing detailed retrospectives throughout each stage of their project. Others might have a more informal approach and only conduct a retrospective discussion at the very end. Whatever your approach, a retrospective helps maintain the working relationships between team members, demarcates the end of the project, and paves the way for improving any subsequent crowdsourcing projects you and your team may wish to undertake.

However you conduct your retrospective, some important features include:

1. Creating a safe environment for the team and stakeholders to openly raise any concerns they have with the operation of the project

2. Making sure each member of the team has the chance to be heard

3. Surfacing the perspective of contributors and participants

4. Covering both the successes and the shortcoming of the project

5. Documenting the discussion and incorporating the lessons learned into planning for new projects

Summary

Project management skills are essential in running a crowdsourcing project. Many projects will not have the scale or budget required to hire a professional project manager, but project management tools and guides to supplement the crowdsourcing-focused suggestions within this chapter are accessible. In smaller crowdsourcing projects, colleagues wear many hats, and project management tasks may be balanced with other duties in or beyond the project.

By its nature, project management will intersect with every area of work and can prepare the various paths for other team members. Documentation and planning are central to these tasks. Documentation that is readily accessible and identifiable is important at all stages, not least in case of turnover in the team to avoid losing knowledge. Planning gives you the confidence that the team will cope with any potential mishaps, and guidance for regular pieces of work, such as communications, expected and unexpected changes, feedback and iteration, tracking progress, evaluation, uploading new content, processing data, and project closure.

As with project values which exist whether you make them explicit or not, your project will manage itself one way or another without these tasks being deliberately taken on. A project may succeed without careful and conscious project management if you are lucky, but where a project fails it will most likely be because it was not well managed. How you manage your project is another opportunity to embed and practice your values.