Appendix A: Research Methodology
Overview
The design of this pilot study evolved based on valuable insights from formal advisors and many informal conversations with museum leaders. At its core, this is a study of workplace equity and culture in art museums, with responses invited from all workers within participating institutions. As the research proceeded, it became clear that HR officers and museum directors each have distinct areas of insight relevant to these themes that would be valuable to include in the research, motivating a separate additional survey for staff in those positions. In total, 1,933 museum staff participated in the Museums Moving Forward research initiative through 54 art museums. In addition, 56 staff from non-participating institutions completed the survey. The response rate for the staff survey among participating institutions was 27% (1,933 responses of 7,141 invites), and the response rate for the museum director and HR officer surveys was 90% (each had 48 responses of 54 invites).
Research Process
The research team developed the MMF survey instruments through a collaborative process with MMF’s advisory board, conversations with art museum leaders, and a series of cognitive interviews testing the instruments. Six one-hour cognitive interviews were conducted with a mix of HR professionals and other staff within art museums to refine question-and-response option wording and to identify areas of confusion or ambiguity. Cognitive interview participants were provided with a $100 incentive in appreciation for their time and feedback.
The MMF team developed a set of three core criteria for participating museums on which it was seeking a range of participants: budget size, geography, and type (collecting/non-collecting, modern/contemporary, encyclopedic, culturally specific, college/university based, and city/county/state/government affiliated). The MMF research team and advisors reached out through their networks to share the open call for any interested institution to participate in the pilot study. One-on-one meetings were held with more than fifty art museum leadership teams to share the opportunity for participation, and a webinar was hosted through the Association of Art Museum Directors (AAMD) to share the research opportunity.
Participating institutions were promised their own institution’s results benchmarked against relevant institutions (by geography, budget size, and type). For the staff survey, participating workers were also provided with the same benchmark comparisons for their institution and the field as the leadership team received. For the HR and director surveys only the officers who completed them for their institutions received access, and, again, benchmarks against comparable institutions were provided. There was no monetary cost for institutions to participate. However, institutions were required to receive at least ten responses to see their organization’s individual staff survey data in order to protect the privacy of participating workers.
Interested institutions had the option of distributing the staff survey links themselves or providing their staff mailing list to the MMF team to distribute the survey (31% were distributed by MMF). The surveys were open to all respondents between October and December 2022. An update was sent to each participating museum midway through data collection with an update on their sample size to date. Any institutions with fewer than ten responses were still provided access to fieldwide aggregate data and relevant benchmark comparisons based on budget, size, and institution type.
The research team worked to ensure complete confidentiality of responses by partnering with two professors of computer science at Boston University (BU)—Dr. Gabe Kaptchuk and Dr. Mayank Varia—experts in cryptography and data security. The BU team implemented a survey platform that used a technology called secure multi-party computation (MPC), a way of extracting collective insights about data without exposing individuals’ responses to anyone on the data analysis team.
Execution of the MPC protocol was done collaboratively between members of the MMF team, Dr. Kaptchuk and Dr. Varia at BU, and Kinan Dak Albab, a PhD student at Brown University. The implementation relied on the JIFF MPC library, and additional details about the computational techniques used will be shared in a separate report. MPC has been used to collect sensitive information to study important questions, like gender and racial wage gaps across the city of Boston.23 While this approach provides total confidentiality of responses, it also limits the type of analysis that can be conducted on the data (no weighting or regressions were available).
A set of analysis queries or splits for the data was determined in three main categories: organizational characteristics, individual demographics, and individual job characteristics. Four splits were run on organizational characteristics: region,24 budget size, collecting/non-collecting, and institution type. An additional five individual demographic splits were run, including race/ethnicity, gender, disability status, sexual orientation, and generation. And five splits were run on individual job characteristics: seniority, department, union membership, full-time or part-time status, and if the worker had any experience of discrimination or harassment in their current workplace. Two additional questions not included in the report appendix were asked of museum staff: whether they are employed directly by the museum or through a government or university, and whether their position is endowed. These questions did not contribute meaningfully to our understanding of the data, but they are available in aggregate upon request.
Based on early discussions of findings with staff in which cost of living was hypothesized as a potentially important influence on pay and promotion, we developed a new split based on cost of living with data from the Council for Community and Economic Research. This zip-code level data ranged from 85 (Mississippi Museum of Art) to 176 (Honolulu Museum of Art), with a score of 100 representing the average cost of living in the United States. We grouped museums into one of three categories: low (85–99), medium (100–130), and high (131+), with groups of approximately equivalent sizes.
Data was provided to the leadership teams of participating museums on April 1, 2023, and the staff survey results were available to all staff who completed the survey on May 1, 2023. A series of three to five convenings to talk through the data and discuss the meaning for the field were held for museum directors, HR officers, and all museum workers. The MMF team polled participating institutions to determine what topics the group wanted to discuss. These sessions were immensely helpful in shaping the storylines included in this report, and the research team would like to thank the participants for their time and feedback.
Profile of Participating Art Museums
The MMF team aligned geographic categories for participating museums with those used by the AAMD: Mid-Atlantic, Midwest, Mountain Plains, New England, Southeast, and Western. Representation was relatively evenly distributed across the regions, with somewhat lower participation from New England (4 museums). In terms of budget size, there were quite a few museums in the $2.5–$5 million budget range (18) and at least three museums in each of the seven budget ranges that were used for analysis. When divided between budget sizes of less than and greater than $10 million the split is almost even, with 28 institutions under $10 million and 26 institutions above $10 million in annual operating budget. The majority of institutions (44) were collecting, with ten non-collecting institutions participating. Five or more museums of each type participated, with culturally specific (5) the least common, followed by city/county/state/government affiliated (7), encyclopedic (8), college/university affiliated (11), and contemporary or modern (17).