Waging Culture 2012: Methodology in short
The Waging Culture survey is motivated by a desire to establish a clear picture of the socio-economic status of visual artists in Canada. While there is some research being done on the sector (see Guy Bellevance’s 2011 The Visual Arts in Canada: A Synthesis and Critical Analysis of Recent Research for an overview of the field), other research projects are based primarily on Census data.
There are some significant issues with using Census data in researching visual artists. As the Census accounts only for the “main” occupation of an individual, those artists who hold day-jobs are not counted as artists, and thus a significant set of artists simply disappear into the need for simplicity in defining occupation. In addition, there is no breakdown of the various source of incomes for those who are identified as artists. In both instances, these lapses can and do lead to significant misunderstandings of the socio-economic health of artists. For example, while the median income of artists in the 2007 study was $20,000, this included income from all sources. Income from studio practice alone, however, was negative $556.
That said, the data derived from the Census can, and is, useful, particularly in tracking trends within the sector. Alas, with the Conservative governments decision to suspend the mandatory long form for the 2011 Census, this is no longer a viable option. The results of this shift in policy have yet to be fully appreciated, but the effects are significant. We had hoped that with this second iteration of the study, we would be able to draw more from the Census in our analysis. In addition, while the Census data may not be complete, it is possible to use the data from the Census as a secondary check on our own analyses. Alas, this is no longer possible. It has also made the necessity of repeating the Waging Culture survey more pressing.
This is the first posting concerning the results of the 2012 Waging Culture survey, and deals with our methodology. (For a more extensive description, please see the 2007 report which outlines in much more detail the usage of RDS sampling.)
What is a Professional Artist:
Any study needs to define its terms, and ours is no exception. In developing the first Waging Culture, we reviewed the literature on definitions of artists extensively, and came to the conclusion that the Canada Council definition was the most useful for our purposes. We thus retained this definition for Waging Culture 2012.
“The Canada Council defines a professional artist as someone who has specialized training in the field (not necessarily in academic institutions), who is recognized as such by her or his peers (artists working in the same artistic tradition), and who has a history of public presentation or publication.”
Our target audience is, in essence, a hidden population. There is no master list of artists in Canada, and without such, generating a statistically representative sample is a difficult task. In our first study, we used Respondent Driven Sampling, a technique developed by Douglas Heckathorn, a sociologist at Cornell University, which allows for the use of peer-referred snowball sampling to generate a statistically valid sample of hidden populations. (A more extensive description of the procedure is available in the methodology section of the first Waging Culture report, while a more extensive description is available on Heckathorn’s website.) We used the same methodology in this study as in the first, with a few minor changes. Instead of a two-stage survey, we integrated the demographic and financial questions into one questionnaire.
In short, we first asked a group of 150 artists to complete the survey. At the end of the survey, these ‘seeds’ were asked to refer 10 additional potential respondents. Each unique referral was in turn asked to complete the survey and refer 10 additional artists (if an artist had already been referred by a previous participant, they were not asked again). In this way, the survey then becomes a self-propagating chain of referrals (the classic snowball survey). All the information on referral chains were recorded, and the number of times that a potential participant was referred was also recorded (some were referred only once, others were referred up to 10 times).
The results of the survey, as well as the referral chains and the number of times an artist was referred (an indirect replacement for Heckathorn’s social circle variable), were loaded into the RDS Analysis Tool 7.1 software package, a software package developed by Heckathorn, et alia, which analyzed the sampling, and generated weighting factors which counteract the shortcomings of snowball sampling. These weighting factors are then used in the analysis of the data.
Starting in June, 2013, we sent the survey to 150 seeds, or first-wave candidates (a sample of the survey is located here). Of these, 47 responded (and 27 supplied referrals). From these initial referrals, we continued to sent invitations through mid-December. In total, the survey was sent to 1,514 potential respondents, with 391 successful completions. The total response rate was around 28%.
In comparison, the response rate for the first half of the 2007 survey (which dealt with demographic questions) was 34%, but the second half of the survey (financial questions) had a response rate of 46%; the end result was a net response rate of 16%. Nevertheless, the distribution of responses (with a higher percentage earlier in the process), meant that the absolute number of responses in the earlier survey was approximately 560.
If we had used traditional sampling techniques, our 391 responses would imply that the data we collected would be within 4.96% [confidence interval], nineteen times out of twenty [confidence level]. In general, Heckathorn qualifies his method by suggesting that one calculate the confidence interval based on half the total sample, which would give us a confidence interval of 7.01%. Not ideal, in that 5% is considered the ideal threshold, but nonetheless the data we collected is for the most part indicative of the state of the arts.
The smaller sample, alas, means that some sub-groupings within the study did not capture enough respondents to be useful to report. These areas will be indicated in the analysis that follows. In addition, some of the difficulties of snowball sampling skewing showed up, with a higher than expected responses from artists 25 to 34 based in Toronto and lower than expected francophone artists within Québec.
Had we been able to use the protocols of RDS sampling techniques more completely (offering rewards for completion; using direct invitations from respondents to referrals), this oversampling might have been mitigated in the analysis, but this did not happen. While this does have implications for comparisons between some sectors, this does not in general affect the quality of the data within these subsectors. Again, we will flag areas that are potentially misleading to the best of our abilities.
This concludes the first post on the 2012 survey. Stay tuned for future installments, starting with our next posting on demographics (still in process, but we’re getting there). If you want to keep up to date on our releases, please follow us on Twitter: https://twitter.com/WagingCulture
Or join the AGYU Facebook page: https://www.facebook.com/ArtGalleryofYorkUniversity
This is one of a series of mini-reports on the results of the 2012 Waging Culture survey, a study of the socio-economic conditions faced by Canadian-resident professional visual artists. Supported by the Art Gallery of York University, it is an undertaking of Michael Maranda. This is the second iteration of the survey. For other mini-reports, and for the full 2007 report, click here. Comments and questions may be directed to firstname.lastname@example.org