Cultural consensus theory is an approach to information pooling[1] (aggregation, data fusion) which supports a framework for the measurement and evaluation of beliefs as cultural; shared to some extent by a group of individuals. Cultural consensus models guide the aggregation of responses from individuals to estimate (1) the culturally appropriate answers to a series of related questions (when the answers are unknown) and (2) individual competence (cultural competence) in answering those questions. The theory is applicable when there is sufficient agreement across people to assume that a single set of answers exists. The agreement between pairs of individuals is used to estimate individual cultural competence. Answers are estimated by weighting responses of individuals by their competence and then combining responses.
Theory
editCultural consensus theory assumes that cultural beliefs are learned and shared across people and that there is a common understanding of what the world and society are all about.[2] Since the amount of information in a culture is too large for any one individual to master, individuals know different subsets of the cultural knowledge and vary in their cultural competence. Cultural beliefs are beliefs held by a majority of culture members. Given a set of questions, on the same topic, shared cultural beliefs or norms regarding the answers can be estimated by aggregating the responses across a sample of culture members. When an agreement is close to absolute, estimating answers is straightforward. The problem addressed by cultural consensus theory is how to estimate beliefs when there is some degree of heterogeneity present in responses. In general, cultural consensus theory provides a framework for determining whether responses are sufficiently homogeneous to estimate a single set of shared answers and then estimating the answers and individual cultural competence in answering the questions. The theory is designed for the estimation of “culturally correct” answers to questions that are unknown a priori to the researcher, as well as item response effects (e.g., knowledge level, response biases, item difficulty).[3]
Cultural consensus models do not create consensus or explain why consensus exists; they simply facilitate the discovery and description of possible consensus. A high degree of agreement among raters must be present in responses to use consensus theory – only with high agreement does it make sense to aggregate responses to estimate beliefs of the group. Although there are statistical methods to evaluate whether agreement among raters is greater than chance (Binomial test, Friedman test, or Kendall's coefficient of concordance), these methods do not provide a best estimate of the “true” answers nor do they estimate competence of the raters. Cultural consensus theory can estimate competence from the agreement between subjects and then, answers are estimated by “weighting” individual responses by competence prior to aggregation.
A very important feature in the aggregation of responses is that the combined responses of individuals will be more accurate than the responses of each individual included in the aggregation. Reliability theory in psychology (specifically, the reliability coefficient and the Spearman–Brown prediction formula) provides a mathematical estimate of the accuracy or validity of aggregated responses from the number of units being combined and the level of agreement among the units. In this case, the accuracy of aggregated responses can be calculated from the number of subjects and the average Pearson correlation coefficient between all pairs of subjects (across questions).
Usage
editTo use cultural consensus theory, at least three assumptions must be met:
- Informants must be asked a series of questions. The questions should all be on the same topic and at the same level of difficulty. This assumption concerns the homogeneity of items[4] and means that items should represent only one topic or domain of knowledge and that competency should be consistent across items, so that if someone does well on one subset of questions, they should also do well on another subset of questions. Responses to questions are not corrected, recoded, transformed, or reflected as they are with knowledge tests and attitudinal scales because the purpose is to use the original responses to estimate culturally correct answers.
- Each informant should provide answers independently of all other informants. This means that answers should be provided by individuals and not groups, and without consultation with others. Consensus models are not appropriate for group interviews.
- The cultural consensus model is applicable only if there is a single set of answers to the questions. This means that there must be a high level of consistency (agreement) in responses among informants, which some refer to as common truth.[5] An aggregation of responses is not a valid estimate unless there is reasonable consistency in the underlying data. An initial step in applying consensus theory is to check whether there is a high degree of agreement among the informants (e.g., to verify that there is only one response pattern present).
Models
editCultural consensus theory encompasses formal and informal models. Practically speaking, these models are often used to estimate cultural beliefs, including the degree to which individuals report such beliefs.[5] The formal cultural consensus model models the decision-making process for answering questions.[6][7] This version is limited to categorical-type responses: multiple-choice type questions (including those with dichotomous true/false or yes/no responses) and responses to open-ended questions (with a single word or short phrase response for each question). This version of the model has a series of additional assumptions that must be met, i.e., no response bias.[6][8] The formal model has direct parallels in signal detection theory and latent class analysis. An informal version of the model is available as a set of analytic procedures and obtains similar information with fewer assumptions.[9] The informal model parallels a factor analysis on people (without rotation) and thus has similarities to Q factor analysis (as in Q Methodology). The informal version of the model can accommodate interval estimates and ranked response data. Both approaches provide estimates of the culturally correct answers and estimates of individual differences in the accuracy of reported information.
One specific method of the formal version used in the analysis of data is the mathematical model, which is a set of logical axioms as well as derived propositions and assumptions that explain how empirical variables fit in the model's parameters.[5] The informal model, on the other hand, uses reliability analysis.[10]
Competence
editCultural competence is estimated from the similarity in responses between pairs of subjects since the agreement between a pair of respondents is a function of their individual competencies. In the formal model, the similarity is the probability that matched responses occur (match method.[6] or the probability of particular response combinations occur (covariance method[7]). Simple match or covariance measures are then corrected for guessing and the proportion of positive responses, respectively. In the informal model, similarity is calculated with a Pearson correlation coefficient.[9]
A matrix of agreement coefficients between all pairs of subjects is then factored with a minimum residual factoring method (principal axis factoring without rotation) to solve for the unknown competence values on the main diagonal. (For the informal model, the maximum likelihood factor analysis algorithm is preferred, but principal axis factoring can be used as well.) To determine whether the solution meets cultural consensus criteria, that only a single factor is present, a goodness of fit rule is used. If the ratio of the first to second eigenvalues is large with subsequently small values and all first factor loadings are positive, then it is assumed that the data contain only a single factor or a single response pattern.
Individual
editIndividual competence values are used to weight the responses and estimate the culturally correct answers. In the formal model, a confidence level (Bayesian adjusted probabilities) is obtained for each answer from the pattern of responses and the individual competence scores. In the informal model, responses are also weighted, using a linear model. When factoring a correlation matrix, the estimated answers appear as the first set of factor scores. Also, note that factor scores are usually provided as standardized variables (mean of zero), but may be transformed back to your original data collection units.
Analysis
editWhen used as a method for analysis, the cultural consensus theory allows the following: the determination whether the observed variability in knowledge is cultural; the measurement of cultural competence that each individual possesses; and, the determination of culturally correct knowledge.[4]
Cultural Consensus analyses may be performed with software applications. The formal consensus model is currently only available in the software packages ANTHROPAC or UCINET. Analysis procedures for the informal model are available in most statistical packages. The informal model can be run within a factor analysis procedure, requesting the minimum-residual (principal axis factoring) algorithm method that solves for the missing diagonal without rotation. When factor analysis is used for consensus applications, ~~the data must be transposed, so that questions are the unit of analysis (the rows in a data matrix) and people are the variables~~ (the columns in the data matrix).
An advantage of cultural consensus is the availability of necessary sample size information and that necessary sample sizes do not need to be very large. Sample size determination in a consensus analysis is similar to other types of analyses; namely, that when variability is low, power is high and small samples will suffice. Here, variability is the agreement (competence) among subjects. For the formal model, sample size can be estimated from the level of agreement (e.g., assuming a low average competence level of .50), the proportion of items to be correctly classified (assuming a high level, .95), and high confidence (.999) a minimum sample size of 29 (per subgroup) is necessary.[1,5] For higher levels of competence and lower levels of accuracy and confidence, smaller samples sizes are necessary. Similarly, sample size can be estimated with reliability theory and the Spearman–Brown prophecy formula (applied to people instead of items). For a relatively low level of agreement (an average correlation of .25 between people, comparable to an average competence of .50) and a high degree of desired validity (.95 correlation between the estimated answers and the true answers), a study would require a minimum sample size of 30 subjects.[11]
In summary, cultural consensus theory offers a framework for estimating cultural beliefs. A formal model is based on the decision-making process model of how questions are answered (with parameters for competence, response bias, and guessing). The model proceeds from axioms and uses mathematical proofs to arrive at estimates of competence and answers to a series of questions. The informal model is a set of statistical procedures that provides similar information. Given a series of related questions, the agreement between people's reported answers is used to estimate their cultural competence. Cultural competence is how much an individual knows or shares group beliefs. Since the extraction of individual competencies depends upon having a single factor solution, the ratio of the first and second eigenvalues (> 3:1) serves as a goodness-of-fit indicator that a single factor is present in the pattern of responses. Culturally correct answers are estimated by weighting and combining individuals’ responses.
References
edit- ^ William H. Batchelder (April 2009). "Cultural Consensus Theory: Aggregating Expert Judgments about Ties in a Social Network". In Michael J. Young; John Salerno; Huan Liu (eds.). Social Computing and Behavioral Modeling. Boston, MA: Springer-Verlag. pp. 1–9. doi:10.1007/978-1-4419-0056-2_5. ISBN 978-1-4419-0056-2.
- ^ Carroll, John (2006). Engaging Society: The Sociologist in a War Zone. Manila: Ateneo University Press. p. 3. ISBN 9789715505055.
- ^ Anders, Royce; Batchelder, William H. (March 2015). "Cultural Consensus Theory for the Ordinal Data Case". Psychometrika. 80 (1): 151–181. doi:10.1007/s11336-013-9382-9. ISSN 0033-3123. PMID 24318769. S2CID 255164220.
- ^ a b Faucher, Colette (2017). Advances in Culturally-Aware Intelligent Systems and in Cross-Cultural Psychological Studies. Cham, Switzerland: Springer. p. 46. ISBN 9783319670225.
- ^ a b c Matthias, Ruth (2015). Handbook of Research methods and Applications in Environmental Studies. Cheltenham, UK: Edward Elgar Publishing. p. 83. ISBN 9781783474639.
- ^ a b c Romney AK, Weller SC., Batchelder WH. Culture as consensus: A theory of culture and informant accuracy. Am Anthro. 1986;88(2):313–38.
- ^ a b Batchelder WH, Romney AK. Test theory without an answer key. Psychometrika. 1988;53(1):71–92.
- ^ Weller, SC and Mann NC. Assessing rater performance without a “gold standard” using consensus theory. Medical Decision Making 1997; 17:71–79.
- ^ a b Romney AK, Batchelder WH, Weller SC. Recent applications of cultural consensus theory. Am Behavioral Scientist. 1987; 31:163–177.
- ^ Vaccaro, Ismael; Smith, Eric Alden; Aswani, Shankar (2010). Environmental Social Sciences: Methods and Research Design. Cambridge: Cambridge University Press. p. 114. ISBN 9780521110846.
- ^ Weller SC: Cultural consensus theory: Applications and frequently asked questions. Field Methods 2007; 19(4): 339–368.