Earlier this week Dr Simon Crouch, of the University of Melbourne, published a one-page interim report on the alleged findings of his study (Australian Study of Child Health in Same-Sex Families, ACHESS) of 500 children, ages 17 and under, of 315 gay, lesbian, and bisexual parents.
The news media in Australia picked up the story and gave it enough credence to imply that immediate policy changes with respect to same-sex marriage should be one consequence of his findings. I do not wish to downplay the potential value of this research; I am very excited to see this sort of research being conducted in Australia.
Nevertheless, there is an important difference in the scholarly credibility of a one-page interim report that has not been peer reviewed or published in a peer-reviewed journal and a peer-reviewed publication. While both can report valid truth, only the latter has received the potential benefits provided by adequate scholarly, and hopefully unbiased and independent, peer review.
There remain a number of questions about the Australian Study of Child Health in Same-Sex Families that might be resolved through effective peer review.
First, the American Psychological Association has long stated in its style manuals that significance levels are not sufficient indicators of the importance of research findings. Effect sizes, rather, should also be reported, along with means and standard deviations, from which effect sizes can be calculated. The interim report of the ACHESS mentions statistically significant findings but nothing about effect sizes. Therefore, it is in violation of APA policy on the best way to report research results.
It is not clear on how “same-sex” was defined, especially those who were allegedly queer or bisexual. In their BMC Health article, it appears that same-sex attraction was to be the basis of defining sexual orientation. However, it is not clear if a term like “queer” pertains to sexual attraction or to sexual identity. It appears that some single-parent families were studied; comparing one and two-parent families would be useful, regardless of parental sexual orientation. Since sexual attraction and sexual behavior can differ, without more information, we do not know how many of the “same-sex attracted” (especially bisexual) parents might have been engaging in heterosexual sexual behavior or partnering.
In Dr Crouch’s article in BMC Health, it is stated that they hoped to study up to 750 children from 400 families, but it appears they only obtained responses from 500 children from 315 families, seeming to suggest that their project objectives were not achieved. Does that suggest some impact of refusal rates or non-response? Some impact of selection effects? We cannot be sure without more information.
Some families may have refused to participate in this study because of the request that they participate in follow-up interviews, which would require that their information be kept confidential in spite of the need to maintain lists of names and addresses for subsequent contacts. Another possibility is that the study was not able to gather as much data as expected because the initial estimates of the number of children living with same-sex parents in Australia (eg, 6,120 children under the age of 25) may have been overestimated.
Another issue is the age of the children. If some children were under the age of 10, as implied in the interim report, it is unlikely that they reported on their own health outcomes, but rather more likely that parental reports were used. There is an inherent self-presentation bias possible in any parent’s report on their child since virtually all parents love their children and want to think the best of them.
If you suspect that a study in which you are participating will help you politically if you say the best possible things about your own children, I would suspect any such bias would be even greater. If the comparison group did not have the same awareness of the purpose of the study, they might have less susceptibility to such social desirability bias in their reports. If social desirability response bias was not measured or controlled in both groups, that would represent an important design flaw in the research program, a serious barrier to accepting the full validity of the research.
Furthermore, there is no indication if there were socioeconomic differences between the different groups of parents or children. If you wish to determine the influence of parental sexual orientation per se on child outcomes, you need to control for family differences in parental education, household income, and per capita household income. It is also possible that the parents who volunteer for research on same-sex families may be self-selecting themselves on the basis of superior mental health, better interpersonal parenting skills, or greater success at dealing with social stigma compared to those traits in a random sample from the general public or to other same-sex parents. It is important to not confound parental sexual orientation with other parental characteristics.
It is also important to measure the “right” outcomes of parenting in the “right” way – that is with scientific reliability and validity. In their BMC Health article, the authors acknowledge that some of their measures have featured subpar reliabilities (eg, below 0.70) and others have yet to be validated in Australian samples. Using measures with low reliability or which are not valid for Australians could attenuate the ability to find meaningful differences across different groups of families.
The possible omission of some important child health-related outcomes (eg, ability to delay gratification, the family history of each child in terms of traumatic caregiver transitions, engagement in early sexual activity, use of illegal substances, experience of child sexual abuse, academic performance) might be another measurement limitation of the study. If you don’t measure something, you will not be able to compare child outcomes on that measure. Absence of measurement does not imply absence of differences or effects, however.
Between the surveys given to the parent and the child, there were nearly 250 questions. Random chance alone would allow for a dozen or more significant findings (p < .05) if each of those questions were compared across same-sex and heterosexual families. Without knowing from the interim report how many statistical tests were performed to obtain the results reported as significant, we cannot be sure of the correct interpretation of those results.
While the ACHESS study is a welcome addition to the literature, a one-page interim report can only be considered an academic “teaser” – we will be much better informed of the validity and value of the study’s results after those results have been improved and evaluated through competent, unbiased, scholarly peer review and published in an accessible scholarly outlet. Until then, I would recommend caution about changing family policies in Australia or elsewhere on the basis of the limited information we have at the moment for this research.
Dr Walter Schumm is Professor of Family Studies in the School of Family Studies and Human Services at Kansas State University.
Crouch, S.R. (2013) The Australian Study of Child Health in Same-Sex Families (ACHESS). Interim Report. Melbourne: The Jack Brockhoff Child Health & Wellbeing Program, University of Melbourne.
Crouch, S.R., Waters, E., McNair, R., Power, J., & Davis, E. (2012) ACHESS – The Australian study of child health in same-sex families: background research, design and methodology. BMC Public Health, 12, 646, 1-9.