ChatGPT and the Future of Qualitative Public Policy Research

    • By,
      Dr. Vishnu S Pillai – Assistant Professor, Kautilya

Thanks to the advancement of deep learning, the GPT-4, or Generative Pre-trained Transformer 4, can generate human-like outputs, respond to the environment, and be perceived as something that can revolutionise the entire qualitative research environment. The research community has always used various tools to increase the researchers’ productivity and, in many cases, their accuracy, particularly in exhaustively searching and analysing to identify research gaps.

Generative artificial intelligence (AI) models can generate new information by discerning patterns within pre-existing data that align with human language. Over time, there has been a notable progression in this area, culminating in the recent achievement of GPT-4, which has effectively fulfilled several academic and professional standards. If fine-tuned adequately, this technology’s ability to provide outputs that resemble human-like responses will have extensive uses for those engaged in Qualitative Public Policy Research.

This includes but is not limited to the systems that can aid in reviewing various reports facilitating content analysis and providing insights from the existing research. This can also be an AI model that aids the researcher in prompting questions during an interview to explore the trends and facilitate the required thought process in the respondents. This can even be an AI system by their “multi-sensory abilities”, simultaneously analysing the visual and textual information in the research setting and developing multiple scenarios as outputs that the researchers can explore further.

In the Public Policy research domain, the key journals usually accept manuscripts with theoretical contributions and practical public policy insights. Such research necessitates higher-order reasoning and a rigorous research procedure that addresses the most significant validity threats. Observations and identified patterns/categories, for instance, should be exhaustive and, to some extent, mutually exclusive. Generative AI models can supplement existing Qualitative Data Analysis methods and increase researchers’ accuracy and efficiency, hence meeting such standards, resulting in fewer submission rejections.

However, this revolutionary technology divided the research community into two camps: those who advocated for its use in critical research areas, such as Public Policy, and those who opposed such applications due to their concern about the technology’s inherent risks.

Considering my previous work with Prof. Kira Matus (Associate Head and Professor, Division of Public Policy, The Hong Kong University of Science and Technology), a risk-based approach to AI will reveal many of the concerns we had (in the construction industry) once such technologies are available for use in Public Policy Research. The risk classifications of AI, liability, data security, unemployment, etc., will manifest once the Generative AI models find widespread applications in qualitative research.

Conducting a thorough study on the possible influence of AI models inside the research community is necessary to provide a comprehensive overview of these risks. Like any AI technology, subsequent iterations of AI models exhibit enhanced levels of accuracy compared to their predecessors. Therefore, academicians in the field of public policy must avoid complacency by examining the flaws of current GPT models and avoid passivity by believing that technology will not surpass human capability. They must research systematically by involving as many stakeholders as possible and learn their perspectives on the potential hazards of such models. Without such research, engaging in additional debates on this topic will likely result in unreasonable worries and actions that impede the effective integration of this technology for further qualitative research in this field.

*The Kautilya School of Public Policy (KSPP) takes no institutional positions. The views and opinions expressed in this article are solely those of the author(s) and do not reflect the views or positions of KSPP.