Signed in as:
filler@godaddy.com
Transforming intelligence
Big Data and National Security: Impacts for Intelligence
My PhD explored the impact of big data on intelligence production and national security decision-making in Australia. The research was conducted at Deakin University from 2017-2021.
This research examined the impact of big data on intelligence production and national security from the perspective of Australian national security leaders and practitioners.Intelligence production is one of the primary mechanisms for framing information and analysis to inform national security decision-making. However, there is little understanding of the way big data impacts on national security and intelligence analysis specifically. This empirical research examined the impact of big data on national security and intelligence production and is the first conducted in all of Australia’s ten National Intelligence Community agencies and their oversight body. Data collection included semi-structured interviews with 47 senior and operational decision-makers, technologists and former officials.
The thesis argued that big data is transforming intelligence knowledge, activities and organisations and the national security threats on which they are focused. This thesis argued that the impacts of big data on the knowledge, activities and organisation of intelligence agencies is challenging some foundational intelligence principles, including the distinction between foreign and domestic intelligence collection. Furthermore, it argued that big data has created emerging threats to national security. For example it enables invasive targeting and surveillance, drives information warfare as well as social and political interference, and challenges the existing models of harm assessment used in national security. The thesis maps broad areas of change for intelligence agencies in the national security context and what they mean for intelligence communities as a whole. It explores how intelligence agencies look out to the rest of society, considering specific impacts relating to privacy, ethics and trust. This research shows that national security leaders and practitioners perceive that big data is profoundly transforming current and future intelligence production and decision-making.
The PhD was supported by a National Security Big Data Scholarship from D2D CRC and a University Postgraduate Research Scholarship from Deakin University. Additional support was also provided by D2D CRC in the form of an Applied Research and Collaboration Award (2017) and an Applied Research Grant (2019). This research was supervised by Dr Chad Whelan and Dr Diarmaid Harkin.
Thesis in detail
Thesis Outline (from introduction)
This thesis argues that big data is transforming intelligence production specifically as well as changing the national security environment broadly, including what is considered a part of national security and the relationships intelligence agencies have with the Australian people. This thesis highlights some of the current and future transformational changes associated with big data in society writ large that have implications for the intelligence community. The introduction introduces the research questions and sets out the aims and rationale for this thesis. It also explains the concepts central to the thesis, big data, national security and intelligence–and briefly outlines how they are impacted by this research.
Chapter 1 reviews the multidisciplinary literature relating to the impact of big data technology and analytics in the field of national security. Firstly, it explores the rise of big data. Secondly, it considers the key debates and concerns surrounding the implementation and application of big data in society, with a particular focus on transparency, identity and power. Finally, the literature review considers the promise of big data for national security including intelligence and predictive applications of big data technologies. The literature review shows that there have been very few empirical studies exploring big data and intelligence and highlights the urgent need for this research to develop an understanding of how big data impacts on intelligence activies and agencies and national security.
Chapter 2 presents the research design of this thesis. Firstly, it outlines the research approach and semi-structured interviews. Secondly, it offers necessary context, describing the environment of the NIC. Thirdly, it covers the research sample and why and how interviewees were selected before, fourthly, explaining the data collection and analysis methods. This research involved semi-structured interviews with 47 participants from within NIC agencies, as well as a small number of subject matter experts. Empirical data collection inside intelligence agencies is notoriously difficult and this is the first known research inside all the Australian intelligence community agencies. These participants hold previously unexplored views and perform roles of growing significance to Australian national security. Finally, Chapter 2 considers ethical considerations in human research relevant to this study.
Chapter 3 examines the impact of of big data and what this means broadly for intelligence in democracies and the NIC. The existing literature reveals the breadth of big data–related technologies and how they are impacting society, and this chapter illustrates how they are impacting national security. This chapter argues that the advent of big data has created three features relevant to the intelligence community: data abundance, digital connectivity and ubiquitous technology. These three features of big data emerged in this research the most current impacts of big data for the intelligence community as well as holding future transformative potential. This thesis argues that they form a big data landscape. In particular, in this study, almost all participants stated that the volume of personally identifying information recorded and available in society is a transformative change for national security, increasing the capacity for remote surveillance from intelligence agencies, but also from companies that collect and trade data. Building on the literature that shows that big data has centralised economic and information power in the hands of a small number of companies, this chapter reveals the potential impacts of this unprecedented power on national security.
Chapter 3 then shows how big data impacts some of the longstanding and foundational principles of intelligence, including broadening the conception of what comprises national security as well as challenging existing mechanisms to identify Australians and their data. The participants indicated that the big data landscape is changing understandings of geographical jurisdiction for intelligence activities and that understanding nationality is more complex as a result of big data. The chapter reveals that big data complicates intelligence analysis relating to technologies as well as challenges the national security approach to innovation and adopting technologies. It demonstrates how big data connectivity tests the fundamental principle of intelligence storage and compartmentalisation, which agencies rely on to reduce security risks. Finally, it shows that the big data landscape challenges existing methods of assessing harms and threats in national security because there are infinitely more connections to assess and many of these reside outside governments’ awareness. According to participants in this research, the focus on physical harm is now disproportionate to the social threats presented by big data and the national security community alone is not equipped to make harms assessments.
Chapter 4 examines the impact of big data on intelligence production. This chapter argues that big data is transforming intelligence in three key areas, which are foundational components of intelligence communities and agency functions: intelligence as knowledge, as an activity and as an organisation. This chapter shows how big data is changing the kinds of knowledge needed for intelligence, largely due to data abundance and the kind of information and data not previously available or recorded. This chapter reveals how big data increases uncertainty in individual pieces of data or information and can increase uncertainty in overall intelligence assessments, which can impact on the ability of intelligence agencies to provide forewarning to governments. This chapter also argues that big data is changing the way that information is digested by decision-makers, due to a more competitive and crowded information environment for policy makers. This has significant ramifications for national security as decision-making around national security often involves restriction of liberties, surveillance, or use of lethal and non-lethal force. This chapter further shows that big data is impacting intelligence activities in almost every aspect of the renowned intelligence cycle, and suggests that there are immediate impacts in collection, analysis and communication of intelligence, with future transformative change in other aspects of the intelligence cycle. This chapter also contends that the impact of big data on intelligence as an organisation cuts across critical aspects of intelligence, from data sharing across the NIC, to the impact of data on the role of analysts, changing how they may be employed in the future.
Chapter 5 analyses the impacts of big data on data privacy. Big data has and continues to radically transform privacy in society broadly. This chapter builds on the extensive literature evidencing that big data is changing privacy norms globally and the perception that in Australia there is a need to rethink the privacy principles underpinning privacy laws. It looks at the way in which big data has changed social conceptions of privacy and challenges the Australian legislative framework for privacy and why this is important for intelligence agencies. This chapter argues the impact of big data on privacy – and privacy regulation – in society at large has potential future implications for the intelligence community. However, this research suggests that currently the direct impacts of big data on privacy in intelligence agencies are limited and predominately dependent on an agency’s role and legislative mandate, affecting some agencies more than others. Participants highlighted that the impact of big data on privacy is characterised by one significant distinction among the AIC collection agencies, that is, whether the agency has a foreign or domestic mandate. Big data is changing how some agencies collect, store and analyse data, particularly if the agency is subject to a legislative requirement to determine whether the data relates to an individual who is Australian.
Chapter 5 argues that the well-documented changing norms and lagging regulation of privacy in society because of big data are creating new social harms which are already national security threats or have the potential to rapidly become threats, to which the intelligence community will need to respond. This chapter demonstrates that the big data landscape enables social and political harm in new ways, building on the literature that shows big data has transformed the capacity for intelligence agencies as well as some private companies to access an abundance of data, the capacity to identify data, link it with other data and use it quickly. This chapter argues that big data has created an element of privacy intrusion that is remote from the individual and less visible. Participants indicated that the vast number of data collectors, sellers and users has led to a complex and confusing privacy landscape for Australians.
Chapter 5 posits that the rapid growth of new data and analytics occurring in the private sector combined with a lack of regulation of private data collection and use is changing the nature of privacy and surveillance. This chapter suggests that who creates and can access data is relevant in this context because it means that some of the most intrusive aspects of intelligence collection (digital surveillance, profiling, influence, monitoring, tracking and targeting) are now available much more broadly to a wider range of actors, such as big tech, app developers, marketing companies and anyone who can purchase or steal data. Big data systems enable tracking, monitoring and analysis and make surveillance capabilities accessible to many more actors. According to participants in this research, this sets the preconditions for adversaries to purchase or acquire data and access to undermine national interests and hinder states’ ability to achieve national security objectives, as well as to harm individuals. This chapter argues that big data can be used to harm individuals and society in new ways, such as by providing mechanisms for information warfare as well as influencing and interfering in political and civic discourse. Participants suggested that the big data landscape is already transforming the national security threats Australia faces, highlighting information warfare and social and political interference in democracy.
Chapter 6 examines how big data impacts on ethics in intelligence. Drawing on the existing literature establishing that there are ethical boundaries of intelligence, this chapter argues that big data is changing where those boundaries lie. According to participants, there are aspects of intelligence where big data and automation will not ever be useful and other situations where more testing and refinement is needed before such systems are introduced. Practitioners perceived that big data increases the need for decision-making processes at all levels to incorporate intelligence ethics. This chapter highlights two ethical dilemmas of big data in intelligence that have not previously been studied. Firstly, some of the ways that decisions around ethics are being automated and applied at scale in social contexts, by private companies, would represent considerable ethical dilemmas if applied to intelligence activities. Secondly, ethics in intelligence includes considering bias. This chapter indicates that intelligence practitioners should be aware of the difference between cognitive bias and data bias as well as the intelligence challenges of incomplete data sets and the bias of intelligence collection itself.
Chapter 7 explores the impact of big data on trust, examining the way in which interviewees perceive their relationships with the public and how big data has and will impact on that relationship. Emerging strongly from the data was a sense that trust is significant in the role of national security agencies in Australia. Participants indicated that they saw big data and the information ecosystem it enables as changing the relationships between intelligence agencies and the public. Furthermore, this chapter argues that big data impacts trust in the entire system of government and public service agencies as it is reliant on trust in the way data is collected and used across all government agencies, not just the national security sector. This chapter proposes that big data is changing the public’s perceptions of the intelligence community around trust, transparency and the legitimacy of intelligence agency operations. It unpacks how participants understand trust and the key concepts of trust, legitimacy and the social contract, which each emerged from the interview data. Secondly, this chapter shows that participants perceive that how trust is built and developed is impacted by big data, with participants suggesting intelligence agencies need to align big data use with agency values and purpose, transparency and public engagement.
The thesis argues that big data is transforming what intelligence is, how it is practised, and the relationships intelligence organisations have with society and with each other. It shows that big data has impacts on many aspects of national security, including our conception of what it constitutes. The impact of big data is especially significant for the knowledge, activities and organisation of intelligence agencies. This thesis highlights specific impacts for intelligence agencies and the production of intelligence, and then examines how intelligence agencies interact with each other and look out to the rest of society. This thesis argues that big data is impacting the relationship between intelligence agencies and citizens, specifically in the areas of privacy, ethics and trust.
