How to Buy AI
The DoD struggles to field AI-enabled capabilities, but the new Artificial Intelligence Acquisition Guidebook published by the Air Force-MIT Artificial Intelligence Accelerator identifies best practices for future AI acquisitions.
BY AIR FORCE MAJOR ANDREW BOWNE AND AIR FORCE CAPTAIN RYAN HOLTE
"America is not prepared to defend or compete in the AI era.”1 The National Security Commission on Artificial Intelligence (NSCAI) issued this blunt assessment to the Department of Defense (DoD) and Intelligence Community (IC). The commission recommended that the DoD and IC become AI-ready by 2025.2
The NSCAI report tagged three areas as critical for widespread integration of AI: building a common digital infrastructure, developing a digitally literate workforce, and instituting more agile acquisition.3
The DoD struggles to field AI-enabled capabilities, the commission found, because AI acquisition, “follows largely rigid, sequential processes, inhibiting early and continuous experimentation and testing critical for AI.”4 While some government organizations have addressed these obstacles,5 few are prepared to buy, build, and use AI.6
Spurred by the commission’s report, the Air Force-MIT Artificial Intelligence Accelerator (AIA) began identifying best practices for acquiring AI-enabled capabilities and technologies. In February 2022, the AIA self-published the “Artificial Intelligence Acquisition Guidebook.”7
AIA Director Air Force Col. Tucker Hamilton explained that the guide, “is a starting point, an opportunity for our acquisition professionals to garner a valuable viewpoint as they deal with a technology that will impact every program office.”8
This article introduces the guidebook, the reasons it was written, the process the AIA employed to create it, and the intended takeaways for acquisition professionals, including program managers, contracting officers, lawyers, engineers, financial managers, data analysts, and industry.
Filling the Void
The AIA stood up in May 2019, when the Air Force entered a cooperative agreement with the Massachusetts Institute of Technology (MIT).9 The Air Force’s only AI-dedicated organization, AIA is a multidisciplinary team of airmen, Space Force Guardians, and faculty, researchers, and students from MIT and Lincoln Laboratory. The accelerator conducts research to enable rapid prototyping and scaling. It also enables ethical application of AI algorithms and systems, tackling some of the most difficult challenges facing the nation.10
Although the AIA is focused on fundamental research, it supports the Air Force in developing frameworks, policy, and education that advance operational objectives. The AIA is in a unique position at the nexus of academia, government, and industry. Taking advantage of this ecosystem, the experience and expertise of its partners, as well as its own Phantom Fellows,11 the accelerator sought to fill the void in guidance and documented practice considerations for acquiring AI-enabled systems.
The Phantoms who contributed to the guidebook were program managers and a contracting officer who contributed personal experience, as well as research, interviews, and collaboration with experts in acquisition, contracting, and law, in government, academia, and industry.
The purpose of the guide is, “to provide a basic understanding of the AI acquisition lifecycle with respect to data, finance, contracting, and legal considerations.”12 The intended audience is entry-level to senior program managers. Successful acquisition and integration of AI-enabled technologies requires cross-functional collaboration. The book also is directed to contracting officers, lawyers, financial managers, engineers, and data scientists.
Understanding the AI Acquisition Guidebook
The guide is premised on three assumptions. First, it assumes readers have a basic technical understanding of AI.13 If not, however, the book provides resources to bring even the most novice data scientist up to speed.14
The other two assumptions are not explicitly stated. One is that AI is a unique yet ubiquitous technology, and acquiring it requires a different perspective. The other is that the technology is developing so rapidly DoD is going to have to adapt to match its requirements with state-of-the-art capabilities. This means the AI Acquisition Guidebook will be perpetually in draft form.
The guide provides an overview of authorities from Congress, the president, and the DoD.15 It is not official guidance, though the authorities section provides a comprehensive literature review of the recommendations from the NSCAI, executive orders, strategies, policies, laws, and National Defense Authorization Act sections. This section is a good place to get an initial understanding of the current dialogue on AI in the federal government.
The next section explains management considerations for AI capabilities, whether the AI is a component of a larger program or when the AI capability itself is a program.16 Like other DoD acquisitions, successful AI programs begin with a well-crafted requirement. However, the requirements process is where mistakes are common and can derail an AI acquisition.
Acquisition officials often have led AI acquisitions with inadequate requirements, for example, “incorporate AI,” and with rigid solicitation methods such as requests for proposal. Strategies like these allow little flexibility for inherently evolutionary AI projects.
Instead, AI projects require robust problem statements and versatile solicitation methods, such as commercial solutions openings, to ensure that training data, algorithms, and model parameters are developed through multiple iterations, user studies, and open communication.17
The guide provides a framework for developing contracting strategies for AI projects that focus on the importance of data.18 Because AI is a ubiquitous technology enabler, it can be contracted for using any authority available to the DoD.
There are numerous options among Federal Acquisition Regulation (FAR) and non-FAR-based contracts. The non-FAR options include other transaction agreements, procurement for experimental purposes, and prize challenges.
That said, choosing the best contract type takes a special understanding of contracting authorities and their capabilities related to licensing data rights, model development, and the intended use case. Contracting officers need technical expert and end-user input to determine the appropriate pathway to acquire AI-enabled capability. This is because the lines between research and development, procurement, and even sustainment often are blurred by the unique nature of machine learning.19
Balancing access to and ability to use the technology with the considerations of the industry partner is critical. The guide provides substantial resources for choosing the appropriate contracting strategy based on the specific stage in the AI lifecycle and type of technology.
The AIA has found that non-FAR contracts generally provide the most freedom for the government to assert data rights and set model quality assurance measures. Generally, in FAR contracts, program offices assert typical data rights such as limited, Government Purpose Rights (GPR), and unlimited rights.
These categories can have many benefits for the government. Unfortunately, due to a lack of understanding of AI, the government often forgets to assert or incorrectly asserts rights to technology derived from government-owned resources.
For the many machine learning models that are created using a mix of proprietary algorithms trained by government-owned data, this oversight could be a costly error.
If government-owned data and sensors are used, the government should assert GPR, unlimited rights, or the equivalent for the model and/or its parameters. Asserting these rights allows control of the model for national security purposes and quality assurance.
Additionally, asserting the appropriate rights ensures data insight for implementation of AI ethics throughout the lifecycle.20 Rights to the data and model allow the government to provide continuous and iterative feedback for model development.
The guide outlines novel, practical contracting, data rights, and quality assurance measures that give insight for the acquisition community to become AI-ready.21
Education, Iteration, and Data
Much as there were no experts at navigating the globe in the days of Magellan, the acquisition map for AI capabilities could figuratively state “Here be dragons.” Challenges abound with no easy fixes, so the AIA advocates for a community of practice to serve as AI-era pioneers.
As the guide outlines, preventing the NSCAI’s baleful assessment of American AI capability from becoming a prophecy requires acquisition professionals to understand AI, develop flexible and iterative acquisition strategies, and treat data as a strategic asset. There will be successes and failures; both bring learning, but it will go faster if lessons are shared. CM
“The views expressed in this article are those of the authors and do not reflect the official guidance or position of the United States Government, the Department of Defense or of the United States Air Force.”
Statement from DoD: The appearance of external hyperlinks does not constitute endorsement by the United States Department of Defense (DoD) of the linked websites, or the information, products, or services contained therein. The DoD does not exercise any editorial, security, or other control over the information you may find at these locations.
Major Andrew Bowne is the Chief Legal Counsel at the Department of the Air Force/MIT Artificial Intelligence Accelerator. He also is a Ph.D. candidate at the University of Adelaide in Australia.
Captain Ryan Holte is a Project Manager in the Space Systems Command Chief Information Office. He was a Phantom Fellow at the Air Force/MIT Artificial Intelligence Accelerator.
ENDNOTES
1 “The Final Report,” National Security Commission on Artificial Intelligence (NSCAI), (Arlington, VA: 2021), 1. https://www.nscai.gov/2021-final-report/.
2 Ibid, 2.
3 Ibid, 9.
4 Ibid, 62.
5 The NSCAI identifies the United States Digital Service, Kessel Run, the Army Artificial Intelligence Task Force, the USAF-MIT AI Accelerator, components of the IC, and the national labs as pockets of excellence, though laments that there are too few. Ibid, 130.
6 Ibid, 121.
7 “Artificial Intelligence Acquisition Guidebook,” Department of the Air Force-Massachusetts Institute of Technology Artificial Intelligence Accelerator (February 14, 2022) [“AI Acquisition Guidebook”]. https://aia.mit.edu/wp-content/uploads/2022/02/AI-Acquisition-Guidebook_CAO-14-Feb-2022.pdf. Both authors of this article were among the group of authors that wrote the AI Acquisition Guidebook.
8 Ibid, 3.
9 “About Us,” USAF-MIT AI Accelerator, https://aia.mit.edu/about-us/.
10 Ibid.
11 The Phantom Fellowship is a specialized AIA program that brings in a hand-selected group of AI-driven acquisition officers and enlisted members for four months to learn how to manage AI programs and develop policies, processes, and lessons learned to expand that knowledge to the greater community. AI Acquisition Guidebook, 3.
12 Ibid, 4.
13 Ibid. AI can enable capabilities, provide unprecedented insight, and increase decision speed, but misunderstanding what the technology can do and how to use it can have drastic effects on its perceived performance. The science of AI is nuanced, and each project’s success is conditioned on the quality of the training data, the choice of algorithm, and nearly every decision from the initial problem statement to operational deployment. AI fundamentally differs from the technologies that the DoD has typically acquired. “DOD recognizes that developing and using AI differs from traditional software. Traditional software is programmed to perform tasks based on static instructions, whereas AI is programmed to learn to improve at its given tasks.” Government Accountability Office, “Artificial Intelligence: Status of Developing and Acquiring Capabilities for Weapon Systems” (February 2022). https://www.gao.gov/assets/gao-22-104765.pdf. Yet, in many cases, program offices have attempted to acquire AI-enabled systems with traditional acquisition processes that are designed for hardware-intensive systems that are linear and time-consuming. Ibid, 23 (explaining “DOD’s traditional acquisition processes were designed for hardware-intensive systems,” noting the NSCAI findings and that industry group officials told GAO that these traditional processes are not well-suited to AI). To avoid these issues, the AI Guidebook emphasizes the criticality of at least a basic AI education prior to developing any acquisition strategy, so that the program manager can develop that strategy with the entire lifecycle of the AI-enabled system and associated software in mind.
14 Ibid, § 5.1 (Education Opportunities).
15 Ibid, Part II.
16 Ibid, Part III.
17 Ibid, § 3.1. See also William E. Novak, “Artificial Intelligence (AI) and Machine Learning (ML) Acquisition and Policy Implications,” 18-22 (February 2021). https://resources.sei.cmu.edu/asset_files/WhitePaper/2021_019_001_652056.pdf.
18 AI Acquisition Guidebook, § 3.2.
19 Machine learning is never truly finished. The algorithm learns with new data to create new statistical models that predict outcomes, and even deployed models require continuous testing and evaluation. This attribute makes contracting challenging and pushes the limits of existing appropriations. The guidebook provides additional guidance on how to determine whether research and development or operations and maintenance funds are appropriate for a given effort. Ibid, Part IV.
20 License rights to the input data (training data) and output data (model) are necessary for the government to understand and trust the performance of the model. This insight permits the government to provide continuous and iterative feedback for model development. Without specially negotiated rights, the government will likely not have license to use, access, or modify some component of the AI pipeline.
21 For a more in depth discussion data licensing for AI, see Andrew Bowne and Benjamin McMartin, “Implementing Responsible AI: Proposed Framework for Data Licensing” George Mason University, White Paper Series No. 10 (April 29, 2022. https://www.gmu.edu/news/2022-04/no-10-implementing-responsible-ai-proposed-framework-data-licensing.