Stop Buying Software That Doesn’t Deliver
Use CSOs, demonstrations, and modular contracting to prolong competition, evaluate unique products that cannot be compared, and drive down the risk of innovative products.
BY KEITH GIBSON
It’s no longer rare or surprising to see innovative acquisition and contracting approaches touted in the media, at conferences, and in the pages of this magazine. What’s harder to find are the stories of how contracting organizations have chosen among these approaches, picked one for a specific purpose, and then applied it to solve a program or governmentwide problem.
That’s what Keith Gibson has done for commercial solutions openings (CSOs). His true story about finding a way to evaluate incomparable innovations and avoid spending money on software that doesn’t work offers rare insight into how CSOs can be used.
It’s also a saga filled with creative and critical thinking. He writes about picking an approach by determining how its attributes apply to a challenge and exploring a way to let buyers test software before committing to buy it. He also describes figuring out a form of consideration other than funds for vendors in exchange for free trials of their products.
Gibson explains and demonstrates the flexibility provided by CSOs. He clarifies the way his organization uses the quick and easy CSO solicitation mechanism to lead to contracts for solutions that have potential across the Department of Defense (DoD) and among civilian agencies.
Non-DoD agencies have more reason now than ever to learn about CSOs. Though a CSO pilot for the General Services Administration and the Department of Homeland Security ended at the end of fiscal 2022, a bill introduced by Senator Gary Peters (D-MI), the 2022 Advancing Government Innovation With Leading-Edge (AGILE) Procurement Act, would extend CSO authority. The measure, sent to the Senate floor by the Committee on Homeland Security and Governmental Affairs on August 3, 2022, would allow the Office of Management and Budget to approve pilot and permanent CSO programs and to raise the maximum CSO award from $10 million to $25 million.
For a thorough examination of the history and performance of CSO pilot programs, see “Matching New Technology Solutions With CSOs,” in the December issue of Contract Management.
"We are not paying $1 million for a software license until we know the application works,” the Air Force Colonel snapped. His tech team had pitched him an exciting, potentially impactful, but untested application for enhancing computer vision images. The application was similar to the “auto-enhance” feature on iPhone videos, except for much larger videos and with much cooler features like stabilization and dehazing.
Stabilizing and enhancing DoD imagery is a top priority, and although the Colonel’s team believed in the software, they approached our organization for advice on managing acquisition risk. Their Colonel was trying to avoid being called on the carpet for paying for yet another piece of software that didn’t deliver.
You’ve likely seen this before. A contract is signed, and money spent, but the organization has little to show for it. You might have heard things like, “We couldn’t get our data into the environment,” “The partner organization held us up,” and other explanations. No one wants to be the person to explain why the DoD paid for a software capability that missed the performance benchmark.
Having worked with the Defense Department for more than 15 years, I am painfully aware of the race we are in with near peer (and peer) adversaries. Capable and expeditious software acquisition truly is a matter of national security and is critical for ensuring U.S. competitive military advantage.
We understood the tech team’s problem and began to creatively troubleshoot the computer vision enhancement challenge. We started to bounce around ideas. What about a Cooperative Research and Development Agreement (CRADA)? Maybe a Partnership Intermediary Agreements (PIA) or a Technology Investment Agreement (TIA) would work. We found ourselves coming back again and again to the Commercial Solutions Opening (CSO) Pilot Program.
The Defense CSO pilot is a competitive program authorized by Section 879 of the 2017 National Defense Authorization Act.2 It’s intended to help the DoD obtain solutions or new capabilities that fulfill requirements, close capability gaps, or provide potential technological advances. Given how new it is, we were not surprised that the CSO had not been widely used, except by a few of our DoD sister organizations.
As we white-boarded and brainstormed, the CSO seemed increasingly capable of solving our problem of mitigating the execution risk of advanced software platforms. We landed on five key positive CSO attributes:
▸ Easy to contractually administer
▸ Low burden for the technical leads
▸ Flexible terms for all parties
▸ Low overall cost
▸ Ability to fail fast
The last concept seemed the most difficult to achieve. DoD acquisition makes it exceedingly difficult to fail fast, although that is the backbone of lean and agile software development. The goal is to iterate often and achieve a decision point of pivot or persevere. Moreover, we needed to reach this decision point responsibly and fast.
The problem we faced is common throughout DoD and government agencies: How do we mitigate the risks of software acquisition by rapidly reaching a pivot-or-persevere decision point?
Try Before Buying
What if we could enable software purchasers to try it before they buy it? It seemed possible. What if we provided vendors access to the data environment where the computer vision data is stored? And in return, the vendor gave us software license rights for a pre-determined timeframe.
If the vendor’s software performed well on real DoD data, in a real DoD data environment, we would have reduced execution risk sub-stantially.
Call it revelatory, or an ah-ha moment, but we thought we were on to something.
As we explored the concept of designing a try-before-you-buy contract vehicle for advanced capabilities, we became more and more op-timistic that the CSO was our best option.
Rather than focus narrowly on capabilities, we decided to solicit in nine artificial intelligence (AI) focus areas:
▸ Data Readiness
▸ AI Assurance
▸ Synthetic Data
▸ Edge/Fog Deployments
▸ Data Labeling
▸ Integration With 5G
▸ Modeling and Simulation
▸ AI Security
▸ AI Ethics
We limited focus areas because we didn’t want to overdo it. But, if we decided tomorrow to add a focus area, a CSO would allow us to do so with little more than additional administrative burden.
The flexibility of the CSO as a competitive, merit-based solicitation has been a pleasant surprise for our team. After a provider submits a white paper to our CSO, our technical leads review the submission and discuss the merit. If the capability is innovative and solves a problem within one facet of DoD, it might have broader usefulness throughout government and industry. Those are the capabilities we’re hoping to find.
Circling back to the fourth key positive attribute, we needed to solve the problem of mitigating cost overruns, even for a short-term demonstration.
Free Demos
If we’re going to be bold with this, we thought, why not see if companies will demonstrate for free. This turned out to be a hugely beneficial aspect of our commercial solutions opening. It made sense. The DoD was providing technical experts to give feedback on the plat-form/product, and we were providing the demonstration environment. If a vendor was confident it had built a better mousetrap, and the DoD was going to see value, why wouldn’t the seller be willing to demo gratis?
As we further explored the viability of the CSO combined with free trials, our teammate made a great point. “We can’t just get it for free, right? There has to be consideration.”
This was true, and after deliberation, we decided to offer demonstration reports at the conclusion of each project as an incentive and consideration for the vendor. The report would be developed collaboratively with the vendor and focus on metrics the vendor found important. For example, for a software platform dehazing full-motion imagery – by what percentage did it increase clarity? The report was valuable to the vendor, but the consideration was non-monetary.
As we know, it can be tricky trying a new approach.
Simple Agreements
As white papers, submissions, proposals (all words for the same thing) began rolling in, we shifted our attention to technical evaluation and the structure of agreements. How were we to document the evaluation process once a technical lead made a determination of merit? And how would we structure the document to be signed?
The form that merit-based decision-making takes under a CSO can be challenging to determine. We were not seeking to compare submissions. Truly innovative technologies are notoriously difficult to compare.
Instead, highly qualified technical experts would review submissions in an effort to find an innovative technology that stood out from the crowd. One that had potential for high-level impact in a DoD area of need. Because the technical experts made their determinations based on technical merit, this process was deemed competitive. And the approach made business sense when dealing with cutting-edge technology. (See Figure 1, p. 10.)
We opted to keep the structure of the legally binding demonstration agreement simple. This wasn’t surprising, as our agreements officer is a strong other transaction authority (OTA) advocate who coined the phrase “don’t FAR on my OTA.” Our agreement is simple and flexible, totaling five pages at most. It includes background, period-of-performance, and bilateral termination language, just in case the demonstration hits a hurdle, and we agree to pivot.
We applied our approach to a demonstration of two data readiness capabilities (full motion video enhancement) designed to improve images before they are fed into an algorithm. The algorithm then performs tasks we won’t dive into in this paper, but the overall idea is simple: Feeding better data into the algorithm will equate to a better output.
One capability used graphics processing unit-accelerated multifilter image processing, while the other used brute force mathematical scoring. It sounds as complicated as it is. That’s why we provide technical leads flexibility in making merit-based decisions based on their expertise.
In essence, the two platforms generated similar outputs via different technical means. This is exactly the kind of demonstration we were after. Our technical leads understand an innovative solution when they see it. The CSO structure allows us to trust their technical knowledge and move forward with capabilities showing the highest likelihood of benefiting the warfighter.
Pivot or Persevere
Having multiple vendors demonstrate capabilities at the same time also is critical for getting the best value for the government and the most impactful capability for the warfighter.
During the demonstration period, typically 90 days, the government receives critical insight on the likelihood an innovative technical capability will have application across the department. The vendor receives similar insight and can tweak its product to better serve the DoD and become more marketable. If both platforms appear promising and could help meet one or more DoD mission requirement, we have ensured competitive leverage and prevented vendor lock.
For example, if Company A is the top-rated platform but Company B is more willing to negotiate intellectual property rights and has more reasonable pricing, we have the flexibility to make a best-value determination of what is best for the government and the organization. If the capabilities are of similar value to DoD, we persevere with the best long-term partner. Maintaining competition through multiple, simultaneous demonstrations is a useful approach known as dual prototyping.
When demonstrations conclude, we are faced with a pivot-or-persevere decision. If the demonstrated product doesn’t meet expectations, or a follow-on agreement to produce it at DoD scale cannot be reached, we might pivot. If so, the pivot includes a final report and briefing for the vendor. Vendors gain valuable feedback and insight and are welcome to refine their products and resubmit to our CSO or others.
In the best-case scenario, all the demonstrating vendors will have capabilities of interest. In that case, we have a decision on how to best persevere. A decision to persevere can take many forms.
Flexible Perseverance
A vendor demonstrated a software platform that seemed promising, but we weren’t quite convinced. We needed more data. We decided to bilaterally modify the demonstration agreement and add another 60 days to the period of performance. That gave us the additional runway to collect enough information to make a pivot-or-persevere decision.
In another case, we liked the vendor’s software platform and considered awarding a six-month, firm-fixed-price contract to continue prototype development. At the end of the six months, we’d reach another pivot-or-persevere decision.
If we find software that perfectly meets a customer need, we may consider awarding an indefinite-delivery, indefinite quantity (IDIQ) contract to continue using the application.
A typical IDIQ takes the form of one base year with four one-year option periods. But, we sometimes use a modular approach and break up the option periods into six-month segments to mitigate risk. The IDIQ structure allows us to lock in long-term pricing and intellectual property agreements to ensure long-term value to the government. Within regulations, the acquisition team has almost unlimited flexibility in how it awards a follow-on contract.
Our CSO’s merit-based decision-making and modular contracting approach for demonstrations of innovative commercial technologies have paid dividends. These concepts and real-world examples can help other organizations and acquisition teams overcome hesitation in using commercial solutions openings.
In early 2022, Congress enacted Section 803 of the 2022 NDAA3 to provide the DoD permanent authority to use CSOs. Agencies and commands are enacting standard operating procedures for their respective CSO processes. They are identifying subject matter experts to lead organization-wide training.
CSOs are becoming more widely adopted, but they are hardly mature by DoD contracting standards. As you review various CSO procedures, you may find substantial variation in process, timeline, and overall philosophy of how to best use CSOs.
The Defense Innovation Unit (DIU) and General Services Administration (GSA) are piloting a program to expand the use of successfully completed DIU prototypes that have wide applicability across the government.4 The goal is to streamline onboarding to GSA contracts of successful prototypes with nontraditional vendors. The partnership has seen early success and might be a model for government-wide implementation. GSA has its own CSO Guide.5
The Defense Acquisition University (DAU) has aggregated various templates and trainings from organizations ranging from the Defense In-novation Unit to the Department of Homeland Security, as well as informative white papers.6
The Department of Homeland security was a relatively early adopter of the CSO, drafting their first CSO guide in 2018. Their most recent CSO guide also is available.7
Along with reviewing CSO guides and white papers, I would encourage those interested to read protests related to CSOs, such as these:
▸ PavCon, LLC, B-420640, July 5, 20228
▸ EH Group, Inc., B-419946.2, March 25, 20229
▸ The Ulysses Group, LLC, B-420566, June 7, 202210
I anticipate information and training on CSOs will become more readily available throughout government contracting. As more procedures and policies hit the mainstream, we will run the risk of overcomplicating and bogging-down the CSO process.
We can overcome that risk by ensuring we understand the intent and spirit of the CSO. We can embrace a common understanding that contracting doesn’t have to be complicated to be effective.
Keith Gibson is Founder and Managing Partner at Runyara. He is a seasoned government acquisition leader with experience across multiple civilian and defense agencies, most recently the Chief Digital and Artificial Intelligence Office. He can be reached at [email protected].
ENDNOTES
https://www.ncmahq.org/Web/Insights/Recent-Issues.aspx
https://www.govinfo.gov/app/details/PLAW-114publ328
https://www.congress.gov/bill/117th-congress/senate-bill/1605/text
“Accessing Non-Traditional Technology Vendors Just Got Easier Thanks to New Partnership Between DIU and GSA,” General Services Administration, May 16, 2022, https://www.gsa.gov/about-us/newsroom/news-releases/accessing-nontraditional-technology-vendors-just-got-easier-thanks-to-new-partnership-between-diu-and-gsa-05162022 https://www.gsa.gov/cdnstatic/PIRC%20CSO%20Guide%2062518.pdf https://www.dau.edu/search/Pages/results.aspx?k=Commercial%20Solutions%20Openings https://www.dhs.gov/sites/default/files/publications/commercial_solutions_opening_pilot_program_guide.pdf
https://www.gao.gov/products/b-420640
https://www.gao.gov/products/b-419946.2
https://www.gao.gov/products/b-420566