Dr Paul G. Kaminski
Chairman of the Board and Chief Executive Officer, Technovation, Inc.
Chair, Committee on Pre-Milestone A Systems Engineering
Air Force Studies Board
Division on Engineering and Physical Sciences
National Research Council
Committee on Armed Services
March 3, 2009
Chairman Levin, and Ranking Member McCain:
Thank you for your leadership on DOD acquisition, and for the invitation to testify on these important acquisition issues. Since you have asked me to testify in my role as Chair, Committee on Pre-Milestone A Systems Engineering, Air Force Studies Board, National Research Council, I will begin by providing a summary of our report, which was approved by the Governing Board of the National Research Council and published in 2008. The report is available to the public through the following link, . After the report summary, on page 13 I will provide my personal views on systems engineering and respond to the key issues you asked that I address.
Recent years have seen serious erosion in the ability of U.S. forces to field new weapons systems quickly in response to changing threats, as well as a large increase in the cost of these weapons systems. Today the military’s programs for developing weapons systems take two to three times longer to move from program initiation to system deployment than they did 30 years ago. This slowdown has occurred during a period in which threats have been changing more rapidly than ever and when technology advances and accumulated experience should have been accelerating rather than slowing the development process.
Many causes for this trend have been suggested, including the increased complexity of the tasks and the systems involved from both technological and human/organizational perspectives; funding instability; loss of “mission urgency” after the end of the Cold War; bureaucracy, which increases cost and schedule but not value; and the need to satisfy the demands of an increasingly diverse user community. The difficulty of focusing on a specific, homogeneous, post-Cold War threat made problems even worse. Yet although the suggested causal factors have merit, a common view is that better systems engineering (SE) and development planning could help shorten the time required for development, making it more like what it was 30 years ago.
Simply stated, SE is the translation of a user’s needs into a definition of a system and its architecture through an iterative process that results in an effective system design. SE applies over the entire program life cycle, from concept development to final disposal.
The Committee on Pre-Milestone A Systems Engineering was tasked by the U.S. Air Force to examine the role that SE can play during the defense acquisition life cycle in addressing the root causes of program failure, especially during the pre-Milestone A and early phases of a program. Currently, few formal SE processes are applied to Air Force development programs before the Milestone A review.1
The committee devoted considerable time and space in its report to trying to define a minimum set of systems engineering processes. The most important of these processes are summarized in the checklist in Box S-1 below. A few of the things that need to be taken care of before Milestone A and just after it are the following: the consideration of alternative concepts (solutions) up front; the setting of clear, comprehensive key performance parameters (KPPs) and system requirements; and early attention to interfaces and interface complexity, to the concept of operations (CONOPS), and to the system verification approach. It is these early-stage processes that are covered in this report. The importance of stable requirements and funding between Milestone B and the achievement of initial operational capability (IOC) is stressed, as are processes including good configuration management and change control. The committee further stresses in the report what it regards as six of the most important process areas in its discussion of six “seeds of failure”.
FIGURE S-1 DOD life cycle acquisition process. Points A, B, and C at the top of the figure represent Milestones A, B, and C. LCC, life cycle cost. SOURCE: Richard Andrews, 2003, An Overview of Acquisition Logistics. Fort Belvoir, Va.: Defense Acquisition University. Available at http://www.afcea.org/events/pastevents/documents/Track4Session4AMCEmphasisonCustomerFocusedITInitiatives.ppt#364,12,Slide 12. Last accessed on November 20, 2007.
SYSTEMS ENGINEERING AND THE DOD ACQUISITION LIFE CYCLE
The use of formal systems engineering practices throughout the life cycle of an acquisition program is critical to fielding the required system on time and within budget. Across the top of Figure S-1 are the points at which important management decisions are made: Milestones A, B, and C. Concept development and refinement occur before Milestone A, and further technology development, to reduce system design and development (SDD) risk, occurs before Milestone B. Only after Milestone B does a program become an enterprise with dedicated funding. Importantly, Figure S-1 shows that about three-quarters of total system life cycle costs are influenced by decisions made before the end of the concept refinement phase at Milestone A, while about three-quarters of life cycle funds are not actually spent until after Milestone C. This means that although high-quality SE is necessary during the entire acquisition cycle, the application of SE to decisions made in the pre-Milestone A period is critical to avoiding (or at least minimizing) cost and schedule overruns later in a program. Much of the value of early, high-quality SE will be manifested as success in fulfilling Milestone B requirements.
MAIN FINDINGS AND RECOMMENDATIONS
The committee’s main findings and recommendations are given below.
Finding. Attention to a few critical systems engineering processes and functions particularly during preparation for Milestones A and B is essential to ensuring that Air Force acquisition programs deliver products on time and on budget.
Today’s weapons systems provide unprecedented capabilities but also involve complex interfaces with external command, control, and communications systems and rely on a greater volume of software than ever before. Early decisions on the weapons system requirements and capabilities have a disproportionately large impact on program cost and schedule. The committee also recognizes that a lack of flexibility (a result of overly rigid processes or a lack of trust among program participants or stakeholders) can limit the ability of a program manager to change early decisions that warrant changing.
The committee found many gaps and inconsistencies in the way that the Air Force manages pre-Milestone A activities. The committee heard from presenters of some cases for which required documents were completed pro forma and filed away, never to be seen again, or for which required steps were skipped completely. The current practice of initiating programs at Milestone B denies the acquisition review authority the earlier opportunity (at Milestone A) to make judgments about the maturity of the technologies on which the program is based and to decide whether technologies need to be further developed prior to making a Milestone B commitment to system development and demonstration.
Recommendation. The Air Force leadership should require that Milestones A and B be treated as critical milestones in every acquisition program and that a checklist such as the “Pre-Milestone A/B Checklist” suggested by the committee (see Box S-1 in this Summary) be used to judge successful completion.
A rigorous, standard checklist of systems engineering issues should be addressed by each program through both the pre-Milestone A and pre-Milestone B phases. The committee’s recommended 20-item checklist is shown in Box S-1.
Pre-Milestone A/B Checklist
1. Have at least two alternative concepts to meet the need been evaluated?
The purpose of alternatives is to stimulate thinking to find the simplest, fastest, and cheapest solution.
2. Can an initial capability be achieved within the time that the key program leaders are expected to remain engaged in their current jobs (normally less than 5 years or so after Milestone B)? If this is not possible for a complex major development program, can critical subsystems, or at least a key subset of them, be demonstrated within that time frame?
Achieving capabilities or demonstrating critical subsystems while key programs leaders remain engaged is important to get the capability into service quickly and cost-effectively and to begin the process of incremental improvements based on operational experience.
3. Will risky new technology have been matured before Milestone B? If not, is there an adequate risk mitigation plan?
The development of risky new technology in parallel with a major development program can be costly in terms of both time and money.
4. Have external interface complexities (including dependencies on other programs) been identified and minimized? Is there a plan to mitigate their risks?
Complex, ill-defined, external requirements and interfaces can be a major source of requirements instability during the development phase. This can be of particular importance when a system must operate in a system-of-systems environment.
Key Performance Parameters and CONOPS
5. At Milestone A, have the KPPs been identified in clear, comprehensive, concise terms that are understandable to the users of the system?
It is important that KPPs be expressed in terms understandable to all of the stakeholders. Failure to define the system’s KPPs simply and clearly at Milestone A is a first step to requirements instability and overruns later.
6. At Milestone B, are the major system-level requirements (including all KPPs) defined sufficiently to provide a stable basis for the development through IOC?
Beginning development without a complete list of stable requirements is one of the key “seeds of failure” described in Chapter 4 in this report. It is important to complete requirements trade-offs prior to the development phase.
7. Has a CONOPS been developed showing that the system can be operated to handle the expected throughput and meet response time requirements?
It can be costly to discover too late that the system as designed cannot be operated to meet its requirements.
Cost and Schedule Scoping
8. Are the major known cost and schedule drivers and risks explicitly identified, and is there a plan to track and reduce uncertainty?
Identifying the major cost and schedule risk areas, with particular attention to this checklist and the six seeds of failure—inexperienced leadership, external interface complexity, system complexity, incomplete requirements at Milestone B, immature technology, and high reliance on new software—can help focus management on these issues early.
9. Has the cost confidence level been accepted by the stakeholders for the program?
It is important that stakeholders understand the degree of risk so that the stakeholders will not disrupt the program as the inevitable development program surprises unfold later on. It will generally not be possible by Milestone A or Milestone B to identify all the risk areas that might surface later in a development program, but a frank, early disclosure of known potentials for risk can help sustain stakeholder support later on.
10. Is there a sufficient collection of models and an appropriate simulation environment to validate the selected concept and the CONOPS against the KPPs?
In large, complex programs, the development of models early on can be very important to later management of requirements changes and performance verification.
11. At Milestone B, do the requirements take into account likely future mission growth over the program life cycle?
The committee advocates freezing new requirements and new technology insertion after Milestone B but also notes that making provisions in the initial requirements to facilitate later upgrades could have great long-term value.
12. Has the system been partitioned to define segments that can be independently developed and tested to the greatest degree possible?
Effective partitioning of a complex system can greatly reduce its development cost.
13. By Milestone A, is there a plan to have information exchange protocols established for the whole system and its segments by Milestone B?
Such a plan developed early on can greatly reduce interface problems later in the development phase when they would be more difficult and costly to fix.
14. At Milestone B, has the government structured the program plan to ensure that the contractor addresses the decomposition of requirements to hardware and software elements sufficiently early in the development program?
The histories of programs with cost and schedule overruns are replete with examples of large software developments that had to be redone because requirements from the hardware side were assigned or determined late.
15. Have the key risk drivers (not only the technology drivers) been identified?
Identifying and managing risk early can pay large dividends; it is important to focus on the six “seeds of failure” (see item 8 above).
Program Implementation Strategy
16. Does the government have access over the life of the program to the talent required to manage the program? Does it have a strategy over the life of the program for using the best people available in the government, the FFRDCs, and the professional service industry?
Seasoned management is critical; the government’s job is to find the best!
17. At Milestone A, is there a plan defining how the pre-Milestone B activity will be done, and by whom?
Identifying the program and system managers early, identifying the FFRDC or SETA support needed, thinking through the use of competitive system concept contracts—all can have a decisive impact on the government’s ability to select the best concept, to define by Milestone B system requirements that can remain stable through IOC, and to select the best development contractors.
18. Is there a top-level plan for how the total system will be integrated and tested?
A well-thought-out strategy for verifying system performance, including optimum phasing of verification tests throughout the assembly process, and well-thought-out use of analytical models and external simulators can have a large positive impact on ultimate cost, schedule, and performance.
19. At Milestone B, have sufficiently talented and experienced program and systems engineering managers been identified? Have they been empowered to tailor processes and to enforce requirements stability from Milestone B through IOC?
Seasoned leaders in these areas are critical to maintaining focus and discipline through IOC.
20. Has the government attempted to align the duration of the program manager’s assignment with key deliverables and milestones in the program?
A combination of assignment extension and time-certain milestones will help align incentives.
NOTE: KPP, key performance parameter; CONOPS, concept of operations; IOC, initial operational capability; FFRDC, federally funded research and development center; SETA, systems engineering and technical assistance.
While the committee considers that each item on the checklist is important, it calls attention to several items that warrant further discussion. Item 2 recognizes that the world changes too fast to be friendly to long development cycles. The committee believes that the Air Force should strive to structure major development programs so that initial deployment is achieved within, say, 3 to 7 years. Thirty years ago, this was a typical accomplishment—for example, nearly 40 years ago, the Apollo program put the first man on the Moon in fewer than 8 years.
The development time issue is addressable by applying systems engineering to Items 3, 4, and 13 through 15 before Milestones A and B. The definition of clear KPPs by Milestone A and clear requirements by Milestone B that can remain stable through IOC can be essential to an efficient development phase. It is also important that critical technologies be sufficiently mature prior to starting SDD. The committee observed that although today’s systems are not necessarily more complex internally than those of 30 years ago, their “external complexity” often is greater, because today’s systems are more likely to try to meet many diverse and sometimes contradictory requirements from multiple users. This kind of complexity can often lead to requirements being changed between Milestone B and IOC, and it can lead to relying on immature technology.
Item 19 of the checklist stresses the importance of placing experienced, domain-knowledgeable managers in key program positions. The committee has observed that many of the truly extraordinary development programs of the past, such as Apollo, the Manhattan Project, the early imaging satellite programs, the U-2, the fleet ballistic missile system, and nuclear submarines, were managed by relatively small (and often immature) agencies with few established processes and controls. In that environment, dedicated managers driven by urgent missions accomplished feats that often seem incredible today.
The committee believes that the accumulation of processes and controls over the years—well meant, of course—has stifled domain-based judgment that is necessary for timely success. Formal SE processes should be tailored to the application. But they cannot replace domain expertise. In connection with item 19, the committee recommends that the Air Force place great emphasis on putting seasoned, domain-knowledgeable personnel in key positions—particularly the program manager, the chief system engineer, and the person in charge of “requirements”—and then empower them to tailor standardized processes and procedures as they feel is necessary.
One key pre-Milestone A task is the analysis of alternatives (AoA), which entails evaluating alternative concepts and comparing them in terms of capabilities, costs, risks, and so on. Checklist items 1 through 4, 12, and 13 should be completed before the AoA, while items 5 through 11 and 14 through 20 may be addressed after the AoA.
Finding. The creation of a robust systems engineering process is critically dependent on having experienced systems engineers with adequate knowledge of the domain relevant to a contemplated program.
While the systems engineering process has broad use, effective application depends on having domain experts who are aware of what has gone wrong (and right) in the past, recognize the potential to repeat the successes under new circumstances, and avoid repeating the errors.
Ideally, a person or persons with domain knowledge would have had experience working on exactly the same problem, or at least a problem related to the one at hand. If that is not so (and it might not be if the problem has never been addressed before, as was the case for Apollo and nuclear submarines), the term could be taken to refer to academic training in the relevant field of engineering or science. It could also refer to the practice of critical thinking and problem solving that comes with learning to be a systems engineer and then building on that foundation to gain the experiential knowledge and understanding of engineering in the context of an entire system. Systems engineering is enabled by tools that have been developed to assist in the management of systems engineering (not to be confused with the practice of systems engineering).
Both industry and Air Force presenters told the committee that there are not enough domain-knowledgeable and experienced systems engineers to support all of the programs that need them.
Recommendation. The Air Force should assess its needs for officers and civilians in the systems engineering field and evaluate whether either its internal training programs, which include assignments on Air Force programs that provide mentoring by experienced people and hands-on experience in the application of systems engineering principles, or external organizations are able to produce the required quality and quantity of systems engineers and systems engineering skills. Based on this assessment, the Air Force first should determine how and where students should be trained, in what numbers, and at what cost, and then implement a program that meets its needs.
The Air Force needs to attract, develop, reward, and retain systems engineers across the full spectrum of relevant domains, engage them in the early (pre-Milestone A) phase of new programs (or modification programs), and sustain their participation throughout the life of the programs. One important step in this process would be to create an Air Force occupational code for systems engineering so that engineers’ experience and education can be tracked and managed more effectively. The Air Force should support an internal systems engineering career track that rewards the mentoring of junior systems engineering personnel, provides engineers with broad systems engineering experience, provides appropriate financial compensation to senior systems engineers, and enables an engineering career path into program management and operations.
Finding. The government, federally funded research and development centers (FFRDCs), and industry all have important roles to play throughout the acquisition life cycle of modern weapons systems.
Since the need for a new or upgraded weapons system is most often first recognized by the military user, it is appropriate for the military to codify its requirements and, with support from FFRDC and independent systems engineering and technical assistance (SETA) contractors, to explore materiel and nonmateriel solutions (such as doctrinal, organizational, or procedural changes) as well as to assess the potential for new technology to provide enhanced capabilities. While it is appropriate and usually desirable to engage development contractors in the pre-Milestone B process using competitive study contracts, the source selection for system development and demonstration should not be made until after the work associated with Milestones A and B is complete.
Recommendation. Decisions made prior to Milestone A should be supported by a rigorous systems analysis and systems engineering process involving teams of users, acquirers, and industry representatives.
Working together, government and industry can develop and explore solutions using systems engineering methodology to arrive at an optimal systems solution.
Finding. The Air Force used to have a development planning organization that applied pre-Milestone A systems engineering processes to a number of successful programs, but that organization was allowed to lapse.
The role of the Air Force development planning organization, which was within the Air Force Systems Command, was to provide standard evaluation tools and perform pre-Milestone A systems engineering functions across acquisition programs. The early 1990s saw an erosion of this front-end planning organization along with its funding as the Air Force Systems Command (now the Air Force Materiel Command) began to play a decreasing role in program execution. In the opinion of several speakers who met with the committee, one main reason for the erosion of funding was a lack of congressional support for the planning function.
Recommendation. A development planning function should be established in the military departments to coordinate the concept development and refinement phase of all acquisition programs to ensure that the capabilities required by the country as a whole are considered and that unifying strategies such as network-centric operations and interoperability are addressed.
The Air Force and the other military services should establish a development planning organization like that which existed in the early 1990s.
The roles and functions of the various organizations involved in acquiring major weapons systems need to be clearly defined. The responsibility for executing systems engineering and program management in the pre-Milestone A and B phases should be vested in the military departments that do the actual development planning functions. This should not be the responsibility of the Office of the Secretary of Defense (OSD) or of the Joint Staff. Instead, those offices need to enable the creation and functioning of military department development planning organizations with policy measures and, where appropriate, resources. The Joint Staff, under the auspices of the Joint Requirements Oversight Council (JROC), may help to define the requirements for major programs in the course of the development planning process, but it should not run the process itself.
The existence of “joint” programs or a program such as Missile Defense, which has several related systems being developed by different military services, requires clear guidance from both OSD and the Joint Staff about who is in charge. These programs need to be harmonized and integrated by the responsible integrating agency. However, development planning activities should still take place in the military departments where the expertise resides. Consequently, the development planning should be managed by that agency.
While this committee cannot predict how Congress will view the revival of a good planning process to support pre-Milestone A program efforts, it is still important for the Air Force and DOD to make the case for the critical importance of this process before Congress and others. A development planning process is important not to start new programs, but rather to ensure that any new program (or a new start of any kind) is initiated with the foundation needed for success. Funding for this planning function needs to be determined by the military services, including both the acquisition communities and those (the warfighters) who generate the operational requirements.
Many of the conclusions reached and recommendations made by the committee are similar to those of previous reviews. Most of the past recommendations were never implemented, so one of this committee’s most critical thoughts relates to the importance of implementation. A sampling of key findings and recommendations from previous studies follows:
• Government Accountability Office (GAO)2, 3
— Separate technology development from systems acquisition. Commit to a program only if the technology is sufficiently mature. Set the minimum Technology Readiness Level (TRL).
— Stabilize the requirements early.
— Employ systems engineering techniques before committing to product development.
— Employ evolutionary approaches that pursue incremental increases in capability.
— Address shortfalls in science, engineering, and program management staff.
• National Defense Industrial Association (NDIA)4
— Increase SE awareness and recognize SE authority in the program formulation and decision process.
— Incentivize career SE positions within the government.
• Defense Science Board (DSB)5
— Overhaul the requirements process.
— Stabilize acquisition tours.
— Establish a robust SE capability.
• Defense Acquisition Performance Assessment (DAPA)6
— Strategic technology exploitation is a key U.S. advantage. Opportunities need to be identified early.
— The U.S. economic and security environments have changed—for example, there are fewer prime contractors, smaller production runs, reduced plant capacity, fewer programs, and unpredictable threats.
— The acquisition system must deal with instability of external funding.
— The DOD management model is based on a lack of trust. Quantity of review has replaced quality. There is no clear line of responsibility, authority, or accountability.
— Oversight is preferred to accountability.
— Oversight is complex, not process- or program-focused (as it should be).
— The complexity of the acquisition process increases costs and draws out the schedule.
— Incremental improvement applied solely to the “little a” acquisition process7 requires all processes to be stable—but they are not.
The committee notes that successful implementation of these recommendations requires the “zipper concept”—making connections at all levels, from the senior leadership of the Air Force and DOD down to the working levels within key program management offices and supervisory staffs.
ADDITIONAL QUESTIONS AND PERSONAL VIEWS
Having summarized the findings and recommendations of the Committee on Pre-Milestone A Systems Engineering, let me now add my personal views on systems engineering, and the two additional questions that you asked me to address: 1) the systemic issues that have contributed to cost, schedule, and performance problems in the acquisition of major weapon systems; and 2) the steps that Congress and the Department need to take to improve performance of the Department’s acquisition programs.
I agree with Secretary Gates who said that there is no one silver bullet that will correct all of the DOD acquisition problems. But I believe that good systems engineering coupled with effective development planning are the two most important contributors to successful acquisition. Our report provided formal definitions of systems engineering and development planning that are somewhat arcane. So rather than provide further definition, I find it easier to illustrate by choosing examples of good and bad systems engineering and development planning. Examples of good work include the Apollo Program and the USAF ICBM programs (e.g., Minuteman, MX) in the 1970s-80s. Apollo succeeded in putting men on the moon in about 8 years. At the start of the program, almost all of the key technologies were immature. But good systems engineering and development planning were applied to develop a systematic approach, reducing risk by taking a series of limited steps, and applying the learning and domain experience gained from each step to the subsequent step. The USAF ICBM programs used a similar approach, beginning with conceptual studies and technology development, and holding initiation of full scale development (FSD) contracts until key guidance system, re-entry system and propulsion technologies had been demonstrated. As a result, the time from initiation of FSD until first flight was typically 3-4 years. An example of poor systems engineering is the SBIRS program, in which a lack of domain experience and analysis led to a failure to anticipate the possibility of severe radio frequency interference between two key payloads - discovering this problem years after program initiation. Inadequate systems engineering and development are therefore the first of five items listed below as systemic contributors to acquisition problems.
Systemic Contributors to Acquisition Problems
1. The lack of early and continuing systems engineering and the absence of a closely-coupled development planning program are a fundamental contributor, as identified in our report. The root causes include: a) lack of sufficient personnel (in both government and industry) with adequate education, training and domain experience (this includes personnel in requirements development as well as in acquisition); and b) lack of sufficient front end investment necessary to understand the key tradeoffs in cost/schedule/ performance, and to identify and address the key risks in a systematic manner.
2. Lack of alignment of responsibility, authority, and accountability of the program manager. In many cases the program manager’s authority is diffused by many levels of oversight in both the Department and in the Congress, and the financial and performance constraints imposed do not allow sufficient freedom of action to apply informed judgment in a timely manner. Flexibility is further limited by application of a “one-size-fits all” approach imposed by the DOD 5000 system, and the oversight practiced by the DOD and the Congress. A program manager needs the freedom to tailor the acquisition approach to the problem, to insure that the program response time will fall within the response time of the threat, and to apply a variety of tools and techniques (such as the use of prototypes, competitive prototypes, modeling and simulation, critical subsystem and component demostration). For this to work, we need program managers with the education, training and domain experience needed to enable timely responses and excellent judgment relevant to the domain.
3. Lack of stability in program funding.
4. Lack of early attention to test and evaluation, with insufficient planning and investment in the tools (e.g., modeling and simulation, test equipment, facilities & personnel) to provide the timely and meaningful results needed by program management and system engineering to continually refine performance objectives and development plans.
5. Excessive (and growing) time from program initiation to fielding. As this time increases from a few years to 15 years or more, it undermines the entire acquisition process by causing key participants to “lose the recipe”, and lose a sense of accountability as well as a sense of being able to make a difference. When new capabilities are developed and fielded in 5 years, engineers, managers, testers, cost analysts, etc. are able to benefit from and apply the experience gained from a previous program or program phase. They can also see the results of their decisions and be held accountable. We can also meaningfully employ past performance of the contractor as a factor in the award of future programs – an important factor in incentivizing contractor performance. This all changes dramatically when the time extends to 15 years, and we have 5 roll-overs of management, engineers, cost analysts, and commercial technology during this time period. This long and growing time period is a result of the inflexibility inherent in our entire system of requirements development, budgeting and acquisition, and it creates a vicious cycle in which it further exacerbates the contributors above, and they in turn further increase the time and cost growth. We see the result when we must discard our current acquisition system in order to deal with urgent needs and field systems such as MRAP and jammers to counter IED’s by forming and using rapid reaction organizations. This cycle must be broken by attacking the root causes.
Steps that Congress and the Department Need to Take
1. The first step is to insure that we not only restore, but enhance early and continuing systems engineering coupled with effective development planning. This will require commitment of more significant investment dollars earlier in our acquisition programs, and a commitment to build a cadre of systems engineers and development planners with the education, training and domain experience needed to be effective. Attracting “best and brightest” to this work - and keeping them - will require a personnel system that will identify and track these important human resources and establish a career path to allow those who are successful to advance to senior program management and leadership positions. Their domain experience will be enhanced by managing the building of critical subsystems during the development planning program, reducing risk and building skills and experience at the same time. The Congress and the Department can assist by providing incentives for attracting and keeping key personnel (not only financial incentives, but educational, training, recognition, and most important – the opportunity to take on challenging developments and see that they can make a difference). We will need metrics to assess how well we are doing in building and applying this cadre, and we must recognize that this will not be accomplished in four, or even eight years. But we must begin in earnest and begin now. Finally, we need a means to insure that we have adequate funding up front for new programs; one approach would require a report at program initiation from an independent cost analyst working with system engineers and development planners who have developed their skills on previous programs.
2. Alignment of the responsibility, authority, and accountability of the program manager requires that a degree of trust be established between the program manager and those responsible for our oversight mechanisms. We must be prepared to delegate authority to the program manager, and provide him or her with some flexibility to manage – to adjust levels and allocation of funding, to adjust the allocation of performance parameters, to adjust schedule, and to tailor the acquisition approach to be responsive to the need. Clearly, there must be bounds established beyond which the program manager must seek approval from oversight authorities. But I believe these bounds are too narrow and inflexible today. One size does not fit all programs. The Congress and the Department should be willing to consider and tailor many of the restrictions which unnecessarily limit and delay program managers today. I have seen many of our successful classified special programs benefit from greater management flexibility than that afforded to their in conventional program counterparts. The good managers of these special programs have used that flexibility to the benefit of the program and the Department by operating with transparency and maintaining trust. I realize that it seems counterintuitive to recommend greater flexibility and trust in an environment rife with acquisition problems, but I believe we need to break the current cycle. One way to begin is with a limited number of pilot programs, with first priority to those programs addressing urgent needs, and assignment of our most experienced program managers to meet those urgent needs. Since these programs will be moving with dispatch, they offer the best opportunity to produce early indications of whether this is a sound approach which should be extended to other programs.
3. Improving funding stability will require that the Department and the Congress be willing to give up some of their flexibility in making annual (or more frequent) adjustments in funding. Doing so will require tradeoffs of the costs and benefits, and I believe it is time to make explicit consideration of these tradeoffs. I have seen the projected benefits of stable funding by looking at theoretical Monte-Carlo simulations (which show efficiency improvements of perhaps 8-10% as a result of holding a small capital reserve of less than 10 %). We can also see the benefits of multiyear procurements saving similar or greater amounts. I have also seen many examples in which funding cuts of x dollars today result in later additions of 3x dollars to catch up.
4. Giving early and serious attention to test and evaluation will require strengthening our test and evaluation organizations and personnel. Test and evaluation is often an afterthought, and contracts are often written without any mention of how we will test the product. We spend large amounts of money when a large development team waits for test results. The alternative is to spend less money and time by considering testing and investment in test resources as part of our systems engineering and development planning efforts. The actions recommended in paragraph 1 above are the same actions required to address these critical test and evaluation needs.
5. Reducing the time from program initiation to fielding will require the combination of all actions suggested above. Further benefits will be derived by placing more emphasis on time-certain acquisition. This will be helped by better development planning and alignment of incentives. With good development planning, we can assign managers to develop prototypes, critical systems or components needed to better understand cost/performance trades and reduce risk. It is reasonable to expect that many of these developments can be completed in 2-4 years, so one manager will be in place from start to delivery. This will help align authority and accountability in both government and industry. As these critical subsystems are delivered and tested, the risk reduction and domain experience gained in both government and industry will allow us to reduce the time required to develop, integrate and test the full system. We can also apply meaningful incentive programs to link profits to demonstrated performance, and use that performance as a factor in making future competitive awards. We can rely on the experience gained during development planning to apply informed judgment to adjust requirements to improve value, reduce time, and better estimate and manage costs. The Department and the Congress can assist by placing more emphasis on time-certain acquisition, with the opportunity for milestone reviews at the completion of major development planning activities.
I believe action on these five issues will have a significant and demonstrable impact on our serious acquisition problems. We need to move now with the same urgency and priority that we expect in combat operations to permit the timely and effective development and fielding of new capabilities and services with what I expect will be more limited future defense dollars
This is a result of the elimination in the 1990s of the development planning function that had existed in the Air Force Systems Command.2.
GAO, 2003, Defense Acquisitions: Improvements Needed in Space Systems Acquisition Management Policy
, September. Available at http://www.gao.gov/new.items/d031073.pdf. Last accessed April 2, 2007.3.
GAO, 2005, Space Acquisitions: Stronger Development Practices and Investment Planning Needed to Address Continuing Problems
, July. Available at http://www.gao.gov/new.items/d05891t.pdf. Last accessed April 2, 2007.4.
NDIA Systems Engineering Division, 2003, Task Report: Top Five Systems Engineering Issues in Defense Industry
, January, Arlington, Va.: NDIA.5.
DSB/Air Force Scientific Advisory Board (AFSAB) Joint Task Force, 2003, Acquisition of National Security Space Programs
, May, , D.C.6.
Ronald Kadish, Gerald Abbott, Frank Cappuccio, Richard Hawley, Paul Kern, and Donald Kozlowski. 2006. Defense Acquisition Performance Assessment
. Available at http://www.acq.osd.mil/dapaproject/documents/DAPA-Report-web/DAPA-Report-web-feb21.pdf. Last accessed on April 2, 2007.7.
The Acquisition—“Big A”—system is often believed to be a simple construct that efficiently integrates three independent processes: requirements, budgeting, and acquisition. “Little a” on the other hand, refers to the acquisition process that focuses on “how to buy” in an effort to balance cost, schedule, and performance; it does not include requirements and budgeting.
An archived webcast of the hearing can be found on .