Measuring efficiency of lean six sigma project implementation using data envelopment analysis at NASA

Purpose: This study aims to review the implementation of the Lean Six Sigma project methodology in the Johnson Space Center (JSC) business environment of National Aeronautics and Space Administration (NASA) with an objective of evaluating performance of individual projects and to develop recommendation for strategies to improve operational efficiencies based on Data Envelopment Analysis (DEA). Design/methodology/approach: In this study, authors propose the Lean Six Sigma project performance evaluation model (LSS-PPEM) based on Data DEA where Critical Success Factors (CSFs) and Total Team Hours serve as inputs while Process Sigma and Cost avoidance are used as outputs. The CSFs are factors that critically affect the performance of LSS at JSC. Six of those are identified by the Black Belts through Analytical Hierarchical Process, and the values of those are decided by project leaders and Green Belts through survey. Eighteen LSS projects are evaluated, and their results are analyzed. Findings: Eventually, four out of the six CSFs are adopted for this study based upon Pearson correlation analysis, and those four include Project execution and follow up of results; Top management’s commitment and participation; The use of data analysis with easily obtainable data; Attention given to both long and short term targets. Using data between the years 2009 and 2011, seven of the eighteen projects are found to be efficient. The benchmark analysis and slack analysis are conducted to provide further recommendation for JSC managers. Three out of those seven efficient projects are most frequently used as an efficient peer.


Introduction
Lean Six Sigma (LSS) is an integrated business process improvement methodology where Lean Manufacturing and Six Sigma systems are combined to improve the quality of business processes through elimination of any non-value adding activities or wastes.LSS has generated substantial benefits and savings within the private sector.For example, the application of LSS within the private sector has proven profitable to companies such as Lockheed Martin, $5 million savings on one project and Stanford Hospital and Clinics, lowering their material costs by $25 million.GE credits LSS in adding $2 billion in profits to the 1999 total of $10.7 billion, and ITT Industries increased profits by $130 million in the second year of their LSS implementation (George, 2003).
Government agencies have surveyed private industry when those come to improving the processes of delivering services related to their mission.Methodologies such as TQM, Just-In-Time, and Plan Do Check Act (PDCA) have been used with some success over the years.
However, these programs have generally failed due to an inability to properly quantify their performance.The most recent move toward continuous improvement methodologies has steered many government agencies into utilizing LSS for process improvement.The Johnson Space Center (JSC) at Houston, one of the National Aeronautics and Space Administration (NASA) centers, embarked on an initiative to implement LSS in 2009 across the Center in an effort to improve operational results.
In general, LSS programs in the private sector can easily be measured in monetary terms given the implementers clear definition of the success of the program.However, the story changes within the public sector, where profit is not a motivating factor.Instead a government organization has a primary goal of completing its mission assigned to it by the current US government administration.Government agencies operate under a controlled budget where profit is not a motivating factor, instead each organization within the agency is held accountable for meeting its performance metrics in relation to reaching the agency's mission.
The government budget process is structured in such a way as to encourage the spending of allocated funds instead of cost savings or profit generation.Hence, some managers regard cost savings or avoidance as a potential loss in their budget because senior management could decide to use the savings elsewhere.Bissell, Garcia and Telles (2006) identified this as a uniquely governmental issue.They noted that savings realized by process improvement were often consumed by preplanned budget cuts by headquarters eliminating the possibility of any reinvestment.Therefore, offices involved in the process improvement were not able to share or benefit in the cost savings.They concluded that it must be made clear that increases in efficiency and elimination of waste are the true benefits to the government, and this fact should not be seen as a disincentive despite the probable lack of direct tangible benefit to the saver.
This study aims to review the implementation of the LSS methodology in the JSC business environment with an objective of evaluating performance of individual projects and to develop recommendation for strategies to improve operational efficiencies.

Lean Six Sigma at NASA-JSC
The JSC was established in 1961 as the Manned Spacecraft Center and, in 1973, renamed in honor of the late President and Texas native, Lyndon B. Johnson.From the early Gemini, Apollo and Skylab projects to today's space shuttle, International Space Station and Exploration Programs, the center continues to lead NASA's efforts in human space exploration.The Center resides on 1,620 acres with over 100 buildings housing the civil service and contractor workforce.The JSC civil service workforce consists of about 3,000 employees, the majority of whom are professional engineers and scientists.About 50 companies provide contractor personnel to JSC with more than 12,000 contractors working onsite or in nearby office buildings and other facilities.JSC's organization structure is divided into 19 programmatic and institutional directorates supporting the overall NASA's mission.Each directorate is broken down into multiple divisions then branches and in some cases, to project team levels.While each directorate is committed to meeting JSC's mission, directors are given latitude in how they chose to manage.This is an important perception to understand as management philosophy and tactics have been observed to vary from group to group, from the highest to the lowest.This provides a great deal of complexity in implementing a continual improvement methodology, such as LSS.Furthermore, there are 109 identified business functions that spread out across the directorates.Each business function has tens of documented processes, bringing the total to several thousand processes which can be reviewed for improvement.
The primary objectives of LSS at JSC are to enhance mission success, improve on cost quality and schedule, reduce variability, and enable consistently high quality products and services.To accomplish this, NASA decided to focus its LSS events on product/process development Kaizens and Value Stream Mapping (VSM) instead of the more traditional Design, Measure, Analyze, Improve and Control (DMAIC) project approach.A "Kaizen event" is a short-term (3 to 5 days) team-based improvement project focused on eliminating waste in and increasing the performance of a specific process or product line, through low cost, creativity-based solutions (Melnyk, Calantone, Montabon & Smith, 1998;Bicheno, 2001).Kaizen events are often associated with the implementation of lean production practices (Vasilash, 1997;Kirby & Greene, 2003) and often employ lean concepts and tools such as, Value Stream Mapping, Failure Mode Effect Analysis, and Work Standardization (Bodek, 2002;Melnyk et al. 1998;Oakeson, 1997).
During this Kaizen event period, participants spend 100% of their time on the project and are removed from the regular responsibilities, while e-mails, voicemails, and other distractions are minimized or forbidden.Additionally Black Belts and team leads in conjunction with the sponsor and champion develop a well-defined project charter and scope prior to the event.All basic data to evaluate the process is gathered by the Black Belt or team lead prior to the event.At the conclusion of the event the proposed solutions are acted upon immediately.The rationale for this method is the belief of doing something now, which is roughly right, is better versus waiting the 3 to 6 months it normally takes for the standard DMAIC approach.

Motivation and Problem Descriptions
This study analyzes the performance of LSS projects implemented at the National Aeronautics and Space Administration-Johnson Space Center (NASA-JSC).JSC's initiative to infuse lean principles into the Directorates has been slowed by management acceptance and understanding.While a majority of the managers have received training as to the methodology and benefits of LSS, there is an apparent reluctance to implement within their offices due to a lack of understanding of how to gauge the success of the projects in terms a government program manager can relate with.Hence, it would be very useful to systematically evaluate and analyze the performance of LSS conducted at JSC.This systematic evaluation approach includes some common factors which critically affect the performance of the LSS, and the project evaluation method.
From a project standpoint, NASA uses the Earned Value Management (EVM) as a means to measure performance.EVM provides a mechanism for measuring the performance of the project against its stated baseline.As long as the project meets the milestones set forth in the project plan, within scope, schedule and budget, it is deemed successful.However, EMV cannot be used to measure how efficiently a project applied its resources or how it is compared with other projects in the same domain.Lean projects differ from traditional projects since they generally do not deliver a unique product but rather are conducted to remove waste and variation within processes.Hence, since both cost savings and EVM fail to adequately measure the true benefit achieved by implementing Lean Six Sigma, a different methodology is required.
A key criterion in understanding the value of a LSS project is to determine how well it has been implemented within an organization.To understand this, one would need to identify and prioritize the important attributes or critical success factors (CSFs) which significantly affect the LSS project performance.Then, the identified CSFs can be viewed as key input factors to the LSS project performance evaluation model (LSS-PPEM).In this study, authors used the Analytical Hierarch Process (AHP)-based survey to identify and prioritize the critical success factors.The critical success factors necessary for implementing a successful LSS project have been discussed in detail in a subsequent section.
The LSS-PPEM is to compare the efficiency of one LSS project to others based on CSFs and other input and output factors.This comparison will give JSC managers insights into the value that the LSS project contributes to the Center.Data Envelopment Analysis (DEA) lends itself to this problem as it provides a model to measure the relative efficiency in situations where there is no clear approach of aggregating pre-determined multiple inputs and outputs into a meaningful index of productive efficiency.Based on the results of the DEA, organizational-wide efficiency can be evaluated and ranked.In DEA, the organization under study is called a decision making unit (DMU).Generically, a DMU is regarded as the entity responsible for converting inputs into outputs and whose performances are to be evaluated -each LSS project in this study.For the purpose of securing relative comparisons, a group of DMUs is used to evaluate each other with each DMU having a certain degree of managerial freedom and decision making (Cooper, Seiford & Tone, 2007).
The specific objective of this study is to develop a LSS-PPEM using AHP-based survey and DEA to provide a mechanism for JSC -possibly for government agencies -to measure the performance of LSS process improvement projects.This study provides several benefits to JSC.Some of them include 1) it can provide a JSC-specific LSS-PPEM based on its own CSFs; 2) the results from LSS-PPEM can provide specific benchmark opportunities among DMUs in terms of inputs and outputs, allowing JSC management insights into possible inefficiencies in systems and management techniques across the Center; 3) this method can be extended into other government agencies.Rockart (1979) defined CSFs as "the limited numbers of areas in which results, if they are satisfactory, will ensure competitive performance for the organization".Brotherton and Shaw (1996) defined CSFs as the essential things that must be achieved by the company or which areas will produce the greatest "competitive leverage".They emphasized that CSFs are not objectives, but are the actions and processes that can be controlled or affected by management to achieve the organization's goals.They also stated that the CSFs are not static, but depend on a combination of where the organization is and where it wants to be.This definition again comes from the private (service) sector and is more a way of managing rather than an assessment of a project's success.Both definitions emphasize gaining a competitive edge, which is not a feature of public sector organizations.Boynton and Zmud (1984) defined CSFs as "those few things that must go well to ensure success".This is a more universal definition which is equally applicable to both the public and private sectors and not restricted to a particular type of project.The importance of defining the CSF for implementation is to increase the success rate, reduce costs and prevent disillusionment with continuous improvement programs.

Critical Success Factors
There are several models of CSFs for the deployment of LSS projects in the literature, and those include various factors identified as critical for success in deploying Six Sigma projects, each of the following authors proposed a different model: Zu, Fredendall and Douglas (2008), Su andChou (2008), Schroeder, Lindermana, Liedtkeb andChoo (2008), Choo, Linderman and Schroeder (2007), Kwak andAnbary (2006), andSmith (2003).(2002), Burton and Sams (2005), Furterer (2004) and Hayes (2002) identified many of the factors highlighted by Ho et al. (2008).Based on the review of literature of the CSFs, authors accept the factors identified and displayed below as those factors that must be embodied within an organization to ensure the successful implementation of a LSS project.

Data Envelopment Analysis for LSS
DEA has been widely researched over the last 30 years.Table 2. DEA popular key words (Excerpted from Emrouznejad, et.al., 2008) An exploration of the keywords discovered that only 43 of the over 4000 articles were regarding the use of DEA in a government environment, be it municipality, state or federal.
Many of those articles pertained to efficiency in accounting practices, employment issues, policy concerns and spending.Hence, authors attempted to search articles that applied DEA to a project for the evaluation purpose.Few articles were found with the keyword of process improvement and one combining the use of DEA and Six Sigma.Kumar, Saranga, Ramírez-Márquez and Nowicki (2007) applied DEA as a method for the selection of Six Sigma projects.More recently, Feng and Anthony (2009) proposed integrating DEA into the Six Sigma framework adding it as a tool for Black Belts.
Lengacher ( 2010) wrote an exploratory article for iSixSima extoling the benefits of DEA as a method to calculate the efficiency of process improvement.Yüksel ( 2012) also claimed that DEA can be applied to evaluate Six Sigma projects and however, he also addressed that this application was not much discussed in the literature.To the best knowledge of authors in the review of the literature, authors feel that the application of DEA to LSS was not much studied.

Overview of research methodology
For the use of DEA as a performance measure for LSS, the following overall procedure in Figure 1 is presented, and it will be applied to the JSC case study.Each phase of the procedure is discussed in detail in this section.

Planning
The planning phase decides the inputs and outputs used in DEA to measure efficiency.The efficiency is a ratio of the weighted sum of outputs to the weighted sum of inputs, and DEA will decide the optimal values of these weights to maximize the efficiency of each DMU using mathematical linear programming.In regards to this study, a DMU is a LSS project conducted at JSC with the objective to improve an underlying process which could either be production or transaction oriented.First, two outputs are determined: Percentage Increase of Process Sigma (PIPS) and Cost avoidance.Process Sigma is a measure of the variation of a LSS project performance (e.g., lead time reduction, defects etc.) in a project relative to the project customer requirements.The higher Process Sigma, the fewer defects.Cost avoidance represents the expected amount of cost saving due to the project.Although there is no exact rule or guideline in selecting inputs and outputs, Malhotra, Malhotra and Lermack (2009) suggested that the level of resources or factor to be minimize should be considered as the inputs while the level of economic variable factors and the degree of their contribution to the overall efficiency should be considered as the outputs.From this perspective, the selection of outputs seems to be very intuitive and straightforward.However, some input factor identification is not so straightforward.First, authors select the Total Team Hours (TTH) spent as an input since it well represents amount of efforts spent for a project -TTH represents the total calendar hours spent by teams to complete a project.Now, authors need to identify what CSFs are meaningful in JSC.For this, authors initially consider 14 CSFs -note that we add 'efficiency of training materials' as an additional factor to those 13 factors listed in Table 1 based on several Black Belts' opinions during the planning phase.
However, the use of 15 inputs -14 CSFs and TTH -into the DEA model posed an issue.As the number of inputs and outputs increases, the power of DEA decreases, to some degree.Malhotra et al. (2009) suggested that the number of samples is greater than or equal to two to three times of the total number of the inputs and outputs to appropriately utilize the power of DEA.Hence, meaningful inputs and outputs selection should be undertaken before DEA implementation.Since TTH is considered as an important quantitative factor, authors decide to keep it.Instead, authors decide to identify the relevant CSFs from those 14 CSFs.At JSC, each unique project has a project lead supported by a LSS Black Belt and Green Belt.The project lead is responsible for the execution and implementation of the project, while the Black Belt and Green Belt are responsible for facilitating the event and assisting in deriving the improvement by using the LSS tool set.Hence, a questionnaire based on the Analytical Hierarchal Process (AHP) is sent to all Black Belts, as the recognized experts in their area on LSS.
AHP is a decision analysis approach developed by Saaty (1980).AHP is capable of extracting the judgment of multiple experts and decision makers, and is mainly applicable to handling the problems arising in an uncertain environment in which multiple evaluation criteria exist.It conducts the pair-wise comparison of any two objects based on the scale displayed in Table 3.
Any Consistency Ratio (CR) greater than 10%, the decisions are considered unreliable because they are too close to randomness and the exercise is insignificant or must be repeated.
Readers are encouraged to refer to Saaty (1980) for further details for AHP.The AHP results provide a prioritized ranking and weight of the importance of the CSFs to events conducted in the Black Belt's area.The acquired weight distribution is expected to be more objective than setting the weight for individual factors.A Pareto chart is constructed using the results of the AHP questionnaires, and the top 70% CSFs are selected as meaningful inputs.Table 3. Description of AHP scale

Data collection
This phase is to collect all data for inputs and outputs selected in the previous phase.
Specifically, it is to quantify the level of input and output values.Since two outputs (PIPS and Cost avoidance) and one input (TTH) are quantitative factors, they can be relatively easily collected from historical documents.However, those CSFs selected in the planning phase need an additional survey-based quantification step to decide their values based on survey participants' judgment.For each CSF, authors provide five questions, and each of those five questions are evaluated by project leads and Green Belts based on the Likert scale of 1 to 5 with score of 1 for "Not at All" and 5 for "Always".

Model verification and other phases
Subsequent to the data collection phase, the model verification phase is conducted.This study proposes the use of DEA for LSS-PPEM.In DEA, a fundamental assumption is if a given DMU is capable of producing units of output with inputs, then other DMUs shall also be able to do the same if they were to operate efficiently.Another important assumption is that the DMUs are homogeneous in terms of their characteristics.Otherwise, the computation of the relative efficiency does not make sense.
Since it is considered the outputs are more controllable than inputs -in fact, measuring CSFs is more qualitative and judgment based, the output-oriented model is adopted.Authors also adopt Constant Return to Scale (CRS) for simplicity.Bo (2005) stated that the inputs and outputs in DEA have the isotropic relation.Based on this and the CRS where it is assumed that an increase in the amount of inputs consumed would lead to a proportional increase the amount of outputs produced, the Pearson correlation analysis between all inputs and outputs are conducted.Through this phase, some inputs may be dropped from further consideration if they do not show the positive relationship with any output.The final check for sample sizes and the number of inputs and output is also conducted based on the suggestion by Malhotra et al. (2009).Once the model is verified, the model is ready for execution and analysis.

Implementation of overall research procedure
The overall research procedure in the previous section is applied to the JSC.51 LSS projects were conducted between March of 2009 and December of 2011.Of that total, 30 projects were for process improvement.Project leads, Black Belts, Green Belts of these events were asked to participate in the questionnaires and surveys.All data was gathered during the normal course of operations.Participants were sent an email containing either a questionnaire or survey depending on their participation in the project.All information was gathered between September 2012 and December 2012.While JSC management supported the effort, no direction was given to event participants requiring them to submit responses.The timing of the data gathering could pose a bias in the study.Some of the responses were for projects conducted over a year ago.

AHP questionnaire
To identify and prioritize the CSFs, a macro enabled Excel ® spreadsheet was developed establishing a pair-wise comparison matrix of the 14 CSFs.The spreadsheet is coded to accept a value or judgment in a cell based on the AHP rating scale and automatically inserts the reciprocal value in the corresponding cell.The spreadsheet calculates the priority vector and eigenvalue as the participant completes the form as well as CR.All cells other than those requiring input by the participant were locked.The spreadsheet was sent to a total of 67 civil servants and contractor Black Belts across JSC with a request to complete and return within a week.They were instructed to maintain CR value less than or equal to 10% while they rate the CSFs based on their expert perception of its importance in the success of a LSS project.Thirtythree forms were returned and evaluated.The ratings from each Black Belt were totaled and averaged across the comparison matrix to create a prioritized ranking of each CSF.Table 4 displays the final ranking of the CSFs.The calculated CR for the matrix result was 0.071.
Therefore the judgments are considered trustworthy and reliable.As noted above, DEA loses some of its power when too many inputs and outputs are used.To solve this, a decision was made to use the top 70 percentile of the CSFs in the DEA model.The bold type in Table 4 indicates the resultant set of CSFs used in the subsequent survey.

CSF Input measure survey
Upon identifying the top CSFs, the following survey in Table 5 was sent out to quantify the level of each CSF.The survey consisted of 6 critical success factors with 5 items each, a total of 30 questions.A Likert scale of 1 to 5 was used with a score of 1 for "Not at All" and 5 for "Always".The survey validation was done by the Cronbach's Alpha test.
As previously described, surveys representing 18 projects were returned by the Project Lead, Green Belt or both in the time frame required.The survey validation was tested with the Cronbach alpha coefficient, with an overall result of 0.8976.The Cronbach alpha coefficient value for each CSF was also larger than 0.89 for all CSFs, indicating that the questionnaire is reliable.Using the Likert scale described above, participants entered a value for each set of questions per CSF.The questions were designed to gauge the CSF level the survey participants observed within their organization.
The scores for each CSF were averaged and divided by 5, the maximum Likert value, to derive a percentage ranking.Table 6 illustrates the survey results from one project.In this particular organization, it is observed that business strategy that focuses on customer demand (CSF3) and a willingness to invest in resources (CSF4) are high, however top management commitment (CSF2) is below 70%, which could possibly explain why project execution and follow up of results is below 50%.If top management is not fully committed, then why should project leads expend resources on execution and follow up.The calculated levels of the CSF for each project will be used as the input values for the DEA model.• I can find relevant data in my company's database • All data from daily work is recorded • When data is needed but not available, I can ask the IT department or members of the Green

Belt project for assistance
• There are guidelines for data preservation that I must follow while working on my Six Sigma project 6. Attention given to both long and short term targets • My Green Belt project is related to my organization's long-term goals (2 or more years) • My Green Belt project is related to my organization's short-term goals (less than 1 year) • One of the foremost considerations in selecting a Green Belt project is whether it can be easily completed • My Green Belt project is related to my daily work • Following evaluation of a proposed project, if the project is considered beneficial, it will be carried out, even if more than a year is needed to complete it It is important to recognize that CSFs are qualitative factors.Qualitative factors may work differently from the quantitative factors in terms of their intensity and magnitude.For example, 4 in the Likert scale does not necessarily mean that its intensity or magnitude is higher than 2 by two times.Several authors have recently proposed diverse DEA models to overcome the limitation of the standard DEA with the qualitative factors.However, none of those can completely manage all issues regarding the qualitative factors.Hence, instead of using those DEA models, authors assumed that all Likert scales can be treated in a numerical sense.Cooper, Seiford and Shu (2011) stated that this is one approach and used by some marketing researchers.

Model verification
The Pearson correlation analysis was conducted on the data to determine if any of the inputs could have negative effect on the DEA model, and the results were shown in Figure 2 and Table 7.The analysis revealed CSF 3, Business Strategy based on customer demands and CSF 4, Investment of essential resources, have a negative correlation to one of the outputs.A decision was made not to include them in the final DEA model.Hence, the final total number of inputs and outputs became seven, and 18 samples (DMUs) are available.Therefore, it still satisfies the recommendation from Malhotra et al. (2009).
In terms of homogeneity of DMUs, all DMUs evaluated here were from a collection of Lean Six Sigma projects conducted at JSC using the same Kaizen or rapid improvement procedure.All the events addressed JSC service-oriented transactional business processes operating within a specific Directorate, Division or Branch.In a few cases, the processes were Center wide or spanned across Divisions, however, none went outside JSC.Similar resources and activities were required to operate within the processes to produce the end product(s).The teams worked under the same guidelines, requiring the generated solutions capable of implementation within a 3 to 6 month time frame.The solutions requiring a longer time frame to implement were put on hold for future evaluation.In-house Black Belts and Green Belts, similarly trained in NASA's Lean Six Sigma approach, facilitated the events.The combination of these characteristics sufficiently clusters the projects as comparable DMUs.Therefore, authors believe that all DMUs are homogeneous.

Input and Output Data
Table 8 displays the final raw input and output data from each LSS project.Each project was given a number and will be represented as such throughout the remaining paper.The organizational level of each CSF as determined by the surveys and the Total Team Hours attributed to the project can be seen across each row identified as an input.The last two columns, 'PSPI' and 'Cost Avoidance', are the calculated output values for each project.

Empirical Analysis
The final input and output variables in  In essence, DMUs with Score 1 were the most efficient in utilizing their input resources when compared to the other DMUs in the study.The reference set consists of four columns, and each column is comprised of the DMU's on the efficient frontier that an inefficient DMU is evaluated against and its associated lambda.The reference set for an inefficient DMU may differ from a DMU to a DMU.The lambda values are the weights given to each DMU in the reference set in order to make the inefficient DMU efficient.For example, the most inefficient project, DMU11, has a reference set of DMU's 12, 14, 16, and 18 with lambda values of 0.160, 0.133, 0.025, and 0.090, respectively.If DMU11 could attain the input and output values derived by adding the product of the lambda values of each input and output values of the DMUs in their reference set, it would be efficient.In this sense, each of DMUs, 12, 14, 16, and 18 is considered as an efficient peer for DMU 11.In addition, DMU11 is more comparable to DMU 12 based on its lambda value.
- From Table 9, useful managerial information can be calculated.The target values of the inputs and outputs that an inefficient DMU needs to attain to become an efficient DMU are easily calculated.These target values can be connected with any operational plan to improve the efficiency.In this respect, DEA is a powerful tool for managers.Senior managers often want to identify a group of projects that they can use as standards for future projects and a means to understand the importance of critical success factors on performance.Theoretically, all efficient DMUs can be a standard DMU.However, authors believe that practically DMUs that most frequently serve as an efficient peer can be a clearer target for this purpose.Hence, the frequency of DMU's service as an efficient peer is calculated from Table 9, and summarized in Table 10.Table 10.Frequency of service as an efficiency peer According to this efficiency peer analysis, DMUs 4, 16 and 18 serve most frequently as an efficient peer.This may indicate impact of efficient DMUs on the organization.Specifically, managers may narrow down their benchmarking focus on these three DMUs to improve any inefficient DMU's efficiency.While DEA does not provide any mechanism to compare efficient DMUs, the frequency of efficiency peer services may be used to evaluate the potential impact of those efficient DMUs.
study whether there is a difference in efficiency for projects grouped by NASA mission directives, Aeronautics Research, Exploration, Science, and Space Operations, Possibilities exist to utilize DEA to measure efficiencies of projects in combination with earned value management or how well a contractor is performing against their stated deliverables.
Further analysis would need to be conducted to determine the appropriate input and output variables to be used within the model.Once obtained, the analysis has potential to be used as a method to determine performance-based fees, areas to focus on for improvement and benchmarks for future projects or contracts to strive for.

Figure 1 .
Figure 1.Overall procedure The evidence favoring one over the other is of the highest possible validity.

•••••••
regularly report on my Green Belt project's progress • After my Green Belt project is completed, I am required to report the results of its implementation My manager or the Black Belt in charge comes to me and asks about the execution and progress of my Green Belt project • Black Belts submit yearly reviews of the results of Green Belt projects • If a Green Belt's project is behind schedule, he/she must submit a report explaining Top management periodically requests a report on my Green Belt project progress • I can ask for management assistance when I encounter problems with my Green Belt project • Top management requires organizations to carry out Green Belt projects • Top management requires that each organization regularly report on the progress of Green Belt projects Top management will assign organizations to carry out specific Green Belt projects 3My Green Belt project is related to customer demands • I know what kind of customer demands our department needs to satisfy • My organization regularly researches customer demands • My organization regularly reviews customer complaints • My organization's strategy is based on customers' demands and If I need manpower assistance to complete my Green Belt improvement project, my manager will help coordinate this • If items need to be purchased for my Green Belt project, I can request my manager's assistance in procuring them • My company will adjust my workload to ensure that I have sufficient time to complete my Green Belt project My company allows me to work overtime (credit or comp time) to complete my Green Belt project • My manager will help me obtain any resources needed for my Green Belt project 5The data needed for the analysis of my Green Belt project are easy to obtain Ho, Chang and Wang (2008)and Wang (2008)highlighted 13 CSFs out of Six Sigma, citing several authors who mentioned each CSF.The study includes a literature review period from 2000 to 2005.The 13 CSFs are listed in Table 1 below.Studies on critical success factors related to Six Sigma and LSS implementation conducted by Coronado and Anthony (Ho et al., 2008)itical Success Factors studies(Ho et al., 2008)With a growing demand for the measurement of LSS implementations, arises the overall goal of this research, to develop a LSS-PPEM based on the most influential CSFs exhibited by the organization.
Emrouznejad, Parker and Tavares (2008) provided a compilation of most of the references published in the field of DEA from 1978 through 2007.The article demonstrated the growth of DEA as an accepted operational tool in a diverse set of fields.Banking, education (including higher education), health care, and hospital efficiency were found to be the most popular application areas.In total, 2500 distinct authors have been identified, with the average number of authors per publication being nearly two.Based on approximately 75% of the database, about 2000 distinct keywords were found, with an average of 3.5 per publication as seen in Table2.List of the most popular keywords by number or publication Eliat, Golany and Shtub (2008)2007)nd Management -http://dx.doi.org/10.3926/jiem.582 to evaluate the performance of engineering design projects.They addressed that DEA could fill a gap not addressed by common project evaluation by comparing a project to its peers with similar characteristics.Linton, Morabito and Yeomans (2007)applied DEA to assess, analyze, and prioritize R&D project.Eliat, Golany and Shtub (2008)embedded the balanced score card in DEA to evaluate the R&D project.All authors mentioned above used a project as a DMU in DEA.

Table 4 .
Final CSF ranking

Table 5 .
Survey questions for evaluating CSF level

Total Team Hours Matrix Plot of Outputs vs Inputs
*Negative correlation between input and output

Table 7 .
Correlation coefficient Table 8 are solved by the software DEA-Solver by

Table 8 .
Data for DEA