Integration of simulation and DEA to determine the most efficient patient appointment scheduling model for a specific healthcare setting

Purpose: This study is to develop a systematic approach for determining the most efficient patient appointment scheduling (PAS) model for a specific healthcare setting with its characteristics of multiple appointments requests in order to increase patients’ accessibility, improve resource utilization, and reduce operation cost. In this study, three general appointment scheduling models, centralized scheduling model (CSM), decentralized scheduling model (DSM), and hybrid scheduling model (HSM), are considered. Design/methodology/approach: This study integrates discrete event simulation and data envelopment analysis (DEA) to determine the most efficient PAS model. Simulation analysis is used to obtain the outputs of different configurations of PAS, and the DEA based on the simulation outputs is applied to select the best configuration in the presence of multiple and contrary performance measures. The best PAS configuration provides an optimal balance between patient satisfaction, schedulers’ utilization, and the cost of the scheduling system and schedulers’ training. Findings: The case study shows that in the presence of high proportion (more than 70%) of requests for multiple appointments, CSM is the best PAS model. If the proportion of requests for multiple appointments is medium (25%-50%), HSM is the best. Finally, if the proportion of requests for multiple appointments is low (less than 15%), DSM is the best. If the proportion is


Introduction
Many outpatient clinics are facing the high pressure of reducing the cost and increasing the patient satisfaction.Patient appointment scheduling (PAS) has a significant impact on the clinic's operation.An efficient PAS model is critical to achieving high accessibility, high resource utilization, and low cost (Gupta & Denton, 2008; Hooten & US ARMY Academy of Health Science, 1990).
Scheduling appointments in outpatient clinics is the process of assigning clinics' timeslots to incoming requests (Guo, Wagner & West, 2004).Patients obtain appointments through a PAS system, which operates, based on the scheduling model.There are three major types of scheduling models: centralize scheduling model (CSM), decentralized scheduling model (DSM), and hybrid scheduling model (HSM).The performance of the PAS system is impacted by the interaction of the scheduling model used and the characteristics of the healthcare setting.The characteristics of the healthcare setting include a set of the attributes in each clinic, such as the arrival rate of demand and the percentage of patients with multiple appointments.If the PAS model is designed by not considering the characteristics of the healthcare setting, the PAS system would operate inefficiently (Nealon & Moreno, 2003).An efficient PAS system has the ability to provide a balance between the three indicators: cost, accessibility, and resource utilization.
The general objective of this research is to reduce the average indirect waiting time (the time before connecting to a scheduler for requesting an appointment), average holding time (the sum of the indirect waiting time and the duration that a patient talk to a scheduler), and the cost related to scheduling system and training, as well as increasing schedulers' utilization through providing a generic framework for determining the most efficient PAS model for a specific clinic setting based on the characteristics of the clinic setting.
-786-Journal of Industrial Engineering and Management -http://dx.doi.org/10.3926/jiem.1058 The methodology used in this study is the integration of simulation and Data Envelopment Analysis (DEA) approaches to select the most efficient PAS model for a specific healthcare setting, as well as analyze the inefficient scheduling models, and identify parameters that may be altered to improve the overall system efficiency.Since it is not practical to apply different PAS models in the same healthcare setting to evaluate their performances due to the high interruption cost, this study applies simulation approach to obtain performance measures of different PAS models' configurations that are needed for DEA outputs.Then DEA is applied to compare different scheduling models and select the most efficient scheduling model.
Comparing the efficiency of scheduling models that have different attributes is a hard task.
However DEA is a perfect tool which has the ability to compare the efficiency (or productivity) across different configurations (Avikran, 2001).According to our knowledge, this is the first research in the domain of healthcare operations that provides a generic framework to integrate simulation and DEA to find the most efficient PAS model for a specific clinic setting.
The structure of this study is as follow: Section 2 presents a literature review for the PAS system and DEA.Section 3 describes the problem statement as well as the framework structure to determine an efficient PAS model for a specific clinic setting.Section 4 presents the proposed methodology.Section 5 provides empirical examples in three different clinic settings to validate the usability of the proposed procedure.Finally, the conclusions for the study and other useful methodology that can be integrated with DEA in future research (for evaluating the efficiency of PAS model) are proposed in Section 6.

Literature Review
There are three major PAS models in terms of assigning timeslot(s) to the requested single or multiple appointments: CSM, DSM, and HSM (Zhang, Dharmadhikari & Song, 2009).
In CSM, a patient obtains appointments for multiple clinics or single clinic through one request, such as a phone call.Any scheduler can schedule appointments for all the clinics.The schedulers can locate in one central scheduling department or locate in different clinics or locations.In DSM, schedulers can only schedule patient appointments for one predefined clinic.
If a patient has to request for multiple appointments, he or she will make multiple phone calls to obtain all the appointments.HSM is the combination of DSM and CSM.In HSM, some clinics operate under CSM and the rest of the clinics operate under DSM (Zhang, Gonela & Aslani, 2011;Berry & Phanthasomchit, 2000).
There are advantages and disadvantage for different PAS models.The advantages of CSM are: 1) patients with multiple appointments are required to contact only one scheduler; 2) the available capacity is used efficiently if clinics interact at medium or high level; and 3) it leads to shorter total processing time and waiting time.In general, CSM provides greater uniformity in how appointments are handled and better ability to monitor the entire process.The disadvantages of CSM are the needs for advanced information system and high skilled schedulers.
The advantages of DSM are: 1) the available capacity of the specific clinic is used efficiently, since the schedulers are familiar with the clinic resource; and 2) schedulers can schedule the clinic with the lowest cost.Implementation of DSM is useful in settings with a small number of clinics (Zhang, Gonela & Aslani, 2010;Zhang, Gonela & Aslani, 2011).However, if the number of patients with multiple appointments increases, the DSM might cause higher communication cost and conflicts among different clinics.
HSM takes the advantages of CSM and DSM.There is no need to have advanced information system and high skill level schedulers in HSM.When high interaction level requires between clinics and there is unavailability of advanced scheduling software, HSM is selected instead of CSM (Zhang, Gonela, & Aslani, 2010;Zhang, Gonela, & Aslani, 2011).The disadvantages of HSM are: sometimes it is required to contact more than one scheduler to schedule some of patients with multiple appointments due to the communication gap between the clinics with CSM and the clinics with DSM.
An inefficient PAS model would impose either high cost or lower patient dissatisfaction under certain healthcare settings.For example, if the healthcare setting is involved high volume of patients with multiple appointments and the PAS model is DSM, patients have to make multiple phone calls to schedule all the appointments.This will cause longer waiting time for the patients to get their appointments (each phone call might cause waiting time) and possible conflicts for the appointments.
In each PAS model (e.g., CSM), there are many configurations.Determining the most efficient appointment scheduling model between different generated configurations is complicated from general point of view.Firstly, different configurations cannot be easily compared because each configuration has its own attributes.Secondly, the objective function includes contrary performance measures.As a result, DEA is selected as the methodology to compare different configurations and select the most efficient configuration as well as analyzing the inefficient configurations.
DEA requires inputs and outputs for difference configurations of PAS model.Health care settings are considered as complex systems because of the existence of high interaction between different departments of a settings as well as the presence of uncertainty such as demand and service time uncertainty.Computer simulation is an effective tool to discover the different outputs for different setting (Shim & Kumar, 2010).There are four major methods for creating the simulation model of a healthcare setting which are, Discrete Event Simulation (DES), system dynamics (SD), Monte Carlo Simulation (MCS) and Agent Based Simulation (ABS) (Mustafee, Katsaliaki & Taylor, 2010).DES is a perfect choice for making improvement at the operational level and suitable to use in studying healthcare systems (Peng, Niu & ElMekkawy, 2013;Jacobson, Hall & Swisher, 2013).Therefore, this study uses DES to obtain outputs of different configurations of PAS models.
DEA is an approach for evaluating the relative efficiency of either different organizations or different units in one organization.DEA has the capability of discriminating between multiple efficient units or organizations and determining the most efficient unit or organization (Yang & Kuo, 2003;Avikran, 2001).Additionally, DEA considers multiple inputs and outputs at the same time in the presence of complex relationships between inputs and outputs.This makes it different from several other efficiency approaches.
There are some studies in manufacturing industry which use DEA as a decision tool to select the most efficient staff allocation, staff training, and technology selection, such as the study for optimal operator allocation in cellular manufacturing (Ertay & Ruan, 2005).
There are some studies in healthcare area that integrate simulation and DEA to find out the most relative efficient system.Weng, Tsai, Wang, Chang and Gotcher (2011) apply the integration of DEA and simulation to find out the most relative efficient combination of nurses and physicians in emergency department through considering the utilization of nurses and physicians as well as patient waiting time as the performance measures.Al-Refaie, Fouad, Li and Shurrab (2014) apply the integration of simulation and DEA in the emergency department to find out the best distribution of nurses in order to increase utilization and decrease patient waiting time.
According to our knowledge, there are no studies in healthcare domain that apply analytical methods to determine the PAS model with the best allocation of schedulers, training level of schedulers, and complexity level of scheduling systems.In order to bridge the gap, this study proposes a method that integrates simulation and DEA to determine the most efficient PAS model for different healthcare settings.

Problem Statement
In order to increase patients' accessibility, resource utilization, and reduce cost in hospitals or other healthcare settings, a framework is proposed to determine the most efficient patient PAS system for a specific healthcare setting with multiple appointments requests.Multiple appointments requests are considered as one of the major problems in appointment scheduling models (Nealon & Moreno, 2003).
The efficiency of PAS system can be evaluated from different dimensions.One of these dimensions is based on PAS model and the context of the healthcare unit.Interaction index is -789-Journal of Industrial Engineering and Management -http://dx.doi.org/10.3926/jiem.1058defined as the ratio of patients requesting multiple appointments to the total patients requesting appointments.This index is used to classify healthcare settings.
Efficiency is evaluated through three categories of indicators relevant to PAS systems: • Satisfaction indicator for patients, staffs, and managers

Proposed Methodology
First, different configurations based on the major PAS models (CSM, DSM, and HSM) are generated.Then, simulation models are built to obtain the outputs for the DEA model.The simulation outputs for the DEA model are staffs' average utilization, patients' average waiting time, and holding time in the system.Finally, the DEA model is implemented to determine the most efficient configuration from the generated configurations, and the reasons of inefficiency of other configurations.
The DEA inputs are the cost of scheduling software, the training cost, and the number of schedulers.The two cost-based inputs are obtained from the hospital administrator and vendors, and the number of schedulers is obtained from the queuing theory formula for the subjective service level (Agnihotri & Taylor, 1991).
The DEA outputs are obtained from simulation study.The simulation inputs are obtained in the data collection stage.The simulation inputs include percentage of multiple appointments versus single appointments, percentage of different types of multiple appointments, the pattern of request arrival for different number of clinics, the scheduling time pattern for the schedulers with different skill level, the service level and the required number of the schedulers for different configuration.
The percentage of multiple appointments is obtained from the available historical data in the database.The clinics' interaction index is the ratio of total number of multiple appointments over total number of appointments.
If the interaction index is less than 15 percent, the interaction level in the setting is low.If the interaction index is from 25 percent to 50 percent, the interaction level in the setting is medium.Finally, if the interaction index is more than 70 percent the interaction level in the setting is high.The interval from 15 percent to 25 percent is the threshold interval and the setting can be considered as either low or medium interaction level.In addition, the interval from 50 percent to 70 percent is the threshold interval and the setting can be considered as either medium or high interaction level.

Configuration generation
Different configurations for the three major PAS models (CSM, DSM, and HSM) are generated to provide sufficient number of alternatives.Sufficient number of alternatives should be evaluated to identify the best configuration of PAS model for a healthcare setting.

Configurations of CSM
Two configurations for CSM are generated in this study: CSM (1) and CSM (2).In CSM (1), the schedulers are single task.They only conduct scheduling task and are centralized in the scheduling department.In CSM (2), the schedulers are multi-task.They conduct scheduling task and other tasks, such as checking in patients.They are located in individual clinics.CSM (2) is implemented in situations where no space is available for centralized scheduling department.The schedulers in CSM (2) are multitasking for providing high scheduler utilization (Ertay & Ruan, 2005).

Configurations of DSM
Two configurations for DSM are generated: DSM (1) and DSM (2).In DSM (1), schedulers are single task because of the high volume of activities for both scheduling and check-in tasks.In DSM (2), schedulers are multitasking because the scheduling and check-in tasks are not overlapped and schedulers' multitasking provides higher resource utilization (Ertay & Ruan, 2005).

Configurations of HSM
HSM configurations are obtained by defining specific number of clusters for each configuration.
Each cluster would include single or multiple clinics.

Simulation study
The followings are the seven steps to conduct the simulation study.
1. Identify feasible number of requested multiple appointments and their probability.7. Run the simulation model and obtain the numerical value for the defined outputs.

DEA approach
DEA approach is applied to identify the most efficient configuration as well as the reason for inefficient configurations.The general guideline for the applied DEA approach in this study is illustrated in Figure 1.

Inefficiency term
General definition Interpretation in this study Applied Model

Technical Inefficiency
The configuration does not use the available capacity efficiently (Venkatesh, 2006;Ozcan, 2008) Performance of scheduling software and schedulers do not match their capability.

Scale Inefficiency
The configuration does not use the proper technology with clinic setting's inputs (Venkatesh, 2006;Ozcan, 2008) The clinic setting requires more complex scheduling software and higher skill level.

Mix Inefficiency
There exists extra in inputs and shortfall in outputs Waiting time and handle time should be decreased and utilization rate should be increased.

CCR output-oriented with slack
Table 1.Different terms of inefficiency

Analyze inefficient configurations
In this study, three terms of inefficiency are evaluated which are technical inefficiency, scale inefficiency, and mix inefficiency (refer to Table 1 for the definitions).The following illustrates the solutions for the three different inefficiency terms.

Technical inefficiency
The models that are introduced by Charnes-Cooper-Rhodes (CCR) and Banker-Charnes-Cooper (BCC) are the major models in DEA (Cooper, Seiford & Tone, 2006).The main difference between CCR and BCC models is that BCC efficiency score does not consider scale efficiency.
As a result BCC efficiency score illustrates pure technical efficiency.
In this study, BCC model is selected to calculate technical efficiency.BCC model can be inputoriented or output-oriented.BCC output-oriented is selected in this study because the inputs (scheduling software cost, training cost and number of staffs) are constant but the outputs can be changed.Equations ( 1)-( 5) are the formula of BCC output-oriented model (Cooper, Seiford & Tone, 2006).5) is non-negative constraint.If ηB is equal to one, the configuration is technically efficient, otherwise the configuration would be inefficient.The benchmark index for the efficient configuration is equal to 1, and zero for the inefficient configuration.The reference set for inefficient configurations, is the configuration with benchmark index greater than zero.

Scale inefficiency
In the first place, the CCR output-oriented model run, and its efficiency score is compared with the obtained BCC efficiency scores for technical inefficiency.If CCR model efficiency score is not equal to the BCC efficiency scores, the configuration has scale inefficiency.In order to calculate scale inefficiency, the fraction is calculated.
Max ηCCR Objective function, Equation ( 6), is to maximize ηCCR for increasing yr0 to ηCCRyr0 without increasing xi0.Equation ( 7) and Equation (8) confirm that the improved decision unit is located in the feasible region.Equation ( 9) is the benchmark for inefficient decision units.The evaluated decision unit with the benchmark index equal to 1 and consequently ηCCR equal to 1 is efficient.The evaluated decision unit with ηCCR greater than 1 is inefficient and the benchmarks for this inefficient decision unit are the decision units with λj greater than zero.

Mix inefficiency
One of the properties of DEA is its capability to calculate the modification that should be done in the inputs and outputs of the inefficient decision units to make them efficient.Cooper, Seiford and Tone (2006) presents the following two-phase formula to calculate mix-inefficiency.Objective function, Equation ( 14), is to maximize the sum of input excess and output shortfall which is mix inefficiency.Equation ( 15) is the constraint for input excess and Equation ( 16) is the constraint for output shortfall.Equations ( 17)-( 19) are non-negative constraints.If w * ≠ 0, it means S -* ≠ 0 or (and) S +* ≠ 0. This indicates there is mix inefficiency.Table 1 summaries the definitions for the three terms of efficiency and the models used.

Select the most efficient configuration
The configuration is identified as efficient if the configuration does not have technical inefficiency, scale inefficiency, and mix inefficiency.It is highly possible that the DEA model identifies multiple configurations as efficient.Unrealistic weight distribution is one of the reasons for multiple efficient configurations (Li & Reeves, 1999).Unrealistic weight distribution is defined as assigning an unrealistic too large weight to a single outputs or/and assigning a too small weight to single input to make the configuration relatively efficient (Li & Reeves, 1999).Li and Reeves (1999), proposed a minimax approach in their study to overcome the problem of unrealistic weight distirbution which is formulated in Equations ( 20)-( 24).
Mix M (20) Subject to (21) Equation ( 20) minimizes M, which is the maximum of all deviation variables.Equation ( 21) is the constraint to linearize the fractional objective function of basic DEA.Equation ( 22) makes the efficiency score be less than one and dj is the deviation variable for each configuration.
Equation ( 23) ensures M be the maximum of all the deviations.Equation ( 24) is non-negative constraint.The efficiency score is calculated as h0 = 1 -d0 and the configuration is minimax efficient if and only if the value d0 that minimizes M is zero (Ertay, Ruan & Tuzkaya, 2006).
-798-Journal of Industrial Engineering and Management -http://dx.doi.org/10.3926/jiem.1058 Minimax efficiency approach discriminates efficient configurations from inefficient configurations more realistically compare to the classic DEA approach.If the origin of inefficiency is not required to be identified, minimax efficiency approach is only applied to select efficient configurations.
The selected efficient configurations through both minimax efficiency approach and classic DEA approach have the deviation variable equal to zero.The minimax efficient configurations have an additional constraint to minimize the maximum of all the deviation variables of other configurations.The additional constraint makes the selected minimax efficient configurations more realistic.
The objective function of minimax efficiency approach (Equation ( 14)) can be modified to "M -kd0".Firstly, the minimax efficiency approach is applied with the coefficient k equal to zero.If more than one configuration is selected as efficient, k ranges between (0, 1) and is determined by trial and error.The number of trials to find the most efficient configuration is limited and the trial that first provides only one configuration as the most efficient is selected.
The modified minimax efficiency score is less than the original minimax efficiency score, so that the modified minimax efficiency approach can identify the most efficient configuration from multiple efficient configurations (Ertay, Ruan & Tuzkaya, 2006).
The reason for selecting only one configuration as the most efficient one is to provide a benchmark.Benchmark configuration is defined as the superior configuration that provides robustness to demand variability and the tradeoff between indicators of efficiency.In case, the hospital is not able to change its current configuration, the benchmark leads to continuous improvement of the implemented configuration (Lai, Huang, & Wang, 2010).

Case Study
In this section, case study is conducted to illustrate the effectiveness of the proposed methodology.

Healthcare settings
The setting in the case study includes two primary care clinics, three specialty clinics, and one laboratory.Demand pattern is defined in terms of arrival rate of requested appointments, appointments' type, and percentage of multiple appointments versus single appointment for the combination of clinics. -799-

Configuration generation
Different configurations of the three major PAS models of CSM, DSM, and HSM are generated.
They are considered as decision units for DEA models.The configurations in this study include: two CSMs, two DSMs, and six HSMs.The first CSM includes a call center and the schedulers are single task.The second CSM, does not include a call center and the schedulers are multitasking.As a result for the second CSM, the utilization of the schedulers is multiplied by a coefficient to obtain the correct value for utilization of staff for performing scheduling task.The third and forth configurations are DSM.Six configurations for HSM are also developed that assign clinics to different clusters and each cluster has its own scheduler.All configurations are summarized in Table 2 and Table 3.

Simulation outputs
The simulation output values for the setting with high interaction level, medium interaction level and low interaction level are illustrated in Table 4 Table 5 and Table 6.

Validation/Verification
Model validation is measured through evaluating how accurate the created model depicts the real system Sensitivity analysis is applied to validate the simulation model through changing the inter-arrival between appointment requests and proportion of multiple appointments.
The inter-arrival time is changed from 1 to 3.5 and the behavior of average waiting time before connecting to a scheduler as well as average holding time are captured.Figure 2 and Figure 3 show the obtained waiting time and holding time by the simulation model.As it is observed, both waiting time and holding time are decreased by increasing the inter-arrival time.This matches the expectation.

Analyze inefficient configurations
The DEA inputs and outputs value for the setting with high interaction level are shown in Table 7 and Table 8.Table 9 presents the results from analyzing different terms of inefficiency for the clinic setting with high interaction level.The presence of efficiency is showed by star.Table 7. DEA inputs for the setting with high interaction

Decision
Equations ( 1)-( 5) are applied to evaluate technical efficiency.Configurations 5 and 7 are technically inefficient.This indicates the scheduling software and the schedulers do not use their available capability.
Equations ( 6)-( 9) are applied to evaluate scale efficiency.Scale inefficiency in the configurations 3, 5, 6, 7 and 9 shows that the complexity of scheduling software and schedulers' skill level in DSM and HSM are not sufficient and improvement is needed in terms of scheduling software and schedulers' skills.Finally, Equations ( 10)-( 19) are applied to evaluate mix inefficiency.Mix inefficiency in configuration 3, 5, 6, 7 and 9 is an evidence for the presence of extra inputs and outputs' shortfalls.
The DEA inputs and outputs value for the setting with medium interaction level are illustrated in Table 10 and Table 11.Table 12 presents the results from analyzing different terms of inefficiency for the clinic setting with medium interaction level.Finally, Equations ( 10)-( 19) are applied to evaluate mix inefficiency.Mix inefficiency in configurations 3, 4, 5, 6, 7 and 9 is an evidence for the presence of extra inputs and output shortfalls.The DEA inputs and outputs value for the setting with low interaction level are illustrated in Table 13 and Table 14.Equations ( 6)-( 9) are applied to evaluate scale efficiency.10 runs and 10 comparisons of the model are performed to identify scale efficient configuration(s).Scale inefficiency in the configurations 1, 5, and 9 shows the complexity of scheduling software and schedulers' skill level in these two configurations are not sufficient and improvement should be applied in the scheduling software and schedulers' skills.

Decision
Finally, Equations ( 14)-( 19) are applied to evaluate mix inefficiency.The mix inefficient configurations are identified through 20 runs of model.Mix inefficiency in configurations 5, 6, 7 and 9 is an evidence for the presence of extra inputs and outputs' shortfalls.

Clinic setting with high interaction level
The configuration is identified as efficient if it is efficient in terms of technical, scale and mix.
20 comparisons are done between terms of efficiency.Table 9 shows that the efficient configurations for clinic setting with high interaction level are: DU1, DU2, DU8, and DU10.

Clinic setting with medium interaction level
The efficient configurations for clinic setting with medium interaction level are: DU1, DU2, DU8, and DU10.

Conclusion
This study focuses on developing a framework, which integrates simulation study with DEA approach, to determine the most efficient PAS model and its configuration for a specific healthcare setting.The efficiency is determined based on three terms of: technical efficiency, scale efficiency, and mix efficiency.Technical efficiency presents scheduling software and schedulers' performances match their capability.Scale efficiency shows complexity of the scheduling software and skill level of schedulers match the clinic setting context.Finally, the mix inefficiency indicates the presence of extra inputs and outputs' shortfalls.The configuration is efficient if it has all the three terms of efficiency simultaneously.
A case study for different types of healthcare settings is conducted to illustrate the effectiveness of the proposed method.The case study shows, the most efficient PAS For the clinic setting with high interaction level is the CSM.For the healthcare setting with medium interaction level is the HSM.For the clinic setting with low interaction level is the DSM (the schedulers are multitasking).The most efficient scheduling models for the three clinic settings in the case study are selected in terms of the balance between patient satisfaction, schedulers' utilization and scheduling system cost.
The developed framework in this study can be used for any real world healthcare setting.
When different healthcare setting is studied, new simulation models should be developed to obtain the DEA outputs.The most efficient configuration for the new setting is determined through the developed DEA approach in this study.
For the future work, further simulation study will be conducted to determine the mutual impact of scheduling model and clinic flows.Queuing theory will be applied to determine the required resource for different configuration.Additional inputs and outputs will be defined to apply in the DEA approach to select the most efficient PAS model.
Resource utilization indictor for schedulers • Cost indicator for both complexity level of scheduling software and skill level of schedulers One or multiple factors will be assigned to each category of indicators for evaluating the efficiency.Accessibility is defined as a factor for patient satisfaction.In this study, accessibility is measured by average waiting time before connecting to a scheduler and average call duration.The utilization of scheduler and the number of schedulers are the factors for resource utilization indicator.Complexity level of scheduling software and skill level of schedulers are the factors to cost indicator.A framework is proposed to integrate simulation and DEA to determine the most efficient PAS model for a specific healthcare setting.The proposed methodology compares different configurations of different PAS models based on the three categories of efficiency indicators.
2. Identify the probability of specific requested combination of clinics for all the feasible multiple appointment.3.Collect simulation model inputs.4.Build the model.5. Determine simulation outputs.6.Apply termination conditions.

r
Index number of outputs r = 1,2,…,s j Index number of decision units j = 1,2,..,N ηCCR = Efficiency Score yrj = Quantity of output r for the decision unit j yr0 = Quantity of output r for the evaluated decision unit xij = Quantity of input i for the decision unit j xi0 = Quantity of input i for the evaluated decision unit λj = The benchmark index for the decision unit j

j
Index number of decision units j = 1,2,..,N ηCCR = Efficiency Score yrj = Quantity of output r for thedecision unit j yr0 = Quantity sdof output r for the evaluated decision unit xij = Quantity of input i for the decision unit j xi0 = Quantity of input i for the evaluated decision unit λj = The benchmark index for the decision unit j = Excess in input i = Shortfall in output r Equations (10)-(13) are the same as Equations (1)-(5).The only difference is that in Equation (11) and Equation (12) the inequality constraints are changed to quality constraints by adding variables for input excess and output shortfall.In phase one, ηCCR is calculated to be substitud in phase two for obtaining mix inefficiency which is the sum of S -* and S +* .ηCCR = Maximum technical Efficiency Score from Phase one

Figure 2 .
Figure 2. The impact of request arrival on waiting time

Figure 4 .Figure 5 .
Figure 4. Impact of proportion of multiple appointments on waiting time

Table 2 .
Journal of Industrial Engineering and Management -http://dx.doi.org/10.3926/jiem.1058Twotypes of appointments, which are appointments through phone calls and appointments requested by providers, are considered.It is assumed that appointments requested by provider transferred to the related scheduler through phone call.Configuration generation for the major scheduling models.The mean time between arrivals of appointments is 2 minutes and follows exponential distribution.Service time follows triangular distribution.The distributions for service time and arrival of appointment request are obtained from analyzing the historical data.

Table 4 .
Simulation outputs for the setting with high interaction

Table 5 .
Simulation outputs for the setting with medium interaction

Table 6 .
Simulation outputs for the setting with low interaction

Table 9 .
Inefficiency terms for the setting with high interaction level

Table 10 .
DEA inputs for the setting with medium interaction

Table 12 .
Inefficiency terms for the setting with medium interaction

Table 13 .
DEA inputs for the setting with low interaction

Table 15 .
Table 15 presents the results from analyzing different terms of inefficiency for the clinic setting with low interaction level.Inefficiency terms for the setting with low interaction