Statistical Analysis of Output from Terminating Simulations Chapter 6 Last revision December 17, 2006 Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 1 of 31 What Well Do ... Time frame of simulations Strategy for data collection and analysis Confidence intervals Comparing two scenarios Comparing many scenarios via the Arena Process Analyzer (PAN) Searching for an optimal scenario with OptQuest Simulation with Arena, 4th ed.

Chapter 6 Stat. Output Analysis Terminating Simulations Slide 2 of 31 Motivation Random input leads to random output (RIRO) Run a simulation (once) what does it mean? Need statistical analysis of output data Was this run typical or not? Variability from run to run (of the same model)? From a single model configuration Compare two or more different configurations Search for an optimal configuration Statistical analysis of output is often ignored This is a big mistake no idea of precision of results

Not hard or time-consuming to do this it just takes a little planning and thought, then some (cheap) computer time Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 3 of 31 Time Frame of Simulations Terminating: Specific starting, stopping conditions Run length will be well-defined (and finite) Steady-state: Long-run (technically forever) Theoretically, initial conditions dont matter

But practically, they usually do Not clear how to terminate a run This is really a question of intent of the study Has major impact on how output analysis is done Sometimes its not clear which is appropriate Here: Terminating (steady-state in Section 7.2) Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 4 of 31 Strategy for Data Collection and Analysis For terminating case, make IID replications Run > Setup > Replication Parameters: Number of Replications field Check both boxes for Initialize Between Replications Separate results for each replication Category by Replication report Model 5-3, but for 10 replications (= Model 6-1) Note

cross-replication variability Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 5 of 31 Strategy for Data Collection and Analysis (contd.) Category Overview report has some statistical analysis results of output across replications How many replications? Trial and error (now) Approximate number for acceptable precision (below) Sequential sampling (Chapter 12) Turn off animation altogether for max speed Run > Run Control > Batch Run (No Animation) Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations

Slide 6 of 31 Confidence Intervals for Terminating Systems Using formulas in Chapter 2, viewing the crossreplication summary outputs as the basic data: Possibly most useful part: 95% confidence interval on expected values This information (except standard deviation) is in Category Overview report If > 1 replication, Arena uses cross-repl. data as above Other confidence levels, graphics Output Analyzer Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 7 of 31 Half Width, Number of Replications Prefer smaller confidence intervals precision Notation:

Confidence interval: Want this to be small, say < h where h is prespecified Half-width = Cant control t or s Must increase n how much? Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 8 of 31 Half Width, Number of Replications (contd.) Set half-width = h, solve for Not really solved for n (t, s depend on n) Approximation: Replace t by z, corresponding normal critical value Pretend that current s will hold for larger samples

s = sample standard Get deviation from initial number n0 of replications Easier but different approximation: h0 = half width from initial number n0 of replications Simulation with Arena, 4th ed. n grows quadratically as h decreases Chapter 6 Stat. Output Analysis Terminating Simulations Slide 9 of 31 Half Width, Number of Replications (contd.) Application to Model 6-1 From initial 10 replications, 95% half-width on Daily Profit was 1605 (7.4% of X = 21,548)

First formula: n 1.962(2243.382/5002) = 77.3, so 78 Second formula: n 10(16052/5002) = 103.0, so 103 Modified Model 6-1 into Model 6-2 Lets get this down to 500 or less Checked Run > Run Control > Batch Run (No Animation) for speed In Run > Setup > Replication Parameters, changed Number of Replications to 110 (conservative based on above) Got 2241.71 413.52, satisfying criterion (overshot a bit?) BTW, from 110 replications got 11.65 0.53 on Percent Rejected Use max of sample sizes for precisions on multiple outputs Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 10 of 31 Interpretation of Confidence Intervals Interval with random (data-dependent) endpoints

thats supposed to have stated probability of containing, or covering, the expected valued Target expected value is a fixed, but unknown, number Expected value = average of infinite number of replications Not an interval that contains, say, 95% of the data Thats a prediction interval useful too, but different Usual formulas assume normally-distributed data Never true in simulation Might be approximately true if output is an average, rather than an extreme Central limit theorem Robustness, coverage, precision see text (Model 6-3) Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 11 of 31 Comparing Two Scenarios

Usually compare alternative system scenarios, configurations, layouts, sensitivity analysis For now, just two scenarios ... more later Model 6-4 Model 6-3, except reduce to 110 replications, add file Total Cost.dat to Statistic module, Output column, Total Cost row Similarly for percent rejected Saves output statistics to these files for each replication Two scenarios Base case all inputs as original Model 5-3, no extra resources More-resources case Add 3 trunk lines (29), 3 each of New Sales, New Tech 1, New Tech 2, New Tech 3, and New Tech All Effect on total cost, percent rejected? Simulation with Arena, 4th ed.

Chapter 6 Stat. Output Analysis Terminating Simulations Slide 12 of 31 Comparing Two Scenarios (contd.) Reasonable but not-quite-right idea Make confidence intervals on expected outputs from each scenario, see if they overlap; look at Total Cost Base case: 22241.71 413.52, or [21828.12, 22655.16] No overlap More-resources case: 24560.12 315.93, or [24244.19, 24876.05] But this doesnt allow for a precise, efficient statistical conclusion Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 13 of 31

Compare Means via Output Analyzer (contd.) Output Analyzer is a separate application that operates on .dat files produced by Arena Launch separately from Windows, not from Arena To save output values (Expressions) of entries in Statistic data module (Type = Output) enter filename.dat in Output File column Did for both Total Cost and Percent Rejected Will overwrite these file names next time Either change the names here or out in the operating system before the next run .dat files are binary can only be read by Output Analyzer Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations

Slide 14 of 31 Compare Means via Output Analyzer (contd.) Start Output Analyzer, open a new data group Basically, a list of .dat files of current interest Can save data group for later use .dgr file extension Add button to select (Open) .dat files for the data group Analyze > Compare Means menu option Add data files A and B for the two scenarios Select Lumped for Replications field Title, confidence level, accept Paired-t Test, do not Scale Display since two output performance measures have different units Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations

Slide 15 of 31 Compare Means via Output Analyzer (contd.) Results: Confidence intervals on differences both miss 0 Conclude that there is a (statistically) significant difference on both output performance measures Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 16 of 31 Evaluating Many Scenarios with the Process Analyzer (PAN) With (many) more than two scenarios to compare, two problems are

Simple mechanics of making many parameter changes, making many runs, keeping track of many output files Statistical methods for drawing reliable, useful conclusions Process Analyzer (PAN) addresses these PAN operates on program (.p) files produced when .doe file is run (or just checked) Start PAN from Arena (Tools > Process Analyzer) or via Windows PAN runs on its own, separate from Arena Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 17 of 31 PAN Scenarios A scenario in PAN is a combination of: A program (.p) file Set of input controls that you choose

Set of output responses that you choose Chosen from automatic Arena outputs or your own Variables Values initially empty to be filled in after run(s) To create a new scenario in PAN, double-click where indicated, get Scenario Properties dialog Chosen from Variables and Resource capacities think ahead You fill in specific numerical values Specify Name, Tool Tip Text, .p file, controls, responses Values of controls initially as in the model, but you can change them in PAN this is the real utility of PAN Duplicate (right-click, Duplicate) scenarios, then edit for a new one Think of a scenario as a row Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 18 of 31

PAN Projects and Runs A project in PAN is a collection of scenarios Program files can be the same .p file, or .p files from different model .doe files Controls, responses can be the same or differ across scenarios in a project usually will be mostly the same Think of a project as a collection of scenario rows a table Can save as a PAN (.pan extension) file Select scenarios in project to run (maybe all) PAN runs selected models with specified controls PAN fills in output-response values in table Equivalent to setting up, running them all by hand but much easier, faster, less error-prone Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations

Slide 19 of 31 Model 6-5 for PAN Experiments Same as Model 6-4 but remove Output File entries in Statistic module Start PAN, New project, double-click for scenario Name = Base Case Program File = Model 06-05.p (maybe with path) Six controls all data type Integer PAN will keep track of outputs itself, so this is faster Stick with 110 replications Resources > capacity of Trunk Line User Specified > New Tech 1, New Tech 2, New Tech 3, New Tech All, New Sales

Responses both from User Specified Total Cost, Percent Rejected Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Could also do a designed experiment with PAN, for more efficient study of controls effects, interactions Slide 20 of 31 Model 6-5 for PAN Experiments (contd.) Experimental (non-base-case) scenarios Suppose you get $1360 more per week for more resources Must spend all $1360 on a single type of resource; could get 13 more trunk lines @ $98 each 4 more of any one of the single-product tech-support people @ $320 each 3 more of the all-product tech-support people @ $360 each

4 more sales people @ $340 each Create six more PAN scenarios Right-click, Duplicate Scenario(s), edit fields See the saved PAN file Experiment 06-05.pan Execute scenarios Select which to run (click on left, Ctrl-Click, Shift-Click) or Run > Go or F5 Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 21 of 31 Model 6-5 for PAN Experiments (contd.) What to make of all this? Statistical meaningfulness? Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 22 of 31

Statistical Comparisons with PAN Model 6-5 scenarios were made with 110 replications each Better than one replication, but what about statistical validity of comparisons, selection of the best? Select Total Cost column, Insert > Chart (or right-click on column, then Insert Chart) or Chart Type: Box and Whisker Next, Total Cost; Next defaults Next, Identify Best Scenarios Smaller is Better, Error Tolerance = 0 (not the default) Show Best Scenarios; Finish Repeat for Percent Rejected Simulation with Arena, 4th ed.

Chapter 6 Stat. Output Analysis Terminating Simulations Slide 23 of 31 Statistical Comparisons with PAN (contd.) Vertical boxes: 95% confidence intervals Red scenarios statistically significantly better than blues Numerical values (including c.i. half widths) in chart right click on chart, Chart Options, Data More precisely, red scenarios are 95% sure to contain the best one Narrow down red set more replications, or Error Tolerance > 0 More details in text So which scenario is best? Criteria disagree.

Combine them somehow? Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 24 of 31 Searching for an Optimal Scenario with OptQuest Scenarios considered via PAN are just a few of many Seek input controls minimizing Total Cost while keeping Percent Rejected 5 Explore all possibilities add resources in any combination New rules: 26 number of trunk lines 50 Total number of new employees of all five types 15 Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 25 of 31 Searching for an Optimal Scenario with OptQuest Formulation Formulate as an optimization problem:

Constraints on the input control (decision) variables Objective function is a simulation-model output Minimize Total Cost Subject to 26 MR(Trunk Line) 50 0 New Sales + New Tech 1 + New Tech 2 + New Tech 3 + New Tech All 15 Percent Rejected 5 Constraint on another output Reasonable start best acceptable scenario so far No PAN scenarios satisfied Percent Rejected 5, so start with more-resources case earlier (29 trunk lines, 3 new employees of each of five types) Where to go from here? Explore all of feasible sixdimensional space exhaustively? No. For this problem, choice (decision) variables are discrete, so can enumerate that there are 1,356,600 feasible scenarios with 110 replications per scenario, would take two months on 2.1GHz PC Simulation with Arena, 4th ed.

Chapter 6 Stat. Output Analysis Terminating Simulations Slide 26 of 31 Searching for an Optimal Scenario with OptQuest Operation OptQuest searches intelligently for an optimum Like PAN, OptQuest ... Unlike PAN, OptQuest ... runs as a separate application can be launched from Arena takes over the running of your model asks you to identify input controls, the output (just one) objective allows you to specify constraints on the input controls allows you to specify constraints on outputs decides itself what input-control-value combinations to try uses internal heuristic algorithms to decide how to change the input controls to move toward an optimum configuration

There are various stopping criteria for search Default is no significant improvement for 100 scenarios Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 27 of 31 Searching for an Optimal Scenario with OptQuest Example Model 6-6 for OptQuest Model 6-5, but OptQuest requires finite Replication Length Make sure Model 6-6 model window is active Make sure the desired model window is active Tools > OptQuest for Arena New Optimization or Browse for saved one (.opt) Tree on left, expand for Controls and Responses Simulation with Arena, 4th ed.

Chapter 6 Stat. Output Analysis Terminating Simulations Slide 28 of 31 Searching for an Optimal Scenario with OptQuest Controls, Responses Controls Resources Trunk Line Controls User Specified New Sales Integer, Lower Bound = 26, Suggested Value = 29, Upper Bound = 50 Integer, Lower Bound = 0, Suggested Value = 3, Upper Bound = 15 Similarly for others ... open Optimum Seeking 06-06.opt Click on Included to collect selections at top or bottom Responses User Specified Output Check Percent Rejected, Total Cost Simulation with Arena, 4th ed.

Chapter 6 Stat. Output Analysis Terminating Simulations Slide 29 of 31 Searching for an Optimal Scenario with OptQuest Constraints, Objective Constraints Objectives Add button, then each of first five controls, +, then <= 15 Add button, then Percent Rejected, then <= 5 Add button, Total Cost, Minimize radio button Options Stopping rules Tolerance for regarding results as equal Replications per simulation Solutions log file location

Stores all scenarios tried, results valuable for second best, etc. Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 30 of 31 Searching for an Optimal Scenario with OptQuest Running or Run > Start or F5 Optimization branch on tree to watch progress, scenarios so far, best scenario so far Cant absolutely guarantee a true optimum But usually finds far better configuration than possible by hand Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations

Slide 31 of 31