2011-12-06 Frank Koormann * test/test_lake.py, test/test_lake3.py: Set test values back to revision 3514: The values introduced with revision 3546 are relevant for newer emission scenarios like roots or streets, aka "USF mode". These are not implemented in this variant of the model suite. 2011-07-26 Andre Heinecke * GreaterModel/Worker/simpletreatmodel.py: Use numpy instead of Numeric. * GreaterModel/Worker/worker.py: Correct Type error by using string representation of a Tuple. Convert Numpy floats to regular python floats for jelling 2007-06-20 Frank Koormann * stop.py (main): Added socket.error exception handling and fixed bug (missing string) in info msg. 2006-05-10 Frank Koormann * GreaterModel/version.py: Updated version information. * GreaterModel/Worker/graph.py (Graph.__init__): Make the insertion of virtual stretches (to keep binary topology) more transparent. * GreaterModel/Worker/model.py: Deactivated DSSRiverModels. * test/support_itter.py (SessionDataMixin.uploadmodelmodes): Added hints on meaning of *COMMON modes. * test/test_model_river2_sediment.py: Updated all tests with reasonable substance and environmental data. Testcases for CSIMSTART_DISS, CSIMSTART_SORB and CSIMSTART_SED are derived by a spreadsheet from CSIMSTART (which is tested by test_model_river2.py) The stochastic tests are executed as coverage test, since the correct implementation of the model equations is tested by the deterministic tests. 2006-04-25 Frank Koormann * GreaterModel/Worker/rivermodels.py (RiverModel2.compute, RiverModel3.compute): Bugfix sediment densities in K_SW calculation. 2005-12-27 Frank Koormann * GreaterModel/Worker/rivermodels.py (RiverModel2.setup, RiverModel3.setup): BugFix sediment densities 2005-12-22 Frank Koormann * GreaterModel/Worker/rivermodels.py (RiverModel1.first_order_elimination): Bugfix c_internal and k=0.0 (RiverModel2.setup): Sediment densities (RiverModel2.compute): Consider sediment densities, unit conversion (RiverModel3.setup): Sediment densities (RiverModel3.compute): Consider sediment densities, unit conversion * test/test_model_river2_sediment.py (SedimentTests): Adapted tests. 2005-12-16 Frank Koormann * GreaterModel/Worker/rivermodels.py (RiverModel1.first_order_elimination): Fixed c_internal model for k = 0.0 * test/test_model_river.py: New, test first_order_elimination. 2005-11-03 Frank Koormann * GreaterModel/Worker/rivermodels.py (RiverModel1.compute, RiverModel2.compute, RiverModel3.compute): Optimization: Only compute, if stretch is expected to be loaded. 2005-10-26 Frank Koormann * GreaterModel/Worker/rivermodels.py (RiverModel3.compute): BugFix #3771: Unit error in k_sedimentation model. 2005-10-26 Frank Koormann * GreaterModel/Worker/rivermodels.py (RiverModel3.setup): Biodegradation, lumped river biomass (parameter X) must be lower case in API call. This bug slipped in with rev1.18. Load rho_sed and eps_sed if not already loaded by sediment extension. * test/support_testdata.py (uploadrivernet): Stretch class parameter X must be lower case. * test/support_itter.py (uploadrivernet): Stretch class parameter X must be lower case. 2005-10-26 Frank Koormann * GreaterModel/Worker/dssmodels.py (DSSRiverModel3._required_parameters): MOD Id for DSS mode 3 correctly set to 53 (was 3 and therewith overloaded the original mode 3) 2005-10-25 Frank Koormann Sediment Extension activated for river mode 3. * GreaterModel/Worker/rivermodel.py (RiverModel2._requiredparameters): Fixed mode ids for generic/class data (MODE_RIVER_COMMON_DATA) and added dependency on MODE_SED for MODE_SED_COMMON_DATA (this affects the required parameter presentation in dialogs). (RiverModel2.setup_general): New, general (i.e. potentially to be inherited) setup steps. (RiverModel2.setup): Model mode specific setup (calls setup_general). (RiverModel3): Now inherits from RiverModel2. (RiverModel3._requiredparameters): Added sediment extension specific parameters. (RiverModel3.setup): Call RiverModel2.setup_general to setup the general parameters (including sediment extension). 2005-10-24 Frank Koormann * GreaterModel/Worker/calculation.py (Calculation.__init__): Store stochastics in session.data before loading the data. Added calls for LoadSSConcBandsStats() requireing this data. * GreaterModel/Worker/data.py (Data.LoadTERRACEFile): Adaptions to new Sediment Extension concept (params are ENV-related). (Data.LoadSSConcBandsStats): New, load and initialize flow exceedance statistics driven SSC. * GreaterModel/Worker/rivermodels.py (RiverModel1.StochasticParameters): New, mix in diffuse input with stochastics (stretch based), mainly to be inherited by higher model modes (RiverModel2.RequiredParameters), (RiverModel2.StochasticParameters), (RiverModel2.setup): Adaption to new sediment modes. * GreaterModel/Worker/session.py (session_required_parameters): Load Environmental Data in addition to model modes. Some Sediment Extension relevant settings are stored under ENV. * test/support_itter.py (SessionDataMixin.uploadmodelmodes): Set new default model modes, dynamic handling of modes. * test/support_testdata.py (SessionDataMixin.uploadmodelmodes): Set new default model modes, dynamic handling of modes. * test/test_model.py (TestStandardCalculation.setUp): Call uploadmodelmodes considering the new modes. * test/test_model_river2_sediment.py (SedimentTests._test_default_stochastic): More dynamic handling of stochastic_params. (TestRiverMode2SedimentFlowExceedance): New, tests for Sediment Extension Mode 3. * test/test_sessionmethods.py (TestParameters.test_missing_parameters): Tests adapted for GREAT-ER Sediment Extension. 2005-10-24 Frank Koormann GREAT-ER Sediment Extension Mode 3: Flow exceedance related SSC * GreaterModel/Worker/suspendedsolids.py: New, classes for the SSC: SSConc and SSConcBand. * test/test_suspendedsolids.py: News, tests for the SSC classes. 2005-08-09 Frank Koormann * test/support_testdata.py (CatchmentDataMixin.uploadrivernet): Added dummy stretch class. 2005-03-10 Frank Koormann * GreaterModel/Worker/session.py (normalize): New helper, convert a list of tuples of various length to length 2. (session_required_parameters): Make use of normalize(). 2005-02-28 Frank Koormann Parameter Definition enhancement: While a tuple with two elements checks for "parameter exists" a tuple with three elements check "parameter exists and matches". * doc/great-er_model.tex: Added parameter definition enhancement to the specification. * test/test_sessionmethods.py (TestParameterDefinitions.test_check_method): Adaption to new parameter definition. (TestParameters.test_missing_parameters): Added tests for new parameter definition. * GreaterModel/Worker/modelbase.py (Model.CheckRequiredParameters, Model.MissingParameters): Adaption to new parameter definition. 2005-02-28 Frank Koormann GREAT-ER Sediment Extension, mode 4 (file based sediments and diffuse input) * GreaterModel/Scheduler/simulation.py (Simulation.add_res_stretch): Exclude sediment concentration from background concentration adding and determination of maximum concentration. * GreaterModel/Worker/data.py (Data.LoadTERRACEFile()): New, load 'SS_CONC' and 'DIFFUSE_INPUT' from TERRACE File. * GreaterModel/Worker/calculation.py (Calculation.__init__): Added session.LoadTERRACEFile() * GreaterModel/Worker/rivermodels.py (RiverModel1): Diffuse input added to required parameter info, setup() and compute(), moved unit conversion [1/h] -> [1/s] from HRT to k, adaption of new first_order_elimination param list (RiverModel1.self.first_order_elimination): New parameters I and V, added diffuse input calculation. (RiverModel2): Diffuse input and river common data mode, i.e. generic, per stretch or per class, added to required parameter info, setup() and compute(), moved unit conversion [1/h] -> [1/s] from HRT to k, adaption of new first_order_elimination param list * test/support_itter.py (ItterDataMixin.uploadrivernet): Fixed river data description. (SessionDataMixin.uploadmodelmodes): Added mode_sediment and mode_river_common. * test/support_testdata.py (SessionDataMixin.uploadmodelmodes): Added mode_sediment and mode_river_common. * test/test_model_river2_sediment.py: Added tests for Sediment Extension and diffuse input. 2004-09-23 Frank Koormann * GreaterModel/Worker/graph.py (Graph.__init__): Consider confluence and fork in lakes (type 6/10). Backported from USF-Branch, fixes bug 2634. 2004-09-17 Frank Koormann Reverted wrong checkins actually for GREAT-ER USF Lake extension branch. 2004-09-09 Frank Koormann * GreaterModel/Worker/rivermodels.py (RiverModel2.compute, RiverModel3.compute): Bugfix calculation CSIMINTERN_SORB 2004-08-23 Frank Koormann Added first version of GREAT-ER Sediment Exposure Module: Single PDF with equilibrium calculation. * test/test_calculation.py (TestModelHelper.setUp): Added ENV data set. (test_04_eval_class_parameter): New, test for eval_class_parameter(). * test/test_model_river2_sediment.py (TestRiverMode2Calculation.test_01_default_deterministic, TestRiverMode2Calculation.test_02_default_stochastic): Activated test cases for three new concentrations. * GreaterModel/Worker/modelbase.py (eval_local_setting): Code cleanup, use eval_parameter to check parameter keys. (eval_class_parameter): New helper: Return one of a pair of parameters depending in a flag, encapsulates an if statement. * GreaterModel/Worker/rivermodels.py (RiverModel2.setup): Parameter setup for sediment module, use_class concept added (RiverModel2.compute): Model equations for sediment module added. 2004-08-23 Frank Koormann * GreaterModel/Worker/dssmodels.py: New, DSS river modes seperated from default model, MODE_RIVER=51..53 * GreaterModel/Worker/model.py: Added DSSRiverModel? * GreaterModel/Worker/modelbase.py (Model.StochasticParameters, Model.MissingParameters): Code cleanup: Use Model.RequiredParameters to determine list of parameters to be processed. * GreaterModel/Worker/parameter.py (Parameter.set): New, set the parameter descriptives. Can be helpful when overwriting a global parameter with local settings. The developer has to take care that descriptives match the already applied distribution (and correlation!) * GreaterModel/Worker/rivermodels.py: General: DSS stuff abstracted. Basic Infrastructure for Sediment Module (RiverModel1.first_order_elimination): Return tuple of Flux and Concentrations instead of writing results directly. (RiverModel1.compute): Write segment results. 2004-08-23 Frank Koormann * test/test_model_river2_sediment.py: New, test for GREAT-ER Sediment Exposure Module with River model mode 2. * test/support.py (SchedulerProcess.__init__, WorkerProcess.__init__): Use DB API module _dagreater_pg as default. * test/support_itter.py, test/support_testdata.py (SessionDataMixin.uploadmodelmodes): Added sediment mode and flag to use stretch classes. * test/test_model_river2.py (TestRiverMode2Calculation.test_02_default_deterministic): Renamed to test_02_default_stochastic(). * test/test_parameter.py (TestParameters.test_06_set_parameter_descriptives): New, test Parameter.set() * test/test_scheduler.py (TestScheduler.test_scheduler_status): Updated to consider new scheduler status messages. 2004-02-25 Frank Koormann * GreaterModel/Worker/rivermodels.py (RiverModel1.first_order_elimination): Bugfix, reported by Sven Lautenbach, USF, Uni Osnabrück:Error on volume calculation in diffuse input mode. 2004-01-07 Frank Koormann * greater-dss.py: Included adaptions made by Riks, NL (new progress report format) Added version info (model system version and file revision) 2003-10-01 Frank Koormann * doc/great-er_model.tex: Naming update. 2003-09-30 Frank Koormann GREAT-ER model system version information * GreaterModel/version.py: New, version information * GreaterModel/Scheduler/scheduler.py (Scheduler.Status): Version information as second item in status dictionary. * test/test_xmlrpc_scheduler.py (TestXMLRPCSchedulerMethods.test_scheduler_status_version): New, test for version information. 2003-09-25 Frank Koormann * GreaterModel/Worker/graph.py, GreaterModel/Worker/model.py, GreaterModel/Worker/rivermodels.py: Added handling of lake type stretches. 2003-09-23 Frank Koormann * GreaterModel/Worker/dischargemodels.py, GreaterModel/Worker/simpletreatmodel.py: Fixed bug in treated fraction. 2003-09-11 Frank Koormann * test/support_itter.py: New, provides itter data for tests. * test/test_model.py, test/test_model_river2.py, test/test_model_river3.py: New, tests for model quality: Compare output against results of GREAT-ER 1.0. * test/support_testdata.py: Comments added. * GreaterModel/Worker/emissionmodels.py: Code formatting * GreaterModel/Worker/rivermodels.py: Fixes and cleanups due to quality audit. 2003-09-05 Frank Koormann * GreaterModel/Worker/dischargemodels.py (DischargeMode1): Added unit conversion to flux_nondom and flux_runoff and moved both from discharge model to emission. Control flow fix in case of local flow values. * GreaterModel/Worker/emissionmodels.py, GreaterModel/Worker/model.py: Moved flux_nondom and flux_runoff from discharge model to emission. * GreaterModel/Worker/rivermodels.py (RiverModel2.compute): Fixed assignement for k_vol and k_sed. 2003-07-28 Frank Koormann * doc/great-er_model_specification.tex, doc/great-er_model.tex: Set document version to 1.0.0 2003-07-25 Frank Koormann * doc/great-er_model_concept.tex, doc/great-er_model_scheduler.tex, doc/great-er_model_worker.tex: gaspell-checked. 2003-07-24 Bernhard Herzog * doc/great-er_model_concept.tex (Scheduler - Worker Concept): Add a paragraph about how the scheduler distributes a simulation to several workers. (Simulation Run Flow): Fix some typos. 2003-07-23 Frank Koormann * doc/Makefile: Updated to consider images and process dia files. * doc/dia/segment.dia, doc/dia/graph.dia, doc/dia/data.dia doc/dia/stochasticresult.dia, doc/dia/parameter.dia, doc/dia/modelclass.dia, doc/dia/model_emission_sewer.dia, doc/dia/model_WWTP.dia, doc/dia/model_river.dia: GREAT-ER II Model Class description. 2003-07-23 Frank Koormann * doc/great-er_model_specification.tex, doc/great-er_model_concept.tex, doc/great-er_model_scheduler.tex, doc/great-er_model_worker.tex: File information update. 2003-07-23 Frank Koormann * doc/great-er_model.tex: gaspell-checked and tex macro to deal with underscore in file name. * doc/gnufdl.tex: Update to comply with file info. 2003-07-23 Frank Koormann : * doc/great-er_model.tex: Typo corrections and three new diagramms illustrating the GREAT-ER Model Suite architectures. * doc/gnufdl.tex: Dummy tag to comply with file info. * doc/architecture_computing_server.sk, doc/architecture_single_user.sk, doc/architecture_distributed_computing.sk: New, diagramms illustrating the GREAT-ER Model Suite architectures. 2003-07-18 Thomas Koester * doc/great-er_model.tex, doc/great-er_model_specification.tex: Use twoside printing with binding correction of 19mm. Added fancy headers/footers for even pages. 2003-07-17 Thomas Koester * doc/great-er_model.tex, doc/great-er_model_concept.tex, doc/great-er_model_specification.tex: Use the term "GREAT-ER II". * doc/great-er_model.tex: Use GNU FDL 1.2. 2003-07-17 Frank Koormann * doc/great-er_model.tex: New, documentation of GREAT-ER II model framework with illustrative examples how to implement a model. * doc/emission_model.tex: New, example of emission model. * doc/river_model.tex: New, example of river model. * doc/gnufdl.tex: Update to version 1.2. * doc/Makefile: Added great-er_model to processing * doc/great-er_model_worker.tex: Added correlation to specification of session_stochatsic_parameters() 2003-07-14 Frank Koormann * GreaterModel/Worker/emissionmodels.py, GreaterModel/Worker/modelbase.py, GreaterModel/Worker/rivermodels.py, GreaterModel/Worker/simpletreatmodel.py: Comment/Code clean up. * GreaterModel/Worker/parameter.py (Parameter.__init__): Check descriptive parameter completeness. * test/test_parameter.py (TestParameters.test_01_staticparameter): Added test for descriptive parameter check. * test/test_sessionmethods.py: Comment/Code clean up. 2003-07-02 Frank Koormann * test/test_parameter.py (test_04_riverlognormalparameter): The test has been removed, since it covered a special behavior removed with the final version of GREAT-ER II: For GREAT-ER 1.0 (and development of GREAT-ER II distributed data of the river network was described by Mean and 5-Percentile, from which a lognormal distribution was derived. With GREAT-ER II the data has to be described by mean and standard deviation. 2003-07-02 Bernhard Herzog * GreaterModel/Scheduler/simulation.py (Simulation.enter_endstate): Do not enter the stopping state if all workers have already stopped (Simulation.calculation_error): Remove the worker which had the error from the running workers list and release all workers if all have finished. * test/test_xmlrpc_simulation.py (XMLRPCSimulationTest.setUp): Turn the dbapi into a class variable so that it's easier to override in derived classes. (TestSingleWorkerError): New test class for a test that checks the scheduler behavior in case of worker errors * test/mock_dbs/test_xmlrpc_simulation.py (ErrorMockDB): New mock db that raises exceptions (error_db): The default instance of ErrorMockDB 2003-06-25 Frank Koormann * GreaterModel/Worker/simpletreatmodel.py (DischargePrimarySimpleTreat.setup, DischargeActivatedSludgeSimpleTreat.setup): Check DWF and set to mean flow if not available. 2003-06-25 Jan-Oliver Wagner * GreaterModel/Worker/simpletreatmodel.py (DischargePrimarySimpleTreat.model): don't try computations if in-flux is zero. (DischargeActivatedSludgeSimpleTreat.model): don't try computations if in-flux is zero. Fixed wrong return value for Q=0-test. 2003-06-16 Frank Koormann * GreaterModel/Worker/parameter.py (Parameter.__init__): Drop Great-er 1.0 relict: All lognormal distributions have to provide mean and standard deviation. 2003-06-13 Frank Koormann * GreaterModel/Worker/rivermodels.py: Added correlation values to required parameters. * GreaterModel/Worker/simpletreatmodel.py (DischargeActivatedSludgeSimpleTreat.__RequiredParameters): New, local copy of Model.RequiredParameters to prevent endless recursion when analysing DischargePrimarySimpleTreat. 2003-06-06 Frank Koormann * test/test_simpletreat.py: New tests for the simple treat WWTP model. * test/test_sessionmethods.py (TestParameterDefinitions.test_check_method): Added doc-string. 2003-06-06 Bernhard Herzog Extend the scheduler so that it only gives workers to a simulation that run on different hosts. This behavior can be influenced with the the --allow-workers-on-same-host option * GreaterModel/Scheduler/run.py (options): New flag allow-workers-on-same-host (run): Pass the value of the new option to the scheduler (start_listeners): Use ModelBrokerFactory instead of BrokerFactory * GreaterModel/Scheduler/schedpb.py (ModelPerspectiveForWorker.__init__) (ModelPerspectiveForWorker.detached) (ModelPerspectiveForWorker.perspective_report_status): Store the worker proxy in self.worker instead of the reference in self.reference (ModelPerspective.brokerAttached): Instantiate a worker proxy for the ModelPerspectiveForWorker. (ModelServerService): New. Our own factory so that we can get at the netword address of the worker * GreaterModel/Scheduler/worker_proxy.py (WorkerProxy.__init__): New parameter with the network address (WorkerProxy.HostName): New. Accessor for the hostname * GreaterModel/Scheduler/scheduler.py (Scheduler): The scheduler never sees the PB references anymore it gets worker proxy instances from the pb interface. (Scheduler.__init__): New parameter force_different_hosts to make simulations never use multiple workers running on the same host. (Scheduler.RegisterWorker, Scheduler.UnregisterWorker) (Scheduler.ReleaseWorkers, Scheduler.worker_for_session_query) (Scheduler.handle_simulation_queue, Scheduler.WorkerReportStatus) (Scheduler.WorkerReportStatus): Updates for the fact that the scheduler never sees the PB references anymore. (Scheduler.choose_workers): New method to choose free workers for a simulation only choosing workers on different hosts if desired. (Scheduler.handle_simulation_queue): Use choose_workers to select workers * test/support.py (SchedulerProcess.__init__) (SchedulerProcess.start): Add support for the allow-workers-on-same-host flag * test/test_xmlrpc_simulation.py (XMLRPCSimulationTest.setUp): Add support for the allow-workers-on-same-host flag (XMLRPCSimulationTest.parse_worker_log) (TestXMLRPCTwoWorkers.parse_worker_log): Move this method from TestXMLRPCTwoWorkers to XMLRPCSimulationTest (TestXMLRPCTwoWorkers): Set allow-workers-on-same-host to true (TestXMLRPCTwoWorkersSameHost): New. Test whether the allow-workers-on-same-host flag works when not set. * test/test_worker_proxy.py (TestWorkerProxy.execute_test) (TestWorkerProxy.test_reference): Reflect worker proxy changes (TestWorkerProxy.test_hostname): New. Test for the HostName method * test/test_scheduler.py (TestScheduler.test_waiting_for_worker) (TestScheduler.test_simulation) (TestScheduler.test_scheduler_status) (TestSchedulerSessionMethods.test_required_parameters) (TestSchedulerSessionMethods.test_missing_parameters): The scheduler gets worker proxies now instead of references * test/test_pb_sched_worker.py (TestPBSchedulerWorker.listen): Use the ModelBrokerFactory instead of the pb.BrokerFactory 2003-06-06 Bernhard Herzog * GreaterModel/Scheduler/schedpb.py: (ModelPerspectiveForWorker, ModelPerspective): Split ModelPerspective into ModelPerspective and ModelPerspectiveForWorker (ModelPerspective.attached): Return a new instance of ModelPerspectiveForWorker instead of what the inherited method would have returned. This way we get one perspective object per worker and we can properly determine which worker called report_status. * GreaterModel/Scheduler/scheduler.py (Scheduler.RegisterWorker) (Scheduler.UnregisterWorker): Log when workers are registered/unregistered with the scheduler * test/test_pb_sched_worker.py (MyModelPerspective): Removed since we don't really need it and its behavior was not sufficiently like the ModelPerspective anymore not that we have different perspective opjects for each worker. (MyModelServerService): Use the normal ModelPerspective class. (MyScheduler.UnregisterWorker): New. Extend inherited method to check number of workers and stop the reactor (MyModelServerService.__init__): Removed. The inherited constructor is good enough. (TestPBSchedulerWorker.listen): Update becazse of the MyModelServerService constructor change (TestPBSchedulerWorker.test): Add "Scheduler.UnregisterWorker" to the list of called methods * test/test_xmlrpc_simulation.py (XMLRPCSimulationTest.setUp) (XMLRPCSimulationTest.start_new_worker): Split starting a worker into a separate method so that some test cases can easily start additional workers (TestXMLRPCSingeWorker.test_start_simulation_after_worker_detached): New. Test for the bug where detaching workers causes problems 2003-06-06 Silke Reimer * GreaterModel/Worker/rivermodels.py: Some corrections in equations for first_order_elimination of diffusive input 2003-06-06 Frank Koormann * test/test_simulation.py (TestSimulation.test_generate_numbers): Make sim.random_numbers_to_do a real copy of sim.stochastic_parameters. Fixes bug in progress calculation for unit test. 2003-06-05 Bernhard Herzog * GreaterModel/Scheduler/scheduler.py (Scheduler.AcquireWorkers) (Scheduler.AcquireWorker): Renamed to AcquireWorkers (Scheduler.ReleaseWorkers): Update doc-string (Scheduler.handle_simulation_queue): Honor the number of workers the simulation requested if possible. * GreaterModel/Scheduler/simulation.py (Simulation.enter_endstate): New. Ending the simulation becomes a bit more complex now that we can have multiple workers so introduce this new method to make sure the worker are stopped when they're running. Stopping multiple workers is done with a new state, "stopping" (Simulation.failure_catcher, Simulation.Stop) (Simulation.WorkerFinished, Simulation.handle_error) (Simulation.got_stochastic_parameters) (Simulation.generate_numbers, Simulation.UseWorkers) (Simulation.calculation_error, Simulation.write_results): Use enter_endstate to end the simulation (Simulation.release_workers, Simulation.release_worker): Renamed to release_workers. Updated callers. (Simulation.release_workers): Log the worker release (Simulation.WorkerProgress): Compute the combined progress correctly and only call set_state once. * GreaterModel/Scheduler/schedxmlrpc.py (SchedulerXMLRPC.xmlrpc_scheduler_status) (SchedulerXMLRPC.xmlrpc_scheduler_shutdown) (SchedulerXMLRPC.xmlrpc_simulation_start) (SchedulerXMLRPC.xmlrpc_simulation_status) (SchedulerXMLRPC.xmlrpc_simulation_stop) (SchedulerXMLRPC.xmlrpc_session_required_parameters) (SchedulerXMLRPC.xmlrpc_session_missing_parameters): Log which XMLRPC calls happen and their parameters * test/support.py (SimulationStatusMixin.assertValidStatus): Add the new "stopping" state * test/test_scheduler.py (MockSimulation.Run): Reflect AcquireWorkers name change * test/test_simulation.py (MockScheduler.AcquireWorkers) (TestSimulationRun.test_run) (TestSimulationRun.test_run_two_workers): Reflect AcquireWorkers name change (TestSimulation.test_WorkerProgress): Set sim.workers too so that the progress calculation works (MockWorkerProxy.StartSimulation, MockWorkerProxy.done): Turn result into an instance variable set in StartSimulation (MockWorkerProxy.StopSimulation): New. Set self.result to "end_stopped". (TestSimulationStopWriting.test_stop_writing_results_two_workers): New. Test whether stopping works with two workers * test/test_xmlrpc_simulation.py (XMLRPCSimulationTest) (TestXMLRPCSingeWorker, TestXMLRPC): Split TestXMLRPC into a base class XMLRPCSimulationTest with code common to all simulation tests via XMLRPC and TestXMLRPCSingeWorker with the actual test cases which use a single worker (XMLRPCSimulationTest.setUp): Make it possible to start multiple workers each with their own log files and a separate log file for the scheduler. (XMLRPCSimulationTest.make_log_dir) (XMLRPCSimulationTest.make_log_file_name): New. Create log files in the logs/ subdirectory with names containging the name of the test case. (XMLRPCSimulationTest.tearDown): stop all workers (TestXMLRPCTwoWorkers): New class with test using two workers. The actual tests are basically the same as in TestXMLRPCSingeWorker. * doc/great-er_model_scheduler.tex: Add the new "stopping" state 2003-06-05 Frank Koormann * test/test_worker.py (TestWorker.test_worker_single_calculation), (TestWorker.test_worker_parallel_calculations): Derive the expected descriptives from the new raw results. 2003-06-05 Frank Koormann Updates to merge multiple worker results in one result per segment (per result type). * GreaterModel/Scheduler/simulation.py (get_stochasticresult): New function to derive mean, standard deviation and number from the raw stochastic results (sum, sumsqr and num). (Simulation.write_results): Copy into the results_chunk the values from all dictionaries in the result list which fit the keys from the key_chunk. (Simulation.add_res_stretch): Merge results for one result type from all result dictionaries (for the distinct workers) in one result (Simulation.add_res_disch): Merge results for one result type from all result dictionaries (for the distinct workers) in one result * GreaterModel/Worker/stochasticresult.py (StochasticResult.get_raw): New, returns sum, sumsqr and num * GreaterModel/Worker/worker.py: (Worker.iterate_calculation): On "end_complete", collect the raw results (get_raw) instead the descriptives (mean, stdev). 2003-06-04 Bernhard Herzog Next step towards multiple workers per simulation. The simulation can now be spread to several workers but there are still some limitations including that the results are not yet combined properly. * GreaterModel/Scheduler/scheduler.py (Scheduler.__init__): Update comment about self.simulation_queue which now holds simulations and how many workers they requested (Scheduler.AcquireWorker): New parameter num_workers to indicate how many workers the simulation wants. (Scheduler.ReleaseWorker): Rename to ReleaseWorkers (Scheduler.ReleaseWorkers): Accept a list of workers instead of a single worker (Scheduler.handle_simulation_queue): Pass a list of workers to the simulation's UseWorkers method. For now it's just a single one. * GreaterModel/Scheduler/simulation.py (Simulation.__init__): Add several new instance variables to keep track of multiple workers (Simulation.Stop): Stop all workers (Simulation.WorkerFinished): New parameter for the worker whose result is given as argument. Release the workers once all workers have finished. Only when all workers have returned results start processing them (Simulation.release_worker): Deal with multiple workers (Simulation.run_simulation): Estimate how many workers to use and request that many workers from the scheduler (Simulation.UseWorker): Renamed to UseWorkers (Simulation.UseWorkers): Accept a list of workers instead of a single worker and split the work package among the workers (Simulation.calculation_error): Add the worker as a parameter (Simulation.WorkerProgress): Make sure the progress value is an int. * GreaterModel/Scheduler/worker_proxy.py (WorkerProxy.__eq__) (WorkerProxy.__hash__): New to provide well defined use of worker proxies as keys in a dictionary * test/test_simulation.py (MockScheduler.__init__) (MockScheduler.set_max_num_workers): Add the max_num_workers instance variable (settable via the new set_max_num_workers) to influence how many workers a simulation can actually get (MockScheduler.AcquireWorker): Reflect changes in scheduler.py and give the simulation all the workers it wants and max_num_workers permits (MockScheduler.ReleaseWorker, MockScheduler.ReleaseWorkers): Reflect changes in scheduler.py (TestSimulation.test_WorkerProgress): Update to reflect that only ints are allowed as values (SimulationReactorTest.setUp): stochastic_parameters is now a dict mapping workers to the stochastic parameter set they are told work on (SimulationReactorTest.stochastic_parameters): Add the worker as parameter (MockWorkerProxy.StartSimulation): Pass self to stochastic_parameters (TestSimulationRun.test_run): The simulation will request 2 workers now. Check whether it actually only got one worker. (TestSimulationRun.test_run_two_workers): New. Test whether the simulation can cope with two workers. * test/test_scheduler.py (MockSimulation.__init__) (MockSimulation.Run, MockSimulation.UseWorkers) (MockSimulation.Stop): Adapt to the simulation changes with respect to multiple workers (TestScheduler.test_simulation): Update comment. 2003-06-04 Frank Koormann * GreaterModel/Scheduler/simulation.py (known_correlations): Mapping dictionary for correlation functions. (Simulation.generate_numbers): Extended to apply correlations if requested. * test/test_simulation.py (TestSimulation.test_generate_numbers): New, tests generate_numbers method with correlated parameters. 2003-06-04 Frank Koormann * GreaterModel/Scheduler/simulation.py (Correlation_normal_normal, Correlation_normal_lognormal, Correlation_lognormal_lognormal, Correlation_lognormal_normal): New, correlation functions. * test/test_simulation.py (TestCorrelationFunctions): New, tests for correlation functions. 2003-06-04 Frank Koormann * test/test_graph.py (TestGraph.test_graphintegritycheck_bifurcation, TestGraph.test_graphintegritycheck_doubleroot): Flow velocity is distributed. 2003-06-03 Frank Koormann First step towards correlations in GREAT-ER II Model System and adaptions to multiple worker concept. * GreaterModel/Scheduler/simulation.py (Simulation.__init__): New instance variable worker_progress: dictionary storing progress of multiple workers (keys: worker proxy objects) (Simulation.WorkerProgress): Cominbe the seperate progress information into an overall simulation progress. (Simulation.generate_numbers): Accept new stochastic parameter format. * GreaterModel/Worker/modelbase.py (Model): new instance variable _correlated_parameters (Model.StochasticParameters): Perform correlation evaluation for non-local parameters. (Model.eval_correlation): New, evaluate if a parameter is correlated. * GreaterModel/Worker/rivermodels.py (RiverModelMode1, RiverModelMode2, RiverModelMode3): Define _correlated_parameters * GreaterModel/Worker/graph.py (Graph.__init__): Adaptions to new stochastic parameter format. * test/test_sessionmethods.py (TestParameters.test_stochastic_parameters): Extend tests to cover the new stochastic parameter format and Model.StochasticParameters. * test/support_testdata.py (CatchmentDataMixin.uploadrivernet): Upload v_dist as distributed parameter * test/test_pb_worker.py (TestPBWorker.test_session_stochastic_parameters): Adaption to new stochastic parameter format * test/test_simulation.py (MockWorkerProxy.SessionStochasticParameters): Adaption to new stochastic parameter format (TestSimulation.test_WorkerProgress): New, test Simulation.WorkerProgress. 2003-06-02 Bernhard Herzog First step to support multiple workers for one simulation: * GreaterModel/Scheduler/simulation.py (Simulation.WorkerProgress): Accept a worker proxy object as additional parameter so that it's possible to compute a progress value even when multiple workers are used. (Simulation.__init__): New instance variable results. A list to accumulate the results of the workers (Simulation.WorkerFinished): Append the result to self.results instead of using it directly as self.result (Simulation.release_worker): Fix typo in doc-string (Simulation.calculation_error): Log the complete failure (Simulation.start_writing_results, Simulation.write_results): Adapt to change from self.result to self.results * GreaterModel/Scheduler/scheduler.py (Scheduler.WorkerReportStatus): Accept the perspective's worker reference as a parameter map it to the worker proxy and pass it to the simulation * GreaterModel/Scheduler/schedpb.py (ModelPerspective.perspective_report_status): Pass the worker reference as parameter 2003-05-28 Frank Koormann * GreaterModel/Worker/model.py: Added RiverSegmentType dictionary * GreaterModel/Worker/graph.py: Lakes as new river type. * GreaterModel/Worker/modelbase.py (remove_parameter): New, remove entry from a (parameters) list. * GreaterModel/Worker/rivermodels.py: Added support for lake stretches (RiverModel3.setup): Environmental parameters for wind_speed, air and water temperature overwrite stretch class values. * GreaterModel/Worker/simpletreatmodel.py (DischargePrimarySimpleTreat, DischargeActivatedSludgeSimpleTreat): Estimation Function clean up 2003-05-26 Frank Koormann Model Mode detection (a model can be used for more than one model mode, to identify this required_parameters list is extended to list of lists) and bug fixes * GreaterModel/Worker/graph.py (__init__): Adaption to new required_parameter list * GreaterModel/Worker/modelbase.py (Model.RequiredParameters, Model.CheckRequiredParameters, Model.StochasticParameters, Model.MissingParameters): Adaption to new required_parameter list. * GreaterModel/Worker/sewermodels.py: Adaption to new required_parameter list. * GreaterModel/Worker/dischargemodels.py (DischargeTFMode1._required_parameters): Adaption to new required_parameter list. * GreaterModel/Worker/simpletreatmodel.py: Adaptions to new required_parameter list. * GreaterModel/Worker/rivermodels.py (RiverModel3): Bug Fixes 2003-05-26 Frank Koormann * GreaterModel/Worker/simpletreatmodel.py: Debug output cleanup. 2003-05-09 Frank Koormann * GreaterModel/Worker/stochastic.py (StdNormToNorm()): Harmonized parameters with other functions: Descripives as tuple. (PercentileFromStdNorm()): Percent value as ]0,100[ (PercentileFromLogNorm()): Return mean if standard deviation is zero. * test/test_stochastic.py: Added tests for PercentileFromStdNorm(), StdNormToNorm(), PercentileFromLogNorm() * GreaterModel/Scheduler/simulation.py (Simulation.write_results()): Changed result format to exponent notation. * test/test_simulation.py (TestSimulationRun): Update to new result format (bin_obj instead of res_det_x) 2003-05-09 Jan-Oliver Wagner * GreaterModel/Worker/stochastic.py (PercentileFromStdNorm, StdNormToNorm, PercentileFromLogNorm): New. 2003-05-07 Frank Koormann * GreaterModel/Scheduler/simulation.py: Change upload of results from da_load_bin_obj() to da_insert_bin_obj(), so the temp file is no longer needed. StringIO is used instead. (Simulation.start_writing_results): Initialize StringIO objects. (Simulation.write_results): Obtain String object from StringIO, derive file_size (long!) and upload to the database (da_insert_bin_obj). 2003-05-06 Frank Koormann * GreaterModel/Scheduler/simulation.py (Simulation.analyse_table_headers): Fixed header typo for stretch results table. 2003-05-06 Frank Koormann * GreaterModel/Scheduler/simulation.py (Simulation.start_writing_results): Get the session privileges for storing results with the same privs later. 2003-05-06 Frank Koormann * GreaterModel/Scheduler/simulation.py (Simulation): Updated result writing framework to store results as BLOBS into the database: (Simulation.analyse_table_headers): New, determine the table headers for the result tables from the availabale results. Currently fixed to results derived by and needed for GREAT-ER II. (Simulation.write_results): Write results into database as BLOB using da_load_bin_obj() (file based). 2003-05-06 Bernhard Herzog * GreaterModel/Scheduler/simulation.py (Simulation.WorkerFinished) (Simulation.calculation_error): Call the new method to releas the worker. (Simulation.release_worker): New method to relase the worker. Make sure to release it only once. 2003-04-30 Frank Koormann New Discharge Model: Simple Treat (Activated Sludge/Primary Settler Mode 2/3) * GreaterModel/Worker/simpletreatmodel.py: New, implements Simple Treat models for Primary Settler and Activated Sludge * GreaterModel/Worker/model.py: Added Simple Treat model to model list 2003-04-22 Frank Koormann Performance-Tuning for Initialization: Derive the list of stochastic parameters once per model instance. Local parameters can be analysed later by the specific model objects. * GreaterModel/Worker/modelbase.py (Model.__init__()): List of stochastic parameters as second option. None by default. * GreaterModel/Worker/emissionmodels.py, GreaterModel/Worker/dischargemodels.py, GreaterModel/Worker/sewermodels.py, GreaterModel/Worker/rivermodels.py (*.setup()): All setup-Methods updated to use self.st_params * GreaterModel/Worker/graph.py (Graph.__init__()): Initialize model objects with stochastic parameters list. 2003-04-17 Frank Koormann The update is dominated by rework of the discharge model section: Further cleanup for mode 1 and preparations for mode 2/3. * GreaterModel/Worker/dischargemodels.py (DischargeMode1): New. The model mode 1 base, which reduces the implementation of specific models to specify name and required parameters. With some already prepared updates it could also serve as base for (a new) mode 2. (DischargePrimMode1, DischargeASMode1, DischargeTFMode1): Reimplemented, derived from DischargeMode1 * GreaterModel/Worker/graph.py (Graph.__init__()): Consider the updated definition of DISCH_CLASS_TAB.WWTP_TYPE * GreaterModel/Worker/modelbase.py: Added constants for model systems: GAS_CONSTANT (R), KELVIN * GreaterModel/Worker/rivermodels.py (RiverModel2.MissingParameters, RiverModel2.MissingParameters): Code cleanup, makeing use of modelbase.eval_parameter() * test/support_testdata.py (CatchmentDataMixin.uploaddischarges): Updated DISCH_CLASS_TAB.WWTP_TYPE to new definition. (SessionDataMixin.uploadmodelmodes): New, seperates the functionality former implemented in uploadsession() and allows to initialize various model modes. * test/test_calculation.py (TestCalculation.setUp): * test/test_graph.py (TestGraph.setUp) * test/test_sessionmethods.py (TestParameters.setUp): Added uploadmodelmodes 2003-04-09 Frank Koormann Model System Cleanup: Share common code between models if possible. * GreaterModel/Worker/modelbase.py (eval_parameter): Helper function: Return parameter if it exists, else None * GreaterModel/Worker/dischargemodels.py (DischargeMainMode1): New class derived from DischargePrimaryMode1, implements the base for discharge main processing, DischargeASMode1 and DischargeTFMode1. The latter two differ only in the initialisation of one parameter, the main processing removal rate. * test/test_calculation.py (TestModelHelper.test_01_eval_parameter, TestModelHelper.test_03_eval_local_stochastic): New tests for model helper functions. 2003-04-08 Frank Koormann * GreaterModel/Worker/rivermodels.py (RiverModel1.first_order_elimination): New method implementing the general first order elimination as used by all river models. (RiverModel2, RiverModel3): Changed accordingly. 2003-04-07 Frank Koormann Merged DSS Elbe Branch (GREAT-ERModel-DSS) back into HEAD: 2003-04-02 Frank Koormann * modelsystem.py: New, module providing classes to manage the model system, including path configuration. * greater-dss.py: New, front-end of the GREAT-ER model system to the DSS Elbe 2003-04-01 Frank Koormann Significant updates have been performed while implemeting the DSS Elbe extension for diffuse input: 1. Database API (dbase file base) 2. Model Implementation 3. Updates to local setting evaluation * GreaterModel/Scheduler/run.py (run): Added DSSFileDB as new API module. * GreaterModel/Worker/workerpb.py (run()): Added DSSFileDB as new API module. * GreaterModel/Worker/data.py (General): Import GreaterDB.Interface instead of the seperate methods. This provides more flexibility when working mit various API modules. As a consequence all API calls are updated to GreaterDB.interface.APImethod() (Data.LoadDiffuseInputDSS()): New: Load diffuse input data sets if USE_DIFFUSE flag is defined as model parameter. * GreaterModel/Worker/calculation.py (Calculation.__init__): Added LoadDiffuseInputDSS(). * GreaterModel/Worker/session.py (session_stochastic_parameters, session_missing_parameters): LoadDiffuseInputDSS * GreaterModel/Worker/graph.py (Graph.__init__): Added list of root stretches to NetworkIntegrity error message. * GreaterModel/Worker/rivermodels.py: All three models are extended by diffuse input according to the DSS documentation (DSS Elbe, Endbericht Phase 1, page 10 pp). All three models now check for L > 0.0. Else PHIout = PHI * GreaterModel/Worker/modelbase.py (eval_local_settings): Return tuple (value, status). Status indicates if the global value (0) or a local setting (1) is used. (eval_local_stochastics(status, stochastics)): New. By definition local settings are not distributed. The method checks the status as returned by eval_local_settings(). The approriate place to call eval_local_stochastics is the parameter list for a Parameter.__init__ call. * GreaterModel/Worker/dischargemodels.py, GreaterModel/Worker/emissionmodels.py (General): Updates to extended local setting processing. * test/test_calculation.py (TestModelHelper.test_01_eval_local_setting): Adapted test to status returned by eval_local_setting() 2003-04-07 Bernhard Herzog * GreaterModel/Scheduler/simulation.py (Simulation.Stop): If the worker is running do not go into end_stopped state immediately. Wait until the worker has actually stopped. Otherwise we get problems if the worker tries to report its status before it honors the stop request because the simulation may already have been removed in the scheduler. 2003-03-27 Frank Koormann * GreaterModel/Worker/rivermodels.py (RiverModel1.compute, RiverMode2.compute, RiverMode3.compute): Harmonized terminology with DSS Branch (use exp_term instead of E_river for exp( - k * HRT ) 2003-03-27 Frank Koormann * GreaterModel/Worker/rivermodels.py (RiverModel1.compute, RiverMode2.compute, RiverMode3.compute): Updated elimination equation to more transparent notation. Changed calculation of CSIMINTERN: now use PHI (as specified by G.Boeije) instead of PHIout if k==0. 2003-03-22 Frank Koormann * GreaterModel/Worker/graph.py, * GreaterModel/Worker/dischargemodels.py: Typos in comments fixed. * doc/great-er_model_specification.tex: Update information extended. 2003-03-20 Frank Koormann * GreaterModel/Scheduler/simulation.py (Simulation.got_stochastic_parameters): Removed call for da_get_mod(). Not needed. 2003-03-06 Frank Koormann Reorganisation of model componentents for better transparency: * GreaterModel/Worker/model.py: All model implementation removed from here, works a interface to model system and provides ModelList. * GreaterModel/Worker/modelbase.py: New, provides Model base class and helpers. * GreaterModel/Worker/emissionmodels.py: New, emission models. * GreaterModel/Worker/sewermodels.py: New, sewer models. * GreaterModel/Worker/dischargemodels.py: New, discharge models. * GreaterModel/Worker/rivermodels.py: New, river models. 2003-03-06 Bernhard Herzog * doc/great-er_model_worker.tex, doc/great-er_model_scheduler.tex: minor wording fixes. * doc/great-er_model_concept.tex: Fix typos and tweak the wording. 2003-03-06 Frank Koormann * doc/Makefile: Extended rule to generate pdf with raster images. * doc/great-er_model_specification.tex: Document info updated * doc/great-er_model_concept.tex (Chapter Concept): Added section on scheduler/worker concept (moved here from Chapter Scheduler). * doc/great-er_model_scheduler.tex (Chapter Scheduler): Concept moved, added section on configuration and start/stop. Switched off open issues. * doc/great-er_model_worker.tex (Chapter Worker): Added section on configuration and start/stop. 2003-03-05 Bernhard Herzog * doc/great-er_model_worker.tex (chapter{Worker}): Update description with brief information about the new scheduling. (start_calculation): Add the simid parameter. * doc/great-er_model_scheduler.tex (report_status): add the simid parameter. 2003-03-05 Bernhard Herzog Change the scheduling so that even if all registered workers are already working on a calculation a new simulation will start running immediately instead of waiting for a free worker. This is achieved by modifying the workers so that they can do several calculations simultaneously. There are three main block of changes: 1. The new worker management: * GreaterModel/Scheduler/scheduler.py (Scheduler.__init__): New instance variable worker_load to keep track of how many calculations a worker is doing simultaneously. (Scheduler.ReleaseWorker): Only put the worker back into the free_workers if its load has become zero. (Scheduler.handle_simulation_queue): Allow using a worker that already is working on a simulation. 2. The worker maintains a list of calculations: * GreaterModel/Worker/worker.py (Worker.__init__): New instance variable calculations for a list of calculation objects (and related information such as simid and deferred). No need for the deferred_calculation ivar any more. (Worker.find_calculation, Worker.remove_calculation): Two new methods to maintain the list of calculations (Worker.start_calculation): New parameter simid. Add the new calculation to the calculations list. (Worker.stop_calculation): New parameter simid so that we can stop the right calculation. (Worker.return_result, Worker.return_error): Removed. It's easier to implement the functionality in iterate_calculation now. (Worker.iterate_calculation): Execute one iteration of one of the calculations in a round robin fashion so that each calculation gets the same priority in terms of number of shots. 3. The worker needs the simulation id so it has to be passed back and forth between worker and scheduler: * GreaterModel/Scheduler/worker_proxy.py (WorkerProxy.__init__): No need to keep track of the simulation object because the worker can now work for several simulations at once. (WorkerProxy.StartSimulation): Pass the simulation id. (WorkerProxy.StopSimulation): Add the simulation argument to pass its id to the worker. (WorkerProxy.ReportStatus): Removed. The scheduler can now get at the simulation directly. * GreaterModel/Scheduler/simulation.py (Simulation.Stop): The worker proxy's StopSimulation now has the simulation as parameter. * GreaterModel/Scheduler/schedpb.py (ModelPerspective.perspective_report_status): Add the parameter simid * GreaterModel/Scheduler/scheduler.py (Scheduler.WorkerReportStatus): Add the simid parameter and use it to pass the status directly to the simulation. Do not go through the worker proxy anymore. * GreaterModel/Scheduler/worker_proxy.py (WorkerProxy.ReportStatus): Removed. The scheduler can now get at the simulation directly. * GreaterModel/Worker/workerpb.py (Scheduler.ReportStatus) (WorkerPerspective.remote_start_calculation) (WorkerPerspective.remote_stop_calculation): Add simid parameter. Test case updates for the new scheduling * test/test_worker.py (MockScheduler.ReportStatus): Add the simid parameter (TestWorker.test_worker_simulation) (TestWorker.test_worker_single_calculation): Rename to the clearer test_worker_single_calculation. Pass a simid to start_calculation (TestWorker.test_worker_parallel_calculations): New testcase for simulataneous calculations in the worker. (TestWorker.multiple_results): Helper method for test_worker_parallel_calculations (TestWorker.test_worker_stop_calculation): Add simid to start_calculation and stop_calculation * test/test_worker_proxy.py (TestWorkerProxy.test_start_calculation) (TestWorkerProxy.test_stop_calculation): Adapt for the new simid parameters 2003-03-05 Bernhard Herzog * test/test_xmlrpc_simulation.py (TestXMLRPC.test_two_parallel_simulation_calls): Update comment. 2003-03-05 Bernhard Herzog * test/test_xmlrpc_simulation.py (TestXMLRPC.run_simulation): Increase the timeout to make the test succeed on windows. * test/test_worker.py (TestWorker.test_worker_simulation): Increase the timeout to make the test succeed on windows. * test/test_pb_sched_worker.py (TestPBSchedulerWorker.test): Increase the timeout to make the test succeed on windows. 2003-02-26 Bernhard Herzog * GreaterModel/Scheduler/simulation.py (Simulation.write_res_record): Remove a debug print. 2003-02-25 Bernhard Herzog * doc/great-er_model_concept.tex (subsubsection{Component Communication Path}): Fix some spelling. * doc/great-er_model_scheduler.tex (session.required_parameters) (session.missing_parameters): Remove the description of error returns. XMLRPC has a fault mechanism which is used instead. 2003-02-24 Bernhard Herzog * test/test_reactor_demo.py: New. Demo test cases to show how the reactor mixins are to be used. Also serves as a test for the mixins. 2003-02-21 Frank Koormann * GreaterModel/Worker/model.py (Model.StochasticParameters): Added check for blocks 'MOD','ENV','SUBST' if key with field_id exists before accessing it. 2003-02-18 Bernhard Herzog * start_worker.py, start_scheduler.py: Add the GREAT-ER-DB directory to sys.path 2003-02-17 Bernhard Herzog * GreaterModel/Scheduler/simulation.py (Simulation.got_stochastic_parameters): Remove a debug print * stop.py: Make sure the Lib directory is in sys.path. Print a message if the process was shutdown successfully 2003-02-17 Frank Koormann * GreaterModel/Worker/model.py: Fix: Estimate function for Kd is based on k_oc (and not on k_ow). This is corrected now. 2003-02-14 Bernhard Herzog * GreaterModel/Worker/workerpb.py (Options): Two new options, admin-port and info-file (run): Start the admin XMLRPC interface if an admin port was specified. Write the info file. * GreaterModel/Worker/admin.py: New. The admin interface * GreaterModel/Scheduler/run.py (Options): Two new options, admin-port and info-file (start_listeners): Start the admin XMLRPC interface if an admin port was specified. Write the info file. * GreaterModel/Scheduler/schedxmlrpc.py (do_shutdown): Factor shutting down the scheduler into the do_shutdown function (SchedulerXMLRPC.xmlrpc_scheduler_shutdown): Use do_shutdown (AdminXMLRPC): New class with the admin interface. * stop.py: New script to stop a scheduler or worker given the info file it has written * test/support.py (SchedulerProcess.__init__): Add admin_port and info_file parameters (SchedulerProcess.start): Pass the new parameters to the scheduler (WorkerProcess.stop): New. Run the stop script * test/test_scheduler_process.py (TestSchedulerProcess.test_admin): New test for the admin interface 2003-02-13 Bernhard Herzog * GreaterModel/Scheduler/simulation.py (Simulation.got_stochastic_parameters): Check whether the BG_CONC value is not an empty string before using it. 2003-02-13 Bernhard Herzog * test/test_simulation.py (MockWorkerProxy.done): Remove the "V_1" result. The worker is not supposed to return that anymore. (SimulationMockDB.__init__): Add a value for the background concentration (SimulationReactorTest.check_final_state): Print the value of the final state too if the final state is not end_complete * GreaterModel/Scheduler/simulation.py (Simulation.__init__): New instance variable background_concentration (Simulation.got_stochastic_parameters): Retrieve background_concentration as well. (Simulation.add_res_stretch): Add the background_concentration to the mean before putting it into the record and updating the maximum 2003-02-12 Bernhard Herzog * GreaterModel/Worker/calculation.py (Calculation.get_results): Only return results for stretches and discharges * GreaterModel/Scheduler/simulation.py: Don't silently ignore the virtual result nodes. Log a message for them too. * test/test_calculation.py (TestCalculation.test_01_deterministic): Extend the test to check whether only stretch and discharge results are returned * GreaterModel/Worker/worker.py (Worker.iterate_calculation): Fix a typo 2003-02-11 Frank Koormann * GreaterModel/Worker/model.py (RiverModel3): New, implements GREAT-ER River Model Mode 3. 2003-02-11 Frank Koormann * GreaterModel/Worker/data.py (Data.connect, Data.disconnect): New methods to explicitely connect and disconnect the database. (all methods): __doc__ added. * GreaterModel/Worker/calculation.py (Calculation.__init__): Data connect / disconnect added. * GreaterModel/Worker/session.py (session_required_parameters, session_stochastic_parameters, session_missing_parameters): Data connect / disconnect added. 2003-02-10 Bernhard Herzog * GreaterModel/Scheduler/simulation.py (Simulation.end_state): New. Refactor testing whether the simulation has ended (Simulation.WorkerFinished): Make it clearer that releasing the worker has to be done first and handle the case that the simulation is already in an endstate. (Simulation.generate_numbers, Simulation.UseWorker): Use the new end_state method (Simulation.write_results): Do nothing if the simulation is in an end state to handle e.g. stopping the simulation while it's writing results. * test/test_simulation.py (TestSimulationStopWriting.test_stop_writing_results): New test case to test stopping simulations while they write results (StoppingMockDB, StoppingWorkerProxy): New helper classes for the new test case (MockWorkerProxy.StartSimulation): Remember the simulation object so that derived classes can use it. (MockScheduler.__init__): New parameter worker_proxy_class so that test cases can easily use other mock worker proxy implementations (MockScheduler.worker_for_session_query) (MockScheduler.AcquireWorker): Use self.worker_proxy_class. (SimulationMockDB.__init__): New class variable mock_worker_proxy_class to indicate which worker proxy implementation to use (SimulationReactorTest.setUp): Use self.mock_worker_proxy_class (TestSimulationRun.run_simulation) (SimulationReactorTest.run_simulation): Moved to SimulationReactorTest so that all derived classes can use it 2003-02-10 Frank Koormann * GreaterModel/Worker/data.py (Data.LoadSessDischData, Data.LoadEmissionData): Check if ids are set (i.e. not -1). Fixes Bug #1587 2003-02-10 Frank Koormann * GreaterModel/Scheduler/simulation.py (Simulation.write_res_record): Set default factor/unit to 1000 / mg/l. 2003-02-07 Bernhard Herzog * GreaterModel/Scheduler/simulation.py (Simulation.write_results): Handle the case of no discharge results. 2003-02-07 Bernhard Herzog * GreaterModel/Scheduler/simulation.py (Simulation.failure_catcher) (Simulation.callLater): Two new methods to make the simulation more robust with respect to exceptions raised in callbacks registered with the reactor's callLater. (Simulation.got_stochastic_parameters) (Simulation.generate_numbers, Simulation.start_writing_results) (Simulation.write_results): Use self.callLater to register a callback instead of the reactor's 2003-02-07 Bernhard Herzog Determine river_factor and stp_factor * GreaterModel/Scheduler/simulation.py (factor_and_unit_from_max): Helper function for the factor computations (Simulation.write_results): Add new state for updating the result record with the factors. (Simulation.start_writing_results): New instance variables to find the maximum and for the state-machine in write_results. (Simulation.write_res_record): Keep the result record so that we can insert the right factor values later (Simulation.add_res_stretch, Simulation.add_res_disch): Keep track of the maximal values written so far. (Simulation.write_factors): New method to update the result record with the factors * test/test_simulation.py (MockWorkerProxy.StartSimulation): Use more realistic values. (TestSimulationRun.test_run): Check the river_factor and stp_factor too (TestSimulationHelpers.test_factor): New. Test for the factor_and_unit_from_max helper function 2003-02-06 Bernhard Herzog Use bulk writes to write results and report the writing progress properly. * GreaterModel/Scheduler/simulation.py (Simulation.start_writing_results): Set up a somewhat different scheme to write the results. We now accumulate all res_stretch and res_disch records before starting to write them (Simulation.write_results): Rework the state machine for the new scheme. (Simulation.write_res_record): Handle all db operations now, including connect/deconnect. (Simulation.add_res_stretch, Simulation.add_res_disch) (Simulation.write_res_stretch, Simulation.write_res_disch): Don't write the records to the DB yet, put them into a list. Because of that rename the methods from write_* to add_* 2003-02-05 Bernhard Herzog * test/test_xmlrpc_session.py (TestXMLRPCSession.test_session_required_parameters): Adapt to model changes * test/test_pb_worker.py (TestPBWorker.test_session_required_parameters): Adapt to model changes * test/test_xmlrpc_simulation.py (TestXMLRPC.run_and_stop_simulation): Fix comment * test/test_simulation.py (MockScheduler.__init__): Fix typos in doc-string 2003-02-04 Frank Koormann * test/test_stochastic.py: New, test for stochastic helper methods. 2003-02-04 Frank Koormann * GreaterModel/Worker/model.py (RiverMode2): Refined MissingParameter method (since RiverMode2 uses estimate functions to fill gaps) * test/test_sessionmethods.py: Update to model changes. 2003-02-04 Frank Koormann * GreaterModel/Worker/model.py (RiverMode2): New, implements GREAT-ER River Model Mode 2. 2003-01-31 Frank Koormann Update Model System: Household Emission, Sewer Mode 1, Sewer Mode 2 * GreaterModel/Worker/model.py (Model.*Parameters): Consider Parameter Blocks with block_id == None as "always required". (HouseholdEmission): New model implementing household emissions. (SewerMode1): New model implementing sewer mode 1 (no elimination) (SewerMode2): New model implementing sewer mode 2 (percentage removal) (DischargeASMode1.compute, DischargeTFMode1.compute): Model parts now considered by Emission/Sewer removed. Concentration calculation zero save. * GreaterModel/Worker/graph.py (Graph.__init__): A discharge is always connected with sewer (S_) and household emission (H_) segment to the river network. * GreaterModel/Worker/session.py (session_required_parameters, session_stochastic_parameters, session_missing_parameters): Consider model descriptions with field_id == None as "always in use". * test/support.py (SchedulerProcess.__init__): Start Scheduler with port scan. * test/support_testdata.py (SessionDataMixin.uploadsession): Added MODE_SEWER flag. * test/mock_dbs/test_pb_worker.py (SimpleMockDB.create_session): Added MODE_SEWER flag. * test/test_graph.py (TestGraph.test_graphinstantiation): Added household emission and sewer segments to test case. * test/test_simulation.py (MockWorkerProxy.done): Updated virtual stretch ID ('A_1' -> 'V_1') 2003-01-30 Bernhard Herzog * GreaterModel/Worker/worker.py (Worker.return_result) (Worker.return_error): Refactor the calls of the deferred for start_calculation into their own methods. (Worker.iterate_calculation): Use the new methods. Handle the case where the calculation was stopped between the callLater call and the call of this method (Worker.stop_calculation): New. Actual implementation of the worker side of stop calculation * GreaterModel/Worker/workerpb.py (WorkerPerspective.remote_stop_calculation): PB interface to the worker's stop_calculation * GreaterModel/Scheduler/worker_proxy.py (WorkerProxy.StopSimulation): Worker proxy side of stop calculation * GreaterModel/Scheduler/simulation.py (Simulation.Stop): Add doc-string. Stop the worker if it's currently working. (Simulation.WorkerFinished): Add doc-string. Handle case of stopped worker. (Simulation.handle_error): Log the error explicitly * test/test_worker.py (TestWorker.test_worker_stop_calculation): New test for stop_calculation * test/test_worker_proxy.py (MockReference.callRemote): allow stop_calculation (TestWorkerProxy.test_stop_calculation): New test for the WorkerProxy's StopCalculation method * test/test_xmlrpc_simulation.py (TestXMLRPC.test_stop_simulation) (TestXMLRPC.test_starting_new_simulation_after_stop): New tests for stopping simulations (TestXMLRPC.run_and_stop_simulation): New. Helper method for the new tests 2003-01-30 Frank Koormann * test/test_graph.py (TestGraph.test_graphinstantiation): Update test case to new naming scheme. * test/test_calculation.py (TestCalculation): Updated test cases to fixed bug in River model. 2003-01-29 Frank Koormann * GreaterModel/Scheduler/simulation.py (Simulation.write_results): Virtual segments inserted for graph topology now are identified by 'V_'. These are now sorted out. * GreaterModel/Worker/graph.py (__init__): Naming changed: "artificial" to "virtual" Virtual stretches created by the pre-processing just hand through substance flux. Allow Discharges linked to a source segment. * GreaterModel/Worker/model.py: General: Implemented local settings for Mode 1 models. (RiverModel1.compute): Zero save implementation * test/test_calculation.py (TestModelHelper): New class to test model helper methods, currently eval_local_setting(). 2003-01-29 Bernhard Herzog * test/test_xmlrpc_scheduler.py (TestXMLRPCSchedulerShutdown.test_scheduler_shutdown) * test/support.py (SchedulerProcess.stop, WorkerProcess.stop): Use wait resp. waitpid only when it's available, i.e. on POSIX systems. The tests seem to work fine on w2000 without waiting for the subprocesses 2003-01-29 Bernhard Herzog * test/support.py (SchedulerProcess.__init__) (WorkerProcess.__init__): Change the default ports. They're now 1000 higher than the defaults used by start_scheduler and start_worker. This should avoid problems when running the tests while a GREAT-ER client is running with its own scheduler and worker. * GreaterModel/Scheduler/run.py (start_listeners): Always announce that a worker connected instead of making this depend on the test-port parameter. This new behavior should be OK for the standalone use at least where we only have one worker. 2003-01-28 Bernhard Herzog * GreaterModel/Scheduler/schedpb.py (ModelServerService.__init__): New. Extend inherited version with the announce_worker_file parameter. (ModelServerService.AnnounceWorker): If announce_worker_file is not None, write a message to it so that the parent process can determine when the worker has connected. (ModelPerspective.attached): Call the service's AnnounceWorker method * GreaterModel/Scheduler/run.py (start_listeners): Use file as the announce_worker_file parameter in tests, i.e. when test_port is given. * test/support.py (SchedulerProcess.wait_for_worker): New. Wait until the worker has connected. * test/test_xmlrpc_simulation.py (TestXMLRPC.setUp): Use wait_for_worker instead of sleep * test/test_xmlrpc_session.py (TestXMLRPCSession.setUp): Use wait_for_worker instead of sleep * test/test_xmlrpc_scheduler.py (TestXMLRPCSchedulerMethods.test_scheduler_status_one_worker): Remove one of two identical definitions of this method. (TestXMLRPCSchedulerMethods.test_scheduler_status_one_worker): Use wait_for_worker instead of sleep * test/test_pb_sched_worker.py (MyModelServerService.AnnounceWorker): Dummy implementation. 2003-01-28 Bernhard Herzog * GreaterModel/Scheduler/run.py (Options): New command line option, port-scan-range. (announce_running): Removed. This functionality is now in reactor_running (run): Refactored. Only the setup that hasn't got to do with listening is done in run now. The rest is done once the reactor is running in reactor_running (reactor_running): New. Set up the factories with start_listeners and print a message that the scheduler is running. (start_listeners): Start all factories depending on the command line options. (find_port_in_range): Helper function that tries to listen on a range of ports to find one that is not used yet. * test/support.py (SchedulerProcess.__init__): New parameter port_scan_range. New instance variables real_client_port and real_worker_port. (SchedulerProcess.start): Put the new parameter into the scheduler's command line (SchedulerProcess.start): Write the command line into the logfile. (SchedulerProcess.wait_for_message): New. Refactored the stdout parsing code out from the start method. The code also looks for the messages with the real worker and client ports now and binds them to the new instance variables. * test/test_scheduler_process.py: New. Test command line args of the scheduler process 2003-01-28 Frank Koormann * GreaterModel/Scheduler/simulation.py (Simulation.generate_numbers): Test "total_numbers > 0" to avoid division by zero. 2003-01-28 Bernhard Herzog * test/test_xmlrpc_session.py (TestXMLRPCSession.setUp) (TestXMLRPCSession.test_session_required_parameters) (TestXMLRPCSession.test_session_missing_parameters): Rename server to scheduler. (TestXMLRPCSession.tearDown): The worker stops automatically now when the scheduler closes the connection so stop the worker before the scheduler to avoid the error messages from trying to stop an already terminated worker. 2003-01-24 Jan-Oliver Wagner * GreaterModel/Scheduler/simulation.py: (Simulation.write_results): bugfix: da_db_commit was missing. 2003-01-24 Jan-Oliver Wagner * GreaterModel/Scheduler/simulation.py: id's of new record changed from 0 to -1. (Simulation.write_res_stretch): bugfix: new results ID now also set and stored for the session. 2003-01-24 Jan-Oliver Wagner * GreaterModel/Scheduler/simulation.py (Simulation.got_stochastic_parameters): bugfix: first cast mc-shots to float before casting it to int ('1E3' failed before). 2003-01-24 Frank Koormann * GreaterModel/Scheduler/simulation.py (Simulation.calculation_error): New, handle caclulation errors. (Simulation.UseWorker): addErrback * GreaterModel/Worker/worker.py (Worker.iterate_calculation): try/except added, in case of an error report it and log via twisted failure. * GreaterModel/Worker/model.py (RiverModel1.compute): Calculation of CSIMINTERN zero save. (eval_local_setting): New, evaluates if local settings overwrite defaults. (DischargeASMode1.compute): Use eval_local_setting() 2003-01-24 Jan-Oliver Wagner * GreaterModel/Scheduler/simulation.py (Simulation.got_stochastic_parameters): fixed two bugs: da_db_connect parameter sequence was wrong and da_get_sess should use parameter '%'. (Simulation.write_results): fixed a bugs: da_db_connect parameter sequence was wrong. 2003-01-23 Bernhard Herzog * GreaterModel/Worker/workerpb.py (Scheduler.connected): Register a callback for disconnects. (Scheduler.on_disconnect): Stop the worker. This is the callback registered in the connected method. 2003-01-23 Bernhard Herzog * GreaterModel/Worker/worker.py: Remove the "fake" calculation and use a real one: (Worker.start_calculation): Instantiate a Calculation (Worker.iterate_calculation): Iterate the calculation. * GreaterModel/Scheduler/simulation.py (Simulation.WorkerFinished): Now that the simulation returns actual results, start writing results when the worker is finished. (Simulation.start_writing_results, Simulation.write_results) (Simulation.write_res_record, Simulation.write_res_stretch) (Simulation.write_res_disch): New methods to handle writing of the results. * test/test_simulation.py (MockWorkerProxy.done): Return a fake result of the right structure. (TestSimulationRun.setUp): Remember the session record so that we can access its sess_id in test cases. (TestSimulationRun.test_run): Check the results were written into the database. (TestSimulationRun.test_run_delete_old_results): New. Test whether old results get deleted properly. * test/test_worker.py: New. Test the worker class. 2003-01-22 Frank Koormann * GreaterModel/Worker/model.py (Model.MissingParameters): Consider also existing parameters with empty value-string as missing. * test/test_sessionmethods.py (TestParameters.test_missing_parameters): Update to new functionality. 2003-01-22 Frank Koormann * GreaterModel/Worker/calculation.py: New,implements Calculation object. * GreaterModel/Worker/session.py: (start_calculation): Removed, functionality moved to Calculation object. (stop_calculation): Removed, no longer needed. * GreaterModel/Worker/stochasticresult.py (StochasticResult.get): Zero-save implementation of calculation of standard deviation. * test/test_calculation.py: New, tests for Calculation object. * test/test_parameter.py: Extended doc string commenting the lognormal test. * test/test_sessionmethods.py (TestSimulation): Removed. Test now implemented in test/test_calculation.py. * test/test_stochasticresult.py: Updated tests for new StochasticResult implementation. 2003-01-21 Bernhard Herzog * GreaterModel/Worker/workerpb.py (Options): Add host option (run): Use the host option to connect to the right host. 2003-01-21 Bernhard Herzog implement and document scheduler.status and scheduler.shutdown * doc/great-er_model_scheduler.tex (scheduler.status) (scheduler.shutdown): Add documentation for scheduler.status and scheduler.shutdown * GreaterModel/Scheduler/run.py (Options): Add allow-shutdown flag. (run): Pass allow-shutdown to SchedulerXMLRPC * GreaterModel/Scheduler/schedxmlrpc.py (SchedulerXMLRPC.__init__): New, to implement the allow_shutdown parameter (SchedulerXMLRPC.xmlrpc_scheduler_status): New. XMLRPC interface part of scheduler.status (SchedulerXMLRPC.xmlrpc_scheduler_shutdown): New. XMLRPC interface part of scheduler.shutdown * GreaterModel/Scheduler/scheduler.py (Scheduler.Status): New. Real implementation of scheduler.status * test/test_scheduler.py (TestScheduler.test_scheduler_status): New. Test for Scheduler.Status() * test/test_xmlrpc_scheduler.py: New. Tests for the XMLRPC interface of scheduler.status and scheduler.shutdown * test/support.py (SchedulerProcess.__init__): Add allow_shutdown parameter (SchedulerProcess.start): Pass allow_shutdown to the scheduler 2003-01-21 Bernhard Herzog Implement report_status * GreaterModel/Scheduler/schedpb.py (ModelPerspective.perspective_report_status): New. The PB interface to report_status * GreaterModel/Scheduler/scheduler.py (Scheduler.WorkerReportStatus): New. The scheduler's part of report_status * GreaterModel/Scheduler/worker_proxy.py (WorkerProxy.ReportStatus): New. Pass the progress to the simulation for statte "running" * GreaterModel/Scheduler/simulation.py (Simulation.WorkerProgress): New. Update the simulation progress. * GreaterModel/Worker/workerpb.py (Scheduler.failure): Log the failure instead of writing it to stdout (Scheduler.ReportStatus): New. Call report_status via PB. (WorkerPerspective.__init__): Pass the scheduler to the worker instance (WorkerPerspective.remote_start_calculation) (WorkerPerspective.remote_session_stochastic_parameters) (WorkerPerspective.remote_session_required_parameters) (WorkerPerspective.remote_session_missing_parameters): Log that these methods have been called. * GreaterModel/Worker/worker.py (Worker.__init__): New parameter scheduler. (Worker.iterate_calculation): Call report_status through the scheduler. 2003-01-21 Bernhard Herzog * test/test_xmlrpc_session.py (TestXMLRPCSession.test_session_calls) (TestXMLRPCSession.test_session_required_parameters) (TestXMLRPCSession.test_session_missing_parameters): Split test_session_calls into two tests test_session_required_parameters and test_session_missing_parameters (TestXMLRPCSession.test_session_required_parameters) (TestXMLRPCSession.test_session_missing_parameters): Update test result values because of changes in the model. * test/test_pb_worker.py (TestPBWorker.test_session_required_parameters) (TestPBWorker.test_session_missing_parameters) (TestPBWorker.test_session_stochastic_parameters): Add/update doc-strings. Update the test result values because of changes in the model. * test/mock_dbs/test_pb_worker.py (RivernetMockDB.__init__): Add DEFAULT_CONSUMPTION as in test/test_sessionmethods.py. * test/support.py (greater_dirs): Removed. It was only needed because the MockDB base class used to be in GREAT-ER/test. (extra_dirs): Removed the greater_dirs calls 2003-01-20 Bernhard Herzog * test/mock_dbs/test_pb_worker.py (SimpleMockDB.create_session): The MockDB methods return the new value of the record now. The do not modify the record in place anymore. * test/test_simulation.py (MockWorkerProxy.__init__) (MockWorkerProxy.StartSimulation): StartSimulation has to return a Deferred now. (MockWorkerProxy.done): Call the deferred's callback 2003-01-20 Frank Koormann * test/test_graph.py: Initialize key STOCHASTICS of data dictionary with None. * test/test_parameter.py: Redefined reaction if parameter is not listed under the currently distributed parameters. 2003-01-20 Frank Koormann Substantial update of Worker: Model implementation and supporting framework: * GreaterModel/Worker/session.py (start_calculation): Full implementation (replaces prior stub) * GreaterModel/Worker/model.py (Model.setup, Model.get_results): New (RiverModel1, DischargeASMode1, DischargeTFMode1): Implement GREAT-ER 1.0 Mode 1 models using parameter wrapper. * GreaterModel/Worker/stochastic.py: New. Stochastic helper functions. * GreaterModel/Worker/parameter.py: New. Parameter Encapsulation. * GreaterModel/Worker/graph.py (Graph.__init__): Get named attributes of models during instantiation instead of storing them. New: Call model method setup (with segment ID) * GreaterModel/Worker/segment.py (Segment.compute): new parameter shot_num. (Segment.get_results): New * GreaterModel/exception.py (GREATER_StochasticParameter): New error class. * GreaterModel/Worker/stochasticresult.py: Added doc-strings. (StochasticResult.combine): New, combine two object contents. * test/support_testdata.py (uploaddischarges): Updated flowdom. * test/test_graph.py: Tests updated to new model requirements. * test/test_sessionmethods.py: Tests updated to new model requirements. (TestSimulation): New, testing deterministic simulation run. Results were proven using a pocket calculator. * test/test_stochasticresult.py (TestStochasticResult.test_07_combine_values): Test combining of stochastic results. * test/test_parameter.py: New. Tests for parameter wrapper. 2003-01-14 Bernhard Herzog Implement a queue of simulations waiting for a free worker so that new simulations can be started even when there is currently no worker free. * GreaterModel/Scheduler/scheduler.py (Scheduler.__init__): New instance variable simulation_queue (Scheduler.handle_simulation_queue): New method to give an available worker if any to a waiting simulation. (Scheduler.RegisterWorker, Scheduler.ReleaseWorker): Call handle_simulation_queue (Scheduler.AcquireWorker): Add the simulation parameter. Put the simulation in the queue and call handle_simulation_queue in case a worker is currently available. * GreaterModel/Scheduler/simulation.py (Simulation.Run): Use the scheduler's worker_for_session_query method to get the worker for the SessionStochasticParameters call. (Simulation.run_simulation): Acquire the 'real' worker. (Simulation.UseWorker): New method. It's called when the worker is available. Actually start the calculation. * test/test_scheduler.py (MockSimulation.Run) (MockSimulation.UseWorker): Adapt to the new way of acquiring workers. (MockSimulation.Stop): Release the worker (TestScheduler.test_acquire_worker): Removed. The test doesn't make sense with the new scheme of acquiring workers. (TestScheduler.test_waiting_for_worker): New. Test the new scheme of acquiring workers. (TestScheduler.test_simulation): Adapt to changes in the MockSimulation implementation. * test/test_simulation.py (MockScheduler.worker_for_session_query): New. (MockScheduler.AcquireWorker): Adapt to new scheme of qorker acquisition. * test/test_xmlrpc_simulation.py (TestXMLRPC.test_two_parallel_simulation_calls): New. Test whether it's possible to request two simulations at almost the same time. * test/mock_dbs/test_pb_worker.py (SimpleMockDB.__init__, SimpleMockDB.create_session): Split the creation of the session into a separate method so that derived classes can easily create more sessions. * test/mock_dbs/test_xmlrpc_simulation.py (SimulationMockDB.__init__): Create two sessions. 2003-01-14 Bernhard Herzog * test/mock_dbs/test_xmlrpc_simulation.py: New. Mock DB for test_xmlrpc_simulation 2003-01-13 Bernhard Herzog * GreaterModel/Scheduler/simulation.py (Simulation.WorkerFinished): Release the worker. * GreaterModel/Scheduler/scheduler.py (Scheduler.ReleaseWorker): New method to release a worker that was previously acquired with AcquireWorker * GreaterModel/Scheduler/worker_proxy.py (WorkerProxy.Reference): New method to return the reference wrapped by the proxy * test/test_xmlrpc_simulation.py (TestXMLRPC.run_simulation): Separate the actual test code into this new method. (TestXMLRPC.test_one_simulation) (TestXMLRPC.test_two_sequential_simulations): New tests that simply use run_simulation once resp. twice. (TestXMLRPC.test_simulation_calls): This is now test_one_simulation * test/test_worker_proxy.py (TestWorkerProxy.execute_test): Add doc-string (TestWorkerProxy.test_reference): New. Test the WorkerProxy.Reference method * test/test_simulation.py (MockScheduler.ReleaseWorker): New. Record that the method has been called. (TestSimulationRun.test_run): Check whether the ReleaseWorker method has been called. * test/test_scheduler.py (TestSchedulerSessionMethods.test_missing_parameters) (TestSchedulerSessionMethods.test_missing_parameters_no_worker): Correct method name in doc-string 2003-01-13 Bernhard Herzog * GreaterModel/Scheduler/scheduler.py (Scheduler.worker_for_session_query): Raise NoWorkerAvailable instead of RuntimeError if no worker is available. * test/test_scheduler.py (RecordingMockWorkerReference): New helper class (TestSchedulerSessionMethods): New class with test cases for the scheduler's session parameters methods. 2003-01-13 Bernhard Herzog * test/test_simulation.py (MockWorkerProxy.SessionStochasticParameters) (TestSimulationRun.test_run) * test/test_sessionmethods.py (TestParameters.test_stochastic_parameters) * test/test_pb_worker.py (TestPBWorker.test_session_stochastic_parameters) * GreaterModel/Worker/model.py (Model.StochasticParameters) * GreaterModel/Scheduler/simulation.py (known_distributions): The log normal distribution is spelled "lognormal" 2003-01-13 Bernhard Herzog * GreaterModel/Scheduler/worker_proxy.py (WorkerProxy.SessionRequiredParameters) (WorkerProxy.SessionMissingParameters) (WorkerProxy.SessionStochasticParameters): Implement correctly. * test/test_worker_proxy.py (MockReference.callRemote): Allow more remote methods (TestWorkerProxy.execute_test): Common implementation of the test cases in TestWorkerProxy (TestWorkerProxy.test_worker_proxy) (TestWorkerProxy.test_start_calculation): Renamed test_worker_proxy to test_start_calculation. (TestWorkerProxy.test_required_parameters) (TestWorkerProxy.test_missing_parameters) (TestWorkerProxy.test_stochastic_parameters): New. 2003-01-13 Bernhard Herzog * GreaterModel/Scheduler/run.py (run, Options): New parameter --db-api-module to specify which DB api implementation to use. * GreaterModel/Worker/workerpb.py (WorkerPerspective.remote_start_calculation): Add the parameter for the random numbers * GreaterModel/Worker/worker.py (Worker.start_calculation): Add the parameter for the random numbers * GreaterModel/Scheduler/exceptions.py: New module with Scheduler specific exceptions. * GreaterModel/Scheduler/worker_proxy.py (WorkerProxy.StartSimulation): Implement this properly. * GreaterModel/Scheduler/simulation.py (Simulation.generate_numbers): Compute the progress correctly. (Simulation.run_simulation): The worker proxy's StartSimulation method returns a deferred * GreaterModel/Scheduler/scheduler.py (Scheduler.__init__): New parameter simulation_class to allow a different Simulation class to be used. Mainly for test cases. Introduce a new instance variable free_workers to keep track of which worker are available for simulations. (Scheduler.RegisterWorker): Initially a worker is available for simulations (Scheduler.UnregisterWorker): Remove the worker from the free_workers dict as well. (Scheduler.AcquireWorker): New. Return a free worker for a simulation (Scheduler.StartSimulation): Use simulation_class to instantiate the simulation. Actually start the simulation. * test/support.py (SchedulerServer, SchedulerProcess): Rename SchedulerServer to SchedulerProcess. * test/test_xmlrpc_simulation.py (TestXMLRPC.setUp) * test/test_xmlrpc_session.py (TestXMLRPCSession.setUp) * test/test_xmlrpc.py (TestXMLRPC.setUp): Update refereces to SchedulerServer accordingly. * test/support.py (SchedulerProcess.__init__): Add the db_api_module parameter (SchedulerProcess.start): Pass the db_api_module parameter to the sub-process * test/test_xmlrpc_simulation.py (TestXMLRPC.setUp): Also start a worker. (TestXMLRPC.tearDown): Stop the worker as well. (TestXMLRPC.test_simulation_calls): Check whether the final status is "end_complete" * test/test_worker_proxy.py (MockSimulation): Complete new implementation. All we need now is the user/db/session information. (MockReference): Complete new implementation. Record which remote methods are being called. (TestWorkerProxy): The tests don't require a reactor any more. (TestWorkerProxy.setUp): Initialize the called_methods instance variable (TestWorkerProxy.test_worker_proxy): A better test for the StartSimulation method that tests whether the correct remote method is being called and tests whether StartSimulation returns a deferred * test/test_scheduler.py (MockSimulation, MockWorkerReference): New classes to help testing the scheduler (TestScheduler.test_acquire_worker): New test case for the AcquireWorker method (TestScheduler.test_simulation): The test requires a worker now 2003-01-10 Bernhard Herzog * test/test_simulation.py (SimulationMockDB.__init__) (TestSimulationRun.setUp) * test/test_pb_worker.py (MockModelPerspective.attached) * GreaterModel/Scheduler/simulation.py (Simulation.got_stochastic_parameters): Use correct second parameter of da_get_sess 2003-01-09 Bernhard Herzog * test/test_simulation.py (TestSimulation, SimulationReactorTest): Split the parts TestSimulation into a new class SimulationReactorTest that are useful to other simulation tests that run a reactor. Derive TestSimulation from SimulationReactorTest (TestSimulationErrors): New class with test case for the error and end-state handling. (SimulationMockDB): New mock DB with a value for MC_SHOTS. (TestSimulationRun.test_run): Extend to use SimulationMockDB to test whether the simulation class uses the correct number of shots * GreaterModel/Scheduler/simulation.py (Simulation.set_state): Add doc-string and make sure that the simulation stays in an end-state once it has entered one. (Simulation.Run): Move the set_state call to before the worker call so that in tests where the worker proxy has the result immediately the simulation is in the right state. (Simulation.handle_error): Print the traceback to ease debugging. (Simulation.got_stochastic_parameters): Determine the number of random shots from the database (Simulation.got_stochastic_parameters): Make random_numbers be a dictionary so that the dictionary doesn't have to be created explicitly (Simulation.generate_numbers): self.random_numbers is now a dictionary (Simulation.run_simulation): self.random_numbers is now a dictionary so we can use it directly. * test/test_xmlrpc_session.py (TestXMLRPCSession.test_simulation_calls) (TestXMLRPCSession.test_session_calls): Rename from test_simulation_calls to test_session_calls as that's what the test is about. * test/test_simulation.py (TestSimulationRun.test_run): Two more checks for the random numbers 2003-01-08 Bernhard Herzog * test/test_simulation.py (MockWorkerProxy.StartSimulation): New parameter stochastic_parameters. Pass this to the testcase so that it can be checked. (MockWorkerProxy.SessionStochasticParameters): New. Mock implementation (TestSimulationRun.setUp): Add doc-string and init the stochastic parameters variable (TestSimulationRun.stochastic_parameters): New. For the mock worker proxy to record the random numbers. (TestSimulationRun.test_run): Check the random numbers. * GreaterModel/Scheduler/worker_proxy.py (WorkerProxy.SessionRequiredParameters) (WorkerProxy.SessionMissingParameters): Update the doc-string to document that the return values are deferreds. (WorkerProxy.SessionStochasticParameters): New method to call session_stochastic_parameters * GreaterModel/Scheduler/simulation.py (known_distributions): New. Mapping to define the known distributions for the random number computation (Simulation.RandomNumbers): Removed. The real code is now got_stochastic_parameters (Simulation.got_stochastic_parameters): New. Start generating the random numbers (Simulation.generate_numbers): Modified to be the real code for the random number generation. (Simulation.run_simulation): New. Called when the random numbers are complete to start the actual simulation. (Simulation.handle_error): New. Errback for deferred worker calls. (Simulation.Run): Compute the random numbers first. 2003-01-08 Frank Koormann * doc/great-er_model_worker.tex: Method description moved here from chapter Scheduler. The model details will later be documented in a seperate document: GREAT-ER Model Description. Updated definition of start_calculation(). * doc/great-er_model_scheduler.tex: Method description of worker moved to the chapter Worker. * doc/great-er_model_specification.tex: Reordering of sections, added GNU FDL. * doc/gnufdl.tex: New 2003-01-07 Bernhard Herzog Implement the Scheduler's XMLRPC interface for session.missing_parameters: * GreaterModel/Scheduler/schedxmlrpc.py (SchedulerXMLRPC.xmlrpc_session_missing_parameters): New. * GreaterModel/Scheduler/scheduler.py (Scheduler.SessionMissingParameters): New. * GreaterModel/Scheduler/worker_proxy.py (WorkerProxy.SessionMissingParameters): New. * test/test_xmlrpc_session.py (TestXMLRPCSession.test_simulation_calls): Add the test for missing_parameters 2003-01-07 Bernhard Herzog Implement the Worker's PB interface for session_required_parameters * GreaterModel/Worker/worker.py (Worker.session_required_parameters): New. * GreaterModel/Worker/workerpb.py (WorkerPerspective.remote_session_missing_parameters): New. * test/test_pb_worker.py (TestPBWorker.test_session_missing_parameters): New. The test case 2003-01-07 Bernhard Herzog * GreaterModel/Worker/worker.py (Worker.session_stochastic_parameters): Replace the dummy code with the real one. (Worker.session_required_parameters): Correct the doc-string * test/mock_dbs/test_pb_worker.py (SimpleMockDB.__init__): Use existing catch_id and subst_id. (RivernetMockDB.__init__): New class for a DB with a river network (dbm, simple_db, river_db): db renamed to simple_db. river_db is an instance of RivernetMockDB * test/test_xmlrpc_session.py (TestXMLRPCSession.setUp): The name of the mock db object has changed * test/test_pb_worker.py (MockModelPerspective.attached) (MockModelServerService.__init__) (TestPBWorker.listen) : Make the Mock objects a bit more flexible so that we can use the same classes for different worker methods (TestPBWorker.setUp, TestPBWorker.tearDown): The worker is now instantiated in the individual tests because different parameters have to be passed (TestPBWorker.test, TestPBWorker.test_session_required_parameters): Rename test to test_session_required_parameters (TestPBWorker.test_session_required_parameters): Adapt to changes above changes. (TestPBWorker.test_session_stochastic_parameters): New test for session_stochastic_parameters. * test/test_simulation.py (TestSimulationRun): Remove some unused code 2003-01-07 Frank Koormann * GreaterModel/Worker/graph.py: Fixed type of stretch_id and related ids (must be long). * test/support_testdata.py: Fixed type of stretch_id and related ids (must be long). * test/test_graph.py (test_graphintegritycheck_bifurcation, test_graphintegritycheck_doubleroot): New tests. 2003-01-07 Frank Koormann * GreaterModel/Worker/graph.py: Added root segment search. Fixed import path for exception. * GreaterModel/Worker/segment.py: Added root attribute needed for root segment search. * GreaterModel/Worker/session.py: Fixed import path for exception. * GreaterModel/exception.py: Added Class GREATER_NetworkIntegrity. 2003-01-07 Frank Koormann * GreaterModel/exception.py: New. GREAT-ER Model specific exceptions. * GreaterModel/Worker/graph.py: New. Graph representing the river network and linked discharges. Provides also a list of segments representing a post-order traversal over the graph. * GreaterModel/Worker/segment.py: New. Segment classes to build the graph * GreaterModel/Worker/model.py: Added compartment info to ModelList. New method get_required_parameter_list(). Doc-strings added. * GreaterModel/Worker/session.py: Consider ModelList compartment info. New method check_model_selection(). Doc-strings added. * test/test_graph.py: New, test graph instantiation. * test/support_testdata.py (CatchmentDataMixin.uploaddischarges): Added discharge classes. * test/test_sessionmethods.py, test/test_stochasticresult.py: Added doc-strings. 2003-01-07 Bernhard Herzog * test/test_pb_worker.py (TestPBWorker.setUp): Remove some useless debug code. 2003-01-06 Bernhard Herzog Turn some directories into Python Packages: * GreaterModel/__init__.py, GreaterModel/Worker/__init__.py: New. Turn GreaterModel and GreaterModel/Worker into python packages The beginnings of the scheduler corresponding tests: * GreaterModel/Scheduler/run.py: Main entry point for the scheduler * GreaterModel/Scheduler/scheduler.py: Main class of the scheduler * GreaterModel/Scheduler/simulation.py: Class to represent a simulation * GreaterModel/Scheduler/schedxmlrpc.py: XML-RPC interface of the scheduler * GreaterModel/Scheduler/schedpb.py: Perspective Broker interface of the Scheduler * GreaterModel/Scheduler/worker.py: The scheduler's representation of a worker * test/test_scheduler.py: Tests for the Scheduler class * test/test_simulation.py: Tests for the Simulation class * test/test_xmlrpc_simulation.py: Test running a simulation via XMLRPC calls to the model server * test/test_xmlrpc.py: Test the XMLRPC interface of the model server * test/test_xmlrpc_model.py: Test the model parameter queries through XMLRPC * test/test_worker_proxy.py: Test cases for the scheduler's representation of a worker. * test/test_pb_sched_worker.py: Test cases for the perspective broker communication between scheduler and worker New files for the Worker and corresponding tests: * GreaterModel/Worker/worker.py: The main class for the worker process * GreaterModel/Worker/workerpb.py: Perspective Broker interface of the worker * GreaterModel/Worker/testsupport.py: Test case support code. * test/test_pb_worker.py: Test the Perspective Broker interface of the worker process Other test changes: * test/runtests.py: The project is GREAT-ERModel not simply GREAT-ER * test/reactormixins.py: New. Mixin classes for test cases that use Twisted reactors * test/support.py: Lots of new support code for the added testcases * test/test_stochasticresult.py, test/test_sessionmethods.py: Adapt to support changes. Worker now is a package Other new files * start_scheduler.py, start_worker.py: Scripts to start the scheduler reps. the worker * Lib/xmlrpclib.py: New. Fredrik Lundh's XMLRPC library. 2003-01-04 Frank Koormann * test/test_sessionmethods.py (TestParameters): Moved test session data setup to support_testdata.py * test/support_testdata.py (SessionDataMixin): New class providing method to load a test session into the test database: uploadsession(). 2003-01-04 Frank Koormann * test/support_testdata.py: New, data structures for river network and discharges to test GREAT-ER Model Worker. * test/runtests.py: New, run all tests for GREAT-ER Model * test/test_sessionmethods.py (test_missing_parameters, test_stochastic_parameters): New 2003-01-02 Frank Koormann * GreaterModel/Worker/stochasticresult.py: New, provide class to collect stochastic simulation results. * test/test_stochasticresult.py: New, test for class StochasticResult. * test/support.py (FloatComparisonMixin): Added class to perform float comparisons. 2003-01-02 Frank Koormann * GreaterModel/Worker/model.py, GreaterModel/Worker/data.py, GreaterModel/Worker/session.py: New modules implementing the worker core functionality. * test/support.py: Support classes and function for the test suite. * test/test_sessionmethods.py: Tests for the session methods. 2002-12-19 Frank Koormann * doc/great-er_model_concept.tex: Updates to workers required parameter list description. * doc/great-er_model_scheduler.tex: Reformatting and update of method descriptions. * great-er_model_specification.tex: New revision date: 19.12.2002 2002-12-11 Bernhard Herzog * doc/great-er_model_specification.tex: Define a new environment for method descriptions, methoddesc * doc/great-er_model_scheduler.tex: Add description of the protocol between scheduler and worker. Use the methoddesc environment for method descriptions. 2002-12-10 Jan-Oliver Wagner * doc/great-er_model_scheduler.tex: added API for required and missing parameters. 2002-12-03 Frank Koormann * doc/great-er_model_concept.tex: Moved Parameter Requirement Information as subsection into new section Parameter Information. Added Parameter Distribution Information and Information Storage subsections. 2002-11-28 Jan-Oliver Wagner * doc/great-er_model_components.sk: Module Dialogs is now only linked to the User Interface. The original idea of using the Dialogs from more than one UI is no more relevant since the ArcView approach is deprecated. Furthermore, the implementation design now draws the line with the concept of SciParam. * doc/great-er_model_component_details.sk: Removed Dialogs module since it does not link to anything else than the user interface. * doc/great-er_model_concept.tex: Added section on Parameter Requirement Information. 2002-11-21 Bernhard Herzog * doc/great-er_model_scheduler.tex: Flesh out the client-scheduler protocol and add a section on open issues. * doc/great-er_model_specification.tex (\APIMethod): remove the space after #1 2002-11-20 Frank Koormann * doc/structure_great-er_1.sk doc/structure_great-er_2.sk, doc/great-er_model_concept.tex: New section on system segmentation. * doc/Makefile: Added dependencies for new diagrams. 2002-11-13 Bernhard Herzog * doc/great-er_model_specification.tex: Split the chapters into separate tex files and include them here. * doc/great-er_model_worker.tex, doc/great-er_model_scheduler.tex, doc/great-er_model_concept.tex: New files for the individual chapters * doc/Makefile: Make the new files dependencies of the dvi 2002-11-12 Frank Koormann * ChangeLog started * doc/Makefile, doc/brieflogo.eps, doc/great-er_model_specification.tex, doc/great-er_model_components.sk, great-er_model_component_details.sk: Initial version: cvs import -m "Imported sources" GREAT-ERModel/ intevation start