\chapter{Scheduler}
\label{chap:scheduler}

The scheduler manages all simulations in GREAT-ER 2. The clients---the
web-server and the user interface---connect to the server to start and
stop simulations, to query the current status of a simulation or to
retrieve information about the supported models.

The actual simulations are run by separate worker processes described in
chapter \ref{chap:worker}. A worker works for only one scheduler
process.

The scheduler can manage multiple concurrent simulations for different
users.

A simulation run generally consist of the following steps:

\begin{enumerate}

  \item \label{sched:sim_start} The client calls the scheduler to start
  a simulation

  \item \label{sched:sim_delegate} The scheduler delegates the
  simulation to one or more worker processes.

  The worker processes may run on separate machines or on the same one.
  The scheduler can be configured to start worker processes on the same
  machine on demand up to a maximum number of processes.

  This step includes generation of (pseudo) random numbers for use in
  the Monte-Carlo shots in the workers. The random numbers are computed
  in the scheduler to make sure that there's no accidental correlation
  due to two workers using the same seeds and algorithms in the random
  number generators. It also allows the use of more systematic
  approaches like Latin-Hypercube and Hammersley whose implementation
  needs central coordination.

  \item \label{sched:partial_result} When a worker is finished with
  their part of a simulation they transfer the aggregated results back
  to the scheduler.

  \item \label{sched:aggregate} When all workers taking part in a
  simulation have completed their work the scheduler aggregates the
  results of the individual workers into the final simulation result.

  \item \label{sched:write_results} The simulation result is written
  into the database.

  \item \label{sched:sim_done} The client determines that the simulation
  is completed. The client has to actively poll for the state of the
  simulation so it can't be informed when the simulation has been
  finished (see section \ref{sec:client-scheduler-xmlrpc} for more
  information)

\end{enumerate}



\section{General Architecture}

The scheduler is implemented in Python using the Twisted framework for
network applications. Twisted is built around non-blocking sockets so
that a program implemented with Twisted can be run in a single thread
that is implemented in an event-oriented way.


\section{Database Access}
\label{sec:sched-database}

Both the scheduler and the workers need access to the database.
Therefore the clients pass the userid and password to use for database
access to the scheduler when starting a simulation. All database access
for a given simulation run is done with these settings through the
DB-API implementation.


\section{Client-Scheduler Network Protocol}
\label{sec:client-scheduler-protocol}

The protocol used between the clients and the scheduler process is
XMLRPC. The data is communicated through the database. Some of the
XMLRPC calls contain ids that refer to records in the database.

\subsection{XMLRPC Methods}
\label{sec:client-scheduler-xmlrpc}

The scheduler implements the following methods:

\begin{methoddesc}{simulation.start(dbname, username, password, sessid)}
\begin{tabular}{|p{0.2\textwidth}|p{0.75\textwidth}|} \hline
{\bf Name} & {\tt simulation.start} \\ \hline

Description &
	Start a new simulation and return its id. 
	\\ \hline 

In 	&
	\parameter{dbname}, \parameter{username}, 
	\parameter{password}, \parameter{sessid}
	\\ \hline

In definitions	&
	 \parbox[t]{\textwidth}{
	    \begin{tabular}[t]{p{0.18\textwidth}p{0.52\textwidth}}
		\parameter{dbname}: & DB instance \\
	 	\parameter{username}: & User name \\
		\parameter{password}: & Password \\
		\parameter{sessid}:   & Session identifier
		\end{tabular}
	  } \\ \hline
	
Return	&
	\parameter{simulationid}
	\\ \hline

Return definitions	&
	The returned simulationid is a signed integer which 
	identifies the simulation run and has to be used when 
	stopping the simulation or querying the status
	of the simulation.
	\\ \hline
\end{tabular}
\end{methoddesc}

\begin{methoddesc}{simulation.stop(simulationid)}
\begin{tabular}{|p{0.2\textwidth}|p{0.75\textwidth}|} \hline
{\bf Name} & {\tt simulation.stop} \\ \hline

Description	&
	Stop the simulation indicated by simulationid. 
	The method has no explicit return value, the calling client 
	can monitor the status of stopping the simulation by using the
	simulation.status call.
	\\ \hline

In	&
	\parameter{simulationid}
	\\ \hline

In defintions &
	simulationid has be the value returned by a simulation.start call.
	\\ \hline 

Return & -
	\\ \hline

Return definition & -
	\\ \hline 
\end{tabular}

\end{methoddesc}


\begin{methoddesc}{simulation.status(simulationid)}

\begin{tabular}{|p{0.2\textwidth}|p{0.75\textwidth}|} \hline
{\bf Name} & {\tt simulation.status} \\ \hline

Description	&
	Return the current status of the simulation indicated by simulationid.
	\\ \hline
In	&
	\parameter{simulationid}
	\\ \hline
In definitions	&
	\parameter{simulationid} references a running simulation.
	\\ \hline
Return	&
	List [\parameter{state}, \parameter{auxiliary value}]
	\\ \hline
Return definitions	&
	The string \parameter{state} indicates the state of a simulation, 
	the \parameter{auxiliary value} (which can be either a string or 
	an integer) and its meaning depend on the state:
\begin{description}

\item[\texttt{"initializing"}]

    The scheduler is initializing the simulation, i.e generating the 
    random numbers to use in the Monte-Carlo shots. 
    This state roughly corresponds to step
    \ref{sched:sim_delegate}.

    The auxiliary value is an integer. It is a progress indicator
    increasing from 0 (just started) to 100 (finished initializing).

\item[\texttt{"waiting"}]

    The simulation hasn't begun execution yet because all worker
    processes are already busy.

    The auxiliary value has no meaning and should be ignored.

\item[\texttt{"running"}]

    The simulation is currently running. This state roughly corresponds
    to the time between steps \ref{sched:sim_delegate} and
    \ref{sched:aggregate}.

    The auxiliary value is an integer. It is a progress indicator
    increasing from 0 (just started) to 100 (finished the simulation).

\item[\texttt{"writing\_results"}]

    The simulation is currently writing the simulation results into the
    database. This state corresponds to step \ref{sched:write_results}.

    The auxiliary value is an integer. It is a progress indicator
    increasing from 0 (just started) to 100 (finished writing).

\item[\texttt{"end\_complete"}]

    The simulation has finished the simulation successfully. The client
    can now read the results from the database. This state corresponds
    to step \ref{sched:sim_done}.

    The auxiliary value has no meaning and should be ignored.

\item[\texttt{"end\_error"}]

    An error has occurred during the simulation and the simulation has
    been aborted.

    The auxiliary value is a string with an error message.
    
\item[\texttt{"end\_stopped"}]

    The simulation has been stopped by the client (presumably by calling
    \APIMethod{simulation.stop()}).
  
    The auxiliary value has no meaning and should be ignored.

\end{description}

Note that all end states are denoted by strings that start with
\texttt{"end\_"} so the client can easily check whether the simulation
is still running.
\\ \hline
\end{tabular}
	
	Since the server doesn't call back to the client, the client is
	responsible for calling simulation.status repeatedly to find out when
	the simulation is finished. Of course, this shouldn't be done more often
	than a few times per second at most.
\end{methoddesc}

\begin{methoddesc}%
{session.required\_parameters(dbname, username, password, sessid)}

\begin{tabular}{|p{0.2\textwidth}|p{0.75\textwidth}|} \hline
{\bf Name} & {\tt session.required\_parameters} \\ \hline

Description	&
	Return a list of those parameters {\em that are required} for
	the corresponding session with its current selections e.g. on model
	modes. 
	\\ \hline

In 	&
	\parameter{dbname}, \parameter{username}, 
	\parameter{password}, \parameter{sessid}
	\\ \hline

In definitions	&
	 \parbox[t]{\textwidth}{
	    \begin{tabular}[t]{p{0.18\textwidth}p{0.52\textwidth}}
		\parameter{dbname}: & DB instance \\
	 	\parameter{username}: & User name \\
		\parameter{password}: & Password \\
		\parameter{sessid}:   & Session identifier
		\end{tabular}
	  } \\ \hline

Return	&
	List of tuples (\parameter{blockid}, \parameter{fieldid}) or error
	code.
	\\ \hline

Return definitions	&
	\parbox[t]{0.75\textwidth}{
	    \vspace{-2\topsep}
	    \begin{itemize}
		\item blockid and fieldid are used as in the 
			PARA\_TREE\_DEF\_TAB and in combination uniquely 
			identify a parameter in the database.
		\item If no parameter is required, an empty list is 
			to be returned.
		\item In case of an error, a simple string is returned 
			identifying the type of the problem:

			\begin{description}
				\item[\texttt{"error\_db"}]
				It was not possible to access the database 
				with the given access parameters.

				\item[\texttt{"error\_session"}]
				It was not possible to access the 
				given session.

				\item[\texttt{"error\_model"}]
				No worker was found that accepts to 
				process the request.

				\item[\texttt{"error\_model\_internal"}]
				The worker accepted, but ran into a problem.

			\end{description}
			The scheduler maintains a log-file with 
			more detailed information on occuring errors.
	    \end{itemize}
	}
	\\ \hline
\end{tabular}
\end{methoddesc}
\newpage

\begin{methoddesc}%
{session.missing\_parameters(dbname, username, password, sessid)}

\begin{tabular}{|p{0.2\textwidth}|p{0.75\textwidth}|} \hline
{\bf Name} & {\tt session.missing\_parameters} \\ \hline

Description	&
	Return a list of those parameters {\em that are required and missing
	(i.e. still set to unknown)} for the corresponding session with its
	current selections e.g. on model modes.
	\\ \hline

In 	&
	\parameter{dbname}, \parameter{username}, 
	\parameter{password}, \parameter{sessid}
	\\ \hline

In definitions	&
	 \parbox[t]{\textwidth}{
	    \begin{tabular}[t]{p{0.18\textwidth}p{0.52\textwidth}}
		\parameter{dbname}: & DB instance \\
	 	\parameter{username}: & User name \\
		\parameter{password}: & Password \\
		\parameter{sessid}:   & Session identifier
		\end{tabular}
	  } \\ \hline

Return	&
	List of tuples (\parameter{blockid}, \parameter{fieldid}) or error
	code.
	\\ \hline

Return definitions	&
	\parbox[t]{0.75\textwidth}{
	    \vspace{-2\topsep}
	    \begin{itemize}
		\item blockid and fieldid are used as in the 
			PARA\_TREE\_DEF\_TAB and in combination uniquely 
			identify a parameter in the database.
		\item If no parameter is required, an empty list is 
			to be returned.
		\item In case of an error, a simple string is returned 
			identifying the type of the problem. The error 
			codes are identical to the ones of 
			\texttt{session.required\_parameters()}.
			
			The scheduler maintains a log-file with more 
			detailed information on occuring errors.
	    \end{itemize}
	    \vspace{-\topsep}
	}
	\\ \hline

\end{tabular}
\end{methoddesc}


\begin{methoddesc}%
{scheduler.status()}

\begin{tabular}{|p{0.2\textwidth}|p{0.75\textwidth}|} \hline
{\bf Name} & {\tt scheduler.status} \\ \hline
Description	&
	Return a struct describing the status of the scheduler.
	\\ \hline

In 	& - \\ \hline
In definitions	& - \\ \hline

Return	& An XMLRPC struct describing the status of the scheduler. \\ \hline

Return definitions	&
	 \parbox[t]{\textwidth}{
	   The struct will have at least these fields:\\
	   \begin{tabular}[t]{p{0.18\textwidth}p{0.52\textwidth}}
	     \parameter{num\_workers}: & Number of workers the
					 scheduler may use  \\
	   \end{tabular}
	 } \\ \hline

\end{tabular}

\end{methoddesc}


\begin{methoddesc}%
{scheduler.shutdown()}

\begin{tabular}{|p{0.2\textwidth}|p{0.75\textwidth}|} \hline
{\bf Name} & {\tt scheduler.shutdown} \\ \hline
Description	&

	Shutdown the scheduler. This method is not available on all
	schedulers. Typically it's only available if the scheduler was
	started by e.g. the GREAT-ER client on a single user machine. In
	that case the client starts the scheduler itself and thus also
	has to be able to shut it down.
\\ \hline

In 	& - \\ \hline
In definitions	& - \\ \hline

Return	& - \\ \hline
Return definitions & - \\ \hline

\end{tabular}

\end{methoddesc}

\newpage

\section{Scheduler-Worker Interface}
\label{sec:scheduler-worker-interface}

The scheduler and the worker processes communicate by the Perspective
Broker protocol of the twisted framework. The connection is initiated by
the worker process which provides the server with a perspective on
itself described in section \ref{subsec:worker-perspective} and it sees the
perspective of the scheduler described in secion
\ref{subsec:scheduler-perspective}.

\subsection{The Worker's Perspective on the Scheduler}
\label{subsec:scheduler-perspective}

The workers perspective on the scheduler has the following methods:

\begin{methoddesc}%
{report\_status(state, parameter)}

\begin{tabular}{|p{0.2\textwidth}|p{0.75\textwidth}|} \hline
{\bf Name} & {\tt report\_status} \\ \hline

Description	&  Tell the scheduler the current state of the calculations. 
		   The worker calls this method during a calculation 
		   started by the scheduler's call to the 
		   \texttt{start\_simulation} method. 
			\\ \hline
In		& {\tt state, parameter}	\\ \hline
In definitions	& 
	\parbox[t]{0.75\textwidth}{
	    \begin{tabular}[t]{p{0.18\textwidth}p{0.52\textwidth}}
		\parameter{state}: & status of the simulation \\
	 	\parameter{parameter}: & parameter to detail the status
					 information\\
	    \end{tabular}
	    \medskip 

		Simulation states are:
		\begin{description}
		    \item[\texttt{"running"}:] The simulation is running 
		    and the corresponding \parameter{parameter} is 
		    a percentage as an integer in the range 0 to 100 
		    indicating how much of the work has been completed.
		\end{description}
	  } \\ \hline
Return & - \\ \hline
Return definitions & - \\ \hline
\end{tabular}
\medskip 

Since the simulation start was issued using the perspective broker 
framework the end of the simulation has not to be reported to 
the scheduler explicitely.

\end{methoddesc}




\section{Open Issues}

\begin{itemize}

    \item The knowledge about the models should be concentrated in the
    workers. This means that there has to be a way for the scheduler to
    find out which models a worker supports and what parameters it
    needs.

    The scheduler may need to know something about the models as well
    because in the current scheme it has to supply the random numbers
    and the distribution to use may depend on the model.


    \item If a scheduler has to manage several simulations 
    simultaneously for different users it has frequently to 
    conncet and disconnect from the database to be able to use 
    different user ids, since the database connection and its 
    associated user id are global settings in the API implementation.

    \item The different ways the workers can be organized have to be
    determined. At least two different scenarios have to be supported:

    \begin{itemize}

        \item A single user system where the scheduler and the workers
        run on the same machine as the client user interface. Maybe
        there only has to be one worker process in this scenario.

	\item The workers run on separate machines on the network.
        
    \end{itemize}

    The strategies to use when distributing the computations to several
    workers have to be worked out.

    \item The protocol between the client and the scheduler is currently
    specified only with the graphical user interface in mind. It might
    turn out that the web-server has some needs that are not yet
    addressed by the specification.

\end{itemize}
