\chapter{Scheduler}
\label{chap:scheduler}

\section{Client-Scheduler Network Protocol}
\label{sec:client-scheduler-protocol}

The protocol used between the clients and the scheduler process is
XMLRPC. The data is communicated through the database. Some of the
XMLRPC calls contain ids that refer to records in the database.

\subsection{XMLRPC Methods}
\label{sec:client-scheduler-xmlrpc}

The scheduler implements the following methods:

\begin{methoddesc}{simulation.start(dbname, username, password, sessid)}
\begin{tabular}{|p{0.2\textwidth}|p{0.75\textwidth}|} \hline
{\bf Name} & {\tt simulation.start} \\ \hline

Description &
	Start a new simulation and return its id. 
	\\ \hline 

In 	&
	\parameter{dbname}, \parameter{username}, 
	\parameter{password}, \parameter{sessid}
	\\ \hline

In definitions	&
	 \parbox[t]{\textwidth}{
	    \begin{tabular}[t]{p{0.18\textwidth}p{0.52\textwidth}}
		\parameter{dbname}: & DB instance \\
	 	\parameter{username}: & User name \\
		\parameter{password}: & Password \\
		\parameter{sessid}:   & Session identifier
		\end{tabular}
	  } \\ \hline
	
Return	&
	\parameter{simulationid}
	\\ \hline

Return definitions	&
	The returned simulationid is a signed integer which 
	identifies the simulation run and has to be used when 
	stopping the simulation or querying the status
	of the simulation.
	\\ \hline
\end{tabular}
\end{methoddesc}

\begin{methoddesc}{simulation.stop(simulationid)}
\begin{tabular}{|p{0.2\textwidth}|p{0.75\textwidth}|} \hline
{\bf Name} & {\tt simulation.stop} \\ \hline

Description	&
	Stop the simulation indicated by simulationid. 
	The method has no explicit return value, the calling client 
	can monitor the status of stopping the simulation by using the
	simulation.status call.
	\\ \hline

In	&
	\parameter{simulationid}
	\\ \hline

In defintions &
	simulationid has be the value returned by a simulation.start call.
	\\ \hline 

Return & -
	\\ \hline

Return definition & -
	\\ \hline 
\end{tabular}

\end{methoddesc}


\begin{methoddesc}{simulation.status(simulationid)}

\begin{tabular}{|p{0.2\textwidth}|p{0.75\textwidth}|} \hline
{\bf Name} & {\tt simulation.status} \\ \hline

Description	&
	Return the current status of the simulation indicated by simulationid.
	\\ \hline
In	&
	\parameter{simulationid}
	\\ \hline
In definitions	&
	\parameter{simulationid} references a running simulation.
	\\ \hline
Return	&
	List [\parameter{state}, \parameter{auxiliary value}]
	\\ \hline
Return definitions	&
	The string \parameter{state} indicates the state of a simulation, 
	the \parameter{auxiliary value} (which can be either a string or 
	an integer) and its meaning depend on the state:
\begin{description}

\item[\texttt{"initializing"}]

    The scheduler is initializing the simulation, i.e generating the 
    random numbers to use in the Monte-Carlo shots. 
    This state roughly corresponds to step
    \ref{sched:sim_delegate}.

    The auxiliary value is an integer. It is a progress indicator
    increasing from 0 (just started) to 100 (finished initializing).

\item[\texttt{"waiting"}]

    The simulation hasn't begun execution yet because all worker
    processes are already busy.

    The auxiliary value has no meaning and should be ignored.

\item[\texttt{"running"}]

    The simulation is currently running. This state roughly corresponds
    to the time between steps \ref{sched:sim_delegate} and
    \ref{sched:aggregate}.

    The auxiliary value is an integer. It is a progress indicator
    increasing from 0 (just started) to 100 (finished the simulation).

\item[\texttt{"stopping"}]

    The simulation has told the workers to stop and is waiting for them
    to actually have stopped. This status is entered when the user told
    the scheduler to stop the simulation or when an error occurred in
    one of the workers.

    The auxiliary value has no meaning and should be ignored.

\item[\texttt{"writing\_results"}]

    The simulation is currently writing the simulation results into the
    database. This state corresponds to step \ref{sched:write_results}.

    The auxiliary value is an integer. It is a progress indicator
    increasing from 0 (just started) to 100 (finished writing).

\item[\texttt{"end\_complete"}]

    The simulation has finished the simulation successfully. The client
    can now read the results from the database. This state corresponds
    to step \ref{sched:sim_done}.

    The auxiliary value has no meaning and should be ignored.

\item[\texttt{"end\_error"}]

    An error has occurred during the simulation and the simulation has
    been aborted.

    The auxiliary value is a string with an error message.
    
\item[\texttt{"end\_stopped"}]

    The simulation has been stopped by the client (presumably by calling
    \APIMethod{simulation.stop()}).
  
    The auxiliary value has no meaning and should be ignored.

\end{description}

Note that all end states are denoted by strings that start with
\texttt{"end\_"} so the client can easily check whether the simulation
is still running.
\\ \hline
\end{tabular}
	
	Since the server doesn't call back to the client, the client is
	responsible for calling simulation.status repeatedly to find out when
	the simulation is finished. Of course, this shouldn't be done more often
	than a few times per second at most.
\end{methoddesc}

\begin{methoddesc}%
{session.required\_parameters(dbname, username, password, sessid)}

\begin{tabular}{|p{0.2\textwidth}|p{0.75\textwidth}|} \hline
{\bf Name} & {\tt session.required\_parameters} \\ \hline

Description	&
	Return a list of those parameters {\em that are required} for
	the corresponding session with its current selections e.g. on model
	modes. 
	\\ \hline

In 	&
	\parameter{dbname}, \parameter{username}, 
	\parameter{password}, \parameter{sessid}
	\\ \hline

In definitions	&
	 \parbox[t]{\textwidth}{
	    \begin{tabular}[t]{p{0.18\textwidth}p{0.52\textwidth}}
		\parameter{dbname}: & DB instance \\
	 	\parameter{username}: & User name \\
		\parameter{password}: & Password \\
		\parameter{sessid}:   & Session identifier
		\end{tabular}
	  } \\ \hline

Return	&
	List of tuples (\parameter{blockid}, \parameter{fieldid}) or error
	code.
	\\ \hline

Return definitions	&
	\parbox[t]{0.75\textwidth}{
	    \vspace{-2\topsep}
	    \begin{itemize}
		\item blockid and fieldid are used as in the 
			PARA\_TREE\_DEF\_TAB and in combination uniquely 
			identify a parameter in the database.
		\item If no parameter is required, an empty list is 
			to be returned.
	    \end{itemize}
	}
	\\ \hline
\end{tabular}
\end{methoddesc}

\begin{methoddesc}%
{session.missing\_parameters(dbname, username, password, sessid)}

\begin{tabular}{|p{0.2\textwidth}|p{0.75\textwidth}|} \hline
{\bf Name} & {\tt session.missing\_parameters} \\ \hline

Description	&
	Return a list of those parameters {\em that are required and missing
	(i.e. still set to unknown)} for the corresponding session with its
	current selections e.g. on model modes.
	\\ \hline

In 	&
	\parameter{dbname}, \parameter{username}, 
	\parameter{password}, \parameter{sessid}
	\\ \hline

In definitions	&
	 \parbox[t]{\textwidth}{
	    \begin{tabular}[t]{p{0.18\textwidth}p{0.52\textwidth}}
		\parameter{dbname}: & DB instance \\
	 	\parameter{username}: & User name \\
		\parameter{password}: & Password \\
		\parameter{sessid}:   & Session identifier
		\end{tabular}
	  } \\ \hline

Return	&
	List of tuples (\parameter{blockid}, \parameter{fieldid}) or error
	code.
	\\ \hline

Return definitions	&
	\parbox[t]{0.75\textwidth}{
	    \vspace{-2\topsep}
	    \begin{itemize}
		\item blockid and fieldid are used as in the 
			PARA\_TREE\_DEF\_TAB and in combination uniquely 
			identify a parameter in the database.
		\item If no parameter is required, an empty list is 
			to be returned.
	    \end{itemize}
	    \vspace{-\topsep}
	}
	\\ \hline

\end{tabular}
\end{methoddesc}


\begin{methoddesc}%
{scheduler.status()}

\begin{tabular}{|p{0.2\textwidth}|p{0.75\textwidth}|} \hline
{\bf Name} & {\tt scheduler.status} \\ \hline
Description	&
	Return a struct describing the status of the scheduler.
	\\ \hline

In 	& - \\ \hline
In definitions	& - \\ \hline

Return	& An XMLRPC struct describing the status of the scheduler. \\ \hline

Return definitions	&
	 \parbox[t]{\textwidth}{
	   The struct will have at least these fields:\\
	   \begin{tabular}[t]{p{0.18\textwidth}p{0.52\textwidth}}
	     \parameter{num\_workers}: & Number of workers the
					 scheduler may use  \\
	   \end{tabular}
	 } \\ \hline

\end{tabular}

\end{methoddesc}


\begin{methoddesc}%
{scheduler.shutdown()}

\begin{tabular}{|p{0.2\textwidth}|p{0.75\textwidth}|} \hline
{\bf Name} & {\tt scheduler.shutdown} \\ \hline
Description	&

	Shutdown the scheduler. This method is not available on all
	schedulers. Typically it's only available if the scheduler was
	started by e.g. the GREAT-ER client on a single user machine. In
	that case the client starts the scheduler itself and thus also
	has to be able to shut it down.
\\ \hline

In 	& - \\ \hline
In definitions	& - \\ \hline

Return	& - \\ \hline
Return definitions & - \\ \hline

\end{tabular}

\end{methoddesc}

\newpage

\section{Scheduler-Worker Interface}
\label{sec:scheduler-worker-interface}

The scheduler and the worker processes communicate by the Perspective
Broker protocol of the twisted framework. The connection is initiated by
the worker process which provides the server with a perspective on
itself described in section \ref{subsec:worker-perspective} and it sees the
perspective of the scheduler described in secion
\ref{subsec:scheduler-perspective}.

\subsection{The Worker's Perspective on the Scheduler}
\label{subsec:scheduler-perspective}

The workers perspective on the scheduler has the following methods:

\begin{methoddesc}%
{report\_status(simid, state, parameter)}

\begin{tabular}{|p{0.2\textwidth}|p{0.75\textwidth}|} \hline
{\bf Name} & {\tt report\_status} \\ \hline

Description	&  Tell the scheduler the current state of the calculations. 
		   The worker calls this method during a calculation 
		   started by the scheduler's call to the 
		   \texttt{start\_simulation} method. 
			\\ \hline
In		& {\tt simid, state, parameter}	\\ \hline
In definitions	& 
	\parbox[t]{0.75\textwidth}{
	    \begin{tabular}[t]{p{0.18\textwidth}p{0.52\textwidth}}
		\parameter{simid}: & id of the simulation the status
				     is reported for \\
		\parameter{state}: & status of the simulation \\
	 	\parameter{parameter}: & parameter to detail the status
					 information\\
	    \end{tabular}
	    \medskip 

		Simulation states are:
		\begin{description}
		    \item[\texttt{"running"}:] The simulation is running 
		    and the corresponding \parameter{parameter} is 
		    a percentage as an integer in the range 0 to 100 
		    indicating how much of the work has been completed.
		\end{description}
	  } \\ \hline
Return & - \\ \hline
Return definitions & - \\ \hline
\end{tabular}
\medskip 

Since the simulation start was issued using the perspective broker 
framework the end of the simulation has not to be reported to 
the scheduler explicitely.

\end{methoddesc}

\newpage

\section{Configuration}

The scheduler provides various options to configure the model system 
for an installation:
\\

\begin{tabular}{|p{0.2\textwidth}|p{0.75\textwidth}|} \hline
{\bf Option} & {\tt Description} \\ \hline
Client Port & The port to listen on for the XMLRPC calls from clients. \\ \hline
Worker Port & The port to listen on for worker connections. \\ \hline
Port Scan Range & This feature is used especially for the GREAT-ER Desktop
                  Version and should not be used for GREAT-ER Web or computing
                  servers! 
                  
                  If specified (with a number of ports to scan) 
                  the scheduler will search for free ports for client and
                  worker connections. The scan starts at the specified client
                  port.
                  \\ \hline
Allow Shutdown & This flag is used especially for the GREAT-ER Desktop
                  Version and should not be used for GREAT-ER Web or computing
                  servers! 
                  
                  If specified the scheduler can be stopped
                  with XMLRPC calls on the client port.\\ \hline

Admin Port & The port to listen on for administrative XMLRPC calls. 
             The scheduler can be shut down via XMLRPC calls on this port if
             the calls originating from localhost. 
             
             This feature is especially for
             the server usages of the GREAT-ER model systems since common users
             typically do not have accounts on dedicated servers.\\ \hline
Info File & If specified process name and admin port are stored in the
            file for later use (e.g. in a stop script, see below).

            This feature is especially for the server uses of the 
            GREAT-ER model systems.
            \\ \hline
Log File & If specified the scheduler writes extensive logs into the file for
            activity monitoring and problem analysis.\\ \hline
Test Port & This feature is especially for the unit testing during 
            development and can be used for special test commands. \\ \hline
DB-API Module & This feature is especially for the unit testing during 
            development and can be used to specify dummy test databases.
            \\ \hline

\end{tabular}

\newpage

\subsection{Starting the Scheduler}

The Scheduler can be started using the \verb+start_scheduler.py+
start script. The script provides several command line options to 
configure the scheduler:

{\scriptsize
\begin{verbatim}
Usage: python start_scheduler.py [options]
Options:
      --allow-shutdown    Allow the scheduler.shutdown XMLRPC call (default off)
  -p, --client-port=      The port to listen on for the XMLRPC calls 
                          [default: 10101]
  -w, --worker-port=      The port to listen on for the worker connections. If 
                          a port-scan-range is given this value will be ignored
                          and the worker port will be chosen automatically.
                          [default: 10102]
      --admin-port=       port to listen on for admin commands
      --info-file=        file to write info about the scheduler process to 
                          (pid and admin port).
      --test-port=        Port number to listen on for special commands from the
                          test-suite
      --port-scan-range=  number of ports starting from client-port to scan for
                          a free port. The ports used will be printed to stdout.
      --log-file=         name of file to log to
      --db-api-module=    Name of the db interface implementation module
                          [default: _dagreater]
      --help              Display this help and exit.
\end{verbatim}
}

\subsection{Stopping the Scheduler}

The scheduler can be stopped using the \verb+stop.py+
script. The script reads the information from a specified info file 
and stops the corresponding scheduler.

{\scriptsize
\begin{verbatim}
Usage: python stop.py [info-file]
\end{verbatim}
}

The GREAT-ER Desktop Client stops a locally started scheduler automatically on
exit.

% \section{Open Issues}
% 
% \begin{itemize}
% 
%     \item The knowledge about the models should be concentrated in the
%     workers. This means that there has to be a way for the scheduler to
%     find out which models a worker supports and what parameters it
%     needs.
% 
%     The scheduler may need to know something about the models as well
%     because in the current scheme it has to supply the random numbers
%     and the distribution to use may depend on the model.
% 
% 
%     \item If a scheduler has to manage several simulations 
%     simultaneously for different users it has frequently to 
%     conncet and disconnect from the database to be able to use 
%     different user ids, since the database connection and its 
%     associated user id are global settings in the API implementation.
% 
%     \item The different ways the workers can be organized have to be
%     determined. At least two different scenarios have to be supported:
% 
%     \begin{itemize}
% 
%         \item A single user system where the scheduler and the workers
%         run on the same machine as the client user interface. Maybe
%         there only has to be one worker process in this scenario.
% 
% 	\item The workers run on separate machines on the network.
%         
%     \end{itemize}
% 
%     The strategies to use when distributing the computations to several
%     workers have to be worked out.
% 
%     \item The protocol between the client and the scheduler is currently
%     specified only with the graphical user interface in mind. It might
%     turn out that the web-server has some needs that are not yet
%     addressed by the specification.
% 
% \end{itemize}
