\renewcommand{\documentID}{}

\chapter{Preprocessing Basics}

Besides the environmental model itself, the availability and quality of
environmental data mainly determine the validity of simulation results.
This and the following sections introduce general aspects and problems
of spatial data
for environmental exposure assessment. A proposed procedure is described for
how to deal with the collection, processing and quality assessment
of data considering
also the origin and purpose of readily available data.
The technical aspects will be focused on but it must also be stressed
that obtaining spatial data in the EU reveals serious coordination and
authorization problems.

An example of the proposed procedure is given in which
the digital river network is the backbone of the spatial data set
for "down-the-drain" exposure assessment.
The example is given for GREAT-ER but the methodology is equally applicable
for similar systems.\\

%introduction: 

The data required for the regional geo-referenced exposure assessment
is characterized by its complexity and inhomogeneity.
This is partly due to the fact that there are large amounts of single data.
But another fact plays an important role: the composition of such data
is not readily available. It is typically scattered among several
competent authorities and other data owners. In the past those
different parties
had seldom experienced the necessity to harmonize their data and hence
it is to be expected that all obtainable data are in several digital
formats and/or geographical resolutions. Inconsistent ID reference systems
are likely to be faced as well.
The approach to regional geo-referenced exposure assessment has created a
need for the coordination and composition of base data which are, though
collected and applied for various topics, nothing
more than {\em raw} data in respect to the new task.

The reason for this lies also in the way in which most geographic data
are used: for mapping purposes where structural integrity is much less
important than visual appearance.

The data composition for exposure assessment in river networks reveals
many problem-specific aspects. When one would set up a very special recipe for
certain data sources or data origins, this task would lack re-use
capabilities and also would not be considered adequate for other
comparable tasks.
In the framework of a simulation system for a growing number of catchments,
re-use capabilities require special attention to ensure the efficient creation
of many digital river network data sets.
The solution for data composition must be simple and open enough to
offer quick understanding, general applicability and also to enable
technical solutions to be adapted.

% how are GI data processed usually?
The usual steps of data composition and processing as a GIS task
are typically single manual steps of applying geographic methods and routines.
The required quality and structure of the final geographic data (i.e. maps)
is approached in a straight-forward manner. The data sources and intermediate
stages are often eliminated as soon as the latest version passes a quality
control. It is unusual for GIS users to automate the processing steps
because each new job reveals other aspects of processing which can not all
be foreseen. To make the results transparent and to avoid later wrong use,
all processing steps must be documented very carefully.
This is especially important when results are not only used
for reporting purposes but also for further processing and comparison
with other data sets processed in a similiar way.

A well defined intermediate stage of the data is a general approach
to separate the specific problems of raw data from the actual
data preparation. Setting up this intermediate state on
text-based formats increases transparency.
A set of pre-defined files requests data at an adequate level of
simplicity while at the same time being able to grant some
aspects of quality and completeness. 
Furthermore, a high degree of automation can be created starting from
pre-defined text files.

As an alternative intermediate state a pre-defined database management
system (DBMS) can also serve the task of separation. Such an approach
creates further needs in software requirements and skills to handle them.
Depending on the frequency of usage, the DBMS solution, once set up and
tested, can be more efficient than the solution based on simple text files.
The advantages arise from the DBMS environment which offers support for
distributed systems (including Internet applications) and high performance in
handling large data sets.
With its higher level of sophistication such a solution loses
self-explaining transparency which needs to be
compensated by comprehensive usage documentation and/or by dedicating
a human operator.
Proving its principle operability with the text files as an intermediate stage
is a recommended step before setting up a DBMS solution.
The two-step approach is therefore choosen for the GREAT-ER data processing.

\begin{center}
	\Fig{\includegraphics[width=130mm]{./images/puzzle_en.pdf}}{
		Two steps of data processing}
\end{center}

Inventing an intermediate stage for data consequently results in two steps
of data processing. In a first step, collected raw data are transformed into
the pre-defined intermediate form. A second and independent step then creates
final ready-to-use data from the intermediate stage.

The intial data processing is very much determined by the gap between demanded
data quality and formats and the present state of the raw data.
The required files have to be produced manually from the original raw data
or with special routines which probably have to be developed anew.

The final data processing is a well defined task, provided that
all required data
are collected and fit the demands of pre-defined formats. Of course,
the final environment of the geo-referenced exposure assessment tool
must also be defined precisely to make the data processing fully automatable.
In this context, the simulation method determines the structural
shape and order of the data while the user interface and data management
modules determine the actual formats of files.


The two steps of data processing are illustrated in figure 5.1. 
An example is given for an exposure assessment tool for river networks.
Three main states of the five data groups are shown. These data groups
cover any information that is related to space in one way or another.
For example, physico-chemical data of a substance are excluded
but the substance's market data are included as regionally resolved
information.

Another view of the finally produced data sets is shown
in figure 5.2. The data groups form layers which build the
visual interface available to the user.

\begin{center}
	\Fig{\includegraphics[width=140mm]{./images/layer.pdf}}{
		Layered view of data groups}
\end{center}

\newpage

\section{Data requirements and composition}

\subsection{Digital river network}

\subsubsection{General overview}

The digital river network describes the structure of the
river network in question.
At the very bottom it consists of lines (also referred to as
'polylines' or 'arcs'), each of which
represents a river segment.
Furthermore, it consists of nodes which represent the
point of connection of two segments.
Lines are expressed as a sequence of at least two coordinate pairs, nodes
are expressed as exactly one coordinate pair (figure 5.3b).

Lines have to be considered as vectors; the first point of the sequence
is the starting point and the last is the end point.
The vector direction is used to represent the flow direction (figure 5.3c).
The length of a segment can be computed from the coordinates.

Each transition from one segment to a subsequent one (sequence,
confluence or bifurcation) is defined by a node which contains
the information 'from' and 'to'.
The ID's of the upstream ('from') and the downstream ('to')
segments are stored.
The geographic position of a node is uniquely defined with the last point of
the upstream and the first point of the downstream segment. If these two
points are not identical, the two stretches are not connected and no corresponding
node exists.

Besides the persistent Arc-Node geometry model, a second, flexible segmentation
can be set on top of it. The two-dimensional geometry model lacks the ability
to precisely locate sites along the lines between two given coordinates.
This can be solved by introducing a one-dimensional measuring system with only
starting and ending information.

These one-dimensional geometric objects are called 'routes'
\label{routesystem}.
For digital river networks, each river is a 'route';
the whole network constitutes a 'route system'.
Any site along the rivers in a route system can be referenced by the route
name and the distance from the route start (e.g. River Rhine at kilometer 744
which is at D�sseldorf, Germany).
The start and end of routes are a  matter of definition.
In Germany, \label{flusskilometrierung} the distance reference of rivers
that lead into the ocean begin at the source.
The distance reference of tributaries begins at 
the point of confluence in reverse flow direction.
Standard GI Systems offer methods to create route
systems from an arc-node model. 
This must be a two-step routine where a raw route system is first created
based only on the information contained in the arc-node representation
(figure 5.3d).
The raw route system requires two further adjustments. First, the digitized
rivers do not necessarily start at the actual source and hence the starting
points of the routes must be set to the correct value for later consistent
referencing. Second, the length computed from the digitized lines depends
on the resolution and hence is always lower than the actual river length. 
A remeasurement routine that is provided with the actual length of a route,
is to be performed
to gain revised coordinates for any event along the stream.
The route system approach for exposure assessment was applied for intermediates
in the River Rhine catchment (Koormann, 1997 and Koorman et al. 1998) and
for a small tributary of the Rhine, the Itter catchment (Schulze, 1998 and
Schulze et al., 1999).

Fundamental to the creation of a digital
river network are maps of the corresponding area.
Traditional printed maps can be used as a basis but usually
such maps now exist in digital format.

\begin{center}
	\Fig{\includegraphics[width=100mm]{./DigitialRiverNetwork.pdf}}{
		Steps to create a digital river network}{
	\begin{minipage}[t]{10cm}
		Steps of creating a digital river network: (a) Reality (i.e. aerial
		photo or comparable basis). (b) Digitized network (crosses and dots =
		coordinates, dots = nodes, numbers = IDs of lines).
		(c) Direction of lines (vectors). (d) Representation as route system.
	\end{minipage}
	}
\end{center}

Digital maps can be created in two ways, either by collecting coordinates
directly with a Global Positioning System (GPS) or from aerial photographs.
While GPS data are naturally already geographic objects in vector format, 
pictures taken from planes or satellites offer information in
a raster format. I.e., surface waters consist of single points
(squares) and
hence do not contain information about e.g. flow direction, but on the
other hand may offer information about areal aspects (e.g. width of rivers,
shape and size of lakes) and possibly further properties (temperature, depth).
The raster information must be converted to vector data to gain a 
digital river network as described above.
This conversion can partially be automated but manual reworking by human
experts is unavoidable for quality assurance.

\subsubsection{Solution for GREAT-ER}

As a requirement for data, the arc-node model is preferred rather than
the route model. This is because, especially for more detailed river networks,
an assignement of river names and remeasurement become
problematic and time-consuming.

In summary, the following demands for a digital river network have been
defined:

\begin{itemize}

\item	Only the coordinate sequences for the river segments are to be
	given. No extra node specification is required.

\item	Should two segments be connected, it is necessary that the
	end point of the upstream and the starting point of the downstream segment
	are identical. This also means that for confluences or bifurcations
	the upper segment(s) always end at that point and one (or two) new
	segments start from the same point.

\item	All river segments must be directed downstream.

\item	All segments must have a unique ID.

\end{itemize}

Only confluences and bifurcations enforce the beginning of new stretches.
Sequences like segments 1 and 2 in figure 5.3 have no topological meaning
for the network. Splitting up segments into a sequence of two or more stretches
becomes important for the semantic purposes the digital river network has to
serve. Simulation models using digital river networks may need to simplify
river segments by assuming stretches to be homogeneous in a hydrological sense.
In this case flow data would be attributed to river stretches and hence long
segments need to be split up according to the required resolution or
available knowledge about
hydrological changes. Additionally, for visualization purposes the level of segmentation
also determines the capabilities to graphically display stretch-specific information.
Hence, when setting up the digital river network the later
purpose of these data should already be considered at this
point and segmentation should be performed. 
This task can also be undertaken automatically provided
that the single coordinates of the lines are more or less equidistant.

\subsection{River network attributes}

Depending on the needs of an applied exposure simulation model,
information about certain properties of the river network are required.
Usually these are floating point numbers (e.g. flow in $[m^3/s]$ or
flow velocity in $[m/s]$). Further attributes such as boolean (shipable [yes$|$no])
and others are conceivable.
Most of the traditionally measured data are expressed as statistics due
to natural fluctuations and seasonal variation.
For example, flow data are usually expressed as log-normal distributions
characterised by the mean and standard deviation (or mean and low, e.g. 5th,
percentile).

However, all discrete attributes (in space) have in common that they correspond
with a certain river segment. A set of discrete parameters for a river network
can hence be given in tabular style. Continuous parameters (in space)
are more difficult
to attach to digital river networks, and applications using such data
also reach a level of complexity which is seldom acceptable for
actual environmental simulations.
Usually distribution curves are derived from temporal data series
which are then applied for a specific task.

Depending on the purpose for which the river attributes are needed, in some
cases the variation of each river segment is of importance and for other
parameters variations are only of interest for entire rivers or river networks.
Furthermore, decisions regarding the stretch-wise resolution of
parameters are influenced by data availability.
Consequently, a task-specific separation of parameters is helpful and
can be solved by defining river classes and associating each river segment
to a river class.

Parameters that are semantically important and presumably different
for each segment are directly associated to river segments.
For many parameters the reasons for actual availability at high resolution
represents the opportunity not only to measure but
also to interpolate between measurements or to estimate them from
other attributes. Hence, again depending on the semantic context,
not all parameters are obligatory. A flexible (interpolating, estimating) system
of optional parameter specification is possible.

The methods of chemical exposure assessment are organized in a tiered approach
in terms of physico-chemical processes. Lumped transport and elimination
processes will be applied first and if it is not capable of explaining
monitored behavior, refined substance-specific processes will be
added (e.g. photodegradation). Any refinement increases data demands,
uncertainties and, on a very practical side, expense and time.

The counterpart of the lumped parameters for chemicals (transport and
elimination) are hydraulics and geometry of rivers. These basically
determine the dilution and dispersion of chemical loads.
Accompanied by length and name, these were chosen as direct attributes.
All other parameters are assembled in river classes.
This choice reflects the data requirement of the simulation software
incorporated in GREAT-ER (Boeije, 1999). The concept of the intermediate 
data stage still allows us to consider data requirements of any
comparable simulation model by providing corresponding
automatic data processing routines for the second step (figure 5.1).

\subsubsection{Flow, Velocity, Depth and Width}

\paragraph{Overview of interrelation}

For a sufficient period of time, the flow passing the full extent of a
river segment can be assumed as constant in space and time. In this case
the continuity equation \ref{continuity_equation} is true.

\begin{equation}
Q = v A	\label{continuity_equation}
\end{equation}

where $Q$ = flow, $v$ = velocity and $A$ = cross-sectional area.

Rivers have no rectangular cross-section and hence $A$ can not simply
be substituted by the product of depth and width.
Depth and width have a correlation
with the flow within certain ranges. A set of power equations
\ref{power_equations} can express
the interrelation of velocity ($v$), width ($w$), depth ($d$) and flow
(Leopold \& Maddock, 1953).

\begin{equation}
v = a Q^\alpha	\qquad ; \qquad 
d = b Q^\beta	\qquad ; \qquad 
w = c Q^\gamma \label{power_equations}
\end{equation}

where both $\alpha + \beta + \gamma$ and $a \cdot b \cdot c$ must be equal to 1.

Compared to velocity and depth, width is generally less variable (in space)
for most rivers (Chapra, 1997).
Especially when dealing with one-dimensional models, width is of
less semantic importance. In contrast, depth is important, i.e. for
chemical volatilization processes. In this approach width is hence dropped
from the list of required river properties. It should of course  be taken
note of when considering transversal dispersion (two-dimensional
river models).

Site-specific measurements and site-specific regression functions are
optimal to gain most adequate information on river properties.
In the absense of such data, and
this is true for most sites, literature or new field studies need to be
undertaken.

Numerous studies were conducted to obtain values for parameters $a, b, c$
and $\alpha, \beta, \gamma$ which vary with different climatic location and
degree of artificial influences on the river networks. 

It has also been shown that the hydraulic geometry can alternatively be
obtained from the corresponding watershed size:
Based on a strong correlation of bankfull
flow and watershed area, a geometric relationship was shown
and extrapolated for all river segments (Miller et al., 1997).
Another method is based on the hortonian stream order for river networks.
This ordering method attaches the order of 1 to all source stretches. The
order of any other stretch is the maximum order of the directly connected
upstream stretches plus 1. A good correlation of the hortonian order
and the river geometry has been proved for a watershed in Arizona, U.S.A.
(Miller et al., 1996).

Besides the statisticial approaches described above, mechanistic techniques
can be applied. The best known method is the Manning equation (\ref{manning_equation})
(Chapra, 1997).

\begin{equation}
v = \frac{R^{2/3}S^{1/2}}{n} \label{manning_equation}
\end{equation}

\begin{tabular}{ll}
where	& $n$ = Manning's roughness coefficient [-] \\
	& $R$ = hydraulic radius (equal to $A/P$) [m] \\
	& $P$ = wetted perimeter [m] \\
	& $S$ = energy slope (here equal to geological slope) [-] \\
\end{tabular}\\

Substitution into the continuity equation results in the approach of
Manning-Strickler (equation \ref{manning_strickler}) (Dyck \& Peschke, 1989).

\begin{equation}
Q = \frac{A R^{2/3} S^{1/2}}{n}	\label{manning_strickler}
\end{equation}

Besides flow and velocity, the depth can also be obtained based on
the Manning equation. The depth is the root of equation \ref{manning_depth}
(Chapra, 1997).

\begin{equation}
f(y) = \frac{1}{n} \frac{[(w_0 + s y) y]^{5/3}}{(w_0 + 2 y \sqrt{s^2 + 1})^{2/3}} S^{1/2} - Q  \label{manning_depth}
\end{equation}
 
where $s$ = side slope and $w_0$ = bottom width.

For the GREAT-ER project a new study has
been undertaken to estimate the velocity of any river reach in the UK
(Round et al., 1998). A statistical approach was chosen based on hydrological
databases for the UK. The quality of the velocity estimation model (equation
\ref{velocity_greater}) was not significantly reduced in terms of
performance by excluding the hydraulic radius. The error analysis showed
that with 68\% confidence the true velocity lies within a factor of
1.59. 

\begin{equation}
v = 10^{-0.583} Q_{mean}^{0.283} \left(\frac{Q}{Q_{mean}}\right)^{0.495} \label{velocity_greater}
\end{equation}

To obtain average depth values, variations in three dimensions have to be
made. First, depth can vary in transversal and longitudinal
extent of the river bed and next, it changes in time due to flow variation.
For unknown geometry and known flow an application of the Regime Theory
provides equations \ref{regime_width} and \ref{regime_depth} (Simons \&
Albertson, 1960).

\begin{equation}
w = 6.175 \sqrt{Q} + 0.305	\label{regime_width}
\end{equation}

\begin{equation}
d = \left\{
	\begin{array}{l@{\quad:\quad}l}
	0.610 + 0.93 r	& r > 2.13 \\
	1.21 r		& r <= 2.13 \\
	\end{array}
	\right.
	\label{regime_depth}
\end{equation}

where $r = 0.57 Q^{0.36}$.

\paragraph{Selection of values and methods}

Literature review has shown examples of the ability to estimate the
interrelated river properties flow, velocity, depth and width
from each other in several ways. Also area and mathematical topology
can be included for estimations. The latter can even be applied when 
no stretch specific properties are known.

In general, higher efforts are required to measure the geometry
of river segments while flows are easier to obtain:
In most European countries a system of
gauging stations is in operation, delivering flow statistics for some sites.
Usually interpolation equations can easily be set up or are even readily
available for ungauged sites between two gauging stations. Ungauged headwaters
are more complicated to establish flow estimation. The flows can be computed
from geo-morphology and other information, i.e. MicroLowFlows
additionally applies soil type and precipitation information for this task
(Gustard et al., 1992).

For the methodology introduced in this chapter, flow data are
chosen as obligatory, and velocity and depth are optional parameters.
Optional parameters will be estimated during the data processing.
Consistency checks based on the literature can be integrated
in the final processing.

Most estimation equations derived from statistical studies are
associated with several limitations concerning the
environmental framework of the investigated river network
(natural/regulated, arid/humid, lowland/highland, etc.).
If estimation equations are available which were set up
for the same or a similar type of river these should be
preferred and the resulting values should be used. Automatic selection
of estimation routines can not decide adequately compared to
human judgement. This is due to the fact that much general information
on a river network is required and this information can not easily
be formalized.

One important property of the discussed parameters is variation
in time, which means that these parameters usually consist
of three subparameters describing their statistical distribution
(i.e. distribution type, mean, standard deviation). This must be
kept in mind when dealing with corresponding data collection and
estimation.
 
\subsubsection{Length}

In conjunction with velocity, the length basically determines
the transport of a (chemical) load.
The process of transport in terms of chemical exposure modeling
is a relatively fast process and hence the travel distance
should be given as accurately (feasibly) as possible.

Obtaining the actual length of rivers should in general not pose a serious
problem because it is relatively easy to measure. For some
countries, canoeing guides offer this information in great detail,
i.e. for Germany (Deutscher Kanu-Verband, 1985).

In theory, there are of course no problems in calculating
the length of each segment based on coordinates.
These values are naturally (sometimes much) lower than the
real length of the river segments.
These differences become obvious when taking into account that
even detailed digitizing (e.g. one point every 300m) can not
reflect the actual shape of a very curvy river. Additionally,
the digitizing itself reveals uncertainties.

\subsubsection{Name}

Names for each river segment are not meant to identify stretches
in terms of database operations. Obviously one river can be split up
in very high numbers of stretches and therefore unique name specification
does not make sense.
The names are required rather for identification and orientation
purposes in the final simulation tool. Names of rivers can hence
be associated with other site names or any information
that is felt to be helpful to users.

\subsubsection{River Classes}

The composition of the river class parameters is to
be selected dependent on the semantic purpose of the
simulation models. This way, the number of properties
can be reduced to a minimum to retain low complexity
and an easy overview.

The methodology presented in this chapter takes the
river class composition as used for the simulation software
in GREAT-ER (Boeije, 1999).
Included are typical water quality parameters, i.e. suspended solids,
dissolved oxygen, biological oxygen demand and pH. All these parameters
are only used in higher model complexity modes, i.e. not applying
lumped degradation rates.
Among the biochemical properties, the type of segment
is also described (river or lake).

\subsection{Catchment boundary}

The catchment of a river network is defined by the most
downstream point, i.e. the most downstream segment.
The catchment boundary represents the natural drainage area. 
Any point in that area will drain to that most downstream segment.
The catchment boundary must always be a closed polygon.

The introduced data composition additionally enables several
separated river networks to be joined.
This helps users to deal with e.g. flat coastal regions, entire
islands or even whole countries.
In such cases the catchment boundary should be the union of all
single subcatchments.

Information about the actual border of a catchment serves two purposes.
First, it supports geographic methods, e.g. spatial selection routines.
Second, within the final tool the user is given further support
for orientation and interpretation.

In regions where water management has a long tradition, watershed boundaries
should be readily available. Any new or re-calculation applying
a GIS will require digital elevation data. This computation is often
performed and several technologies are available, partly already built into
a GIS.

\subsection{Discharge data}

The discharge data, in this context, are a set of points through which
substances are released into the river networks, e.g. sewage treatment plants.
The discharge information will not be able to express diffuse input
(unless it is made discrete as a series of point inputs).
Diffuse input is not focused in the presented methods because a different
class of simulation models would be required.

Discharge points with geographical reference define an area where
any release will pass this point travelling "down-the-drain".
Such areas are scale-independent geographical units interconnected
by a network topology (figure 5.3). This scale independency allows
us to flexibly {\em refine} or {\em aggregate} (parts of) the
geographic data where required.
Refinement on the one hand offers dynamic, problem-oriented
geographical resolution to
avoid unnecessary data collection and helps to deal adequately
with data gaps and
to zoom into areas of special interest.
Aggregation  on the other hand enables us to reduce complexity, to compare
regions by abstracting data and to calibrate actually needed scales of data.

\begin{center}
	\Fig{\includegraphics[width=140mm]{./images/scales.pdf}}{
		Geographic units (GU): scale independent}
\end{center}

\paragraph{Coordinates}

The coordinates of a discharge site are obligatory. In the case
of treatment works they describe the position of the actual plant
and not the point at which the emission enters the river. The connection
to the river will be established by explicit specification of the corresponding
river segment. This enables provision of location information to the user who
is supported in interpretation by offering the distance of the plant
to the river and
whether the emission takes place on the left or right bank.
 
\paragraph{Population}

For simulations of chemical ingredients in consumer products, the per capita
consumption is usually available in mass per time. To properly estimate
the release, the actual real population connected to a discharge is required.
These numbers should be readily available from responsible environmental
protection authorities, water authorities or water supply companies.
For purely industrial discharges the population is zero.

It is not feasible to use population equivalents, because these numbers
include industrial emissions based on a special selection and weighting
of water quality parameters. For example, in Germany a per capita emission
of 200 liters of waste water with an equivalent load of 60 $g$
of biological oxygen demand
(BOD) is assumed. Based on this, several population equivalent types are
defined (DIN 4045 (12/1985); R�mpp, 1997):
Equivalents based on BOD (EGWB60), based on waster water emission (EGWW200),
resulting sludge volume (EGWS2,0) and resulting dry sludge (EGWTS80).
Across Europe several national standards are to be expected; for some countries
variations between water authorities are even possible (e.g. in the U.K.).
For most consumer substances no correlation with the standard water quality
properties exists and hence population equivalents may reveal helpful
background information, but are not required for chemical exposure assessment.

\paragraph{Flows}

Flows within the waste water and associated paths are measured and defined
in several ways across Europe.
For the data requirement definition a most common and practical selection
is presented here. Above all the most important value is the flow
leaving the plant and entering the river network. The actual daily flow (ADF)
will mean the average (measured) emission of a plant in $m^3/d$.

In many cases consented maximum emissions are available. Such values
are only variable with the consenting interval and do not necessarily
represent the real emissions. For those sites for which no ADF is available
the consented daily flow (CDF) is the best approximation, since the operators
of waste water treatment facilities usually try to fill the consented flow.
 
The inflow of a treatment plant consists of three components:
Runoff, non-domestic and domestic flow.

\paragraph{Discharge Type}

The type of a discharge is considered in the simulation
with specific models to estimate removal efficiency.
Usually the technological type of the corresponding waste
water treatment plant is required (i.e. trickling filter plant,
activated sludge plant), but the discharge type can be "direct discharge"
(no chemical removal) as well.

Some sewage treatment plants are actually a combination
of different technologies. A specific modeling of such situations
is not feasible. A conservative simplification choosing the
less efficient type is recommended, but for transparency the
situation should be noted (see the {\em Name} below).


\paragraph{River connection}

Discharges are connected to the first point of the given segment which
is to be specified with its ID. For an optimal representation
of the actual river entry point, the river network digitizing or
segementation should already consider these points.

In principle, GIS routines (distance measures) can be applied to find out
the most reasonable stretches. These methods are automatic but always
require human plausibility control.

\paragraph{Name}

The only purpose of the names is to support user orientation and interpretation
as well as reporting. The names need not to be unique and should
contain further information or hints when necessary.

\subsection{Background maps}

Though not used in the simulation models, background information
is important for the final end user system.
They allow visual orientation and plausibility control as well as
aiding interpretation of simulation results, which is carried out
by the user.

Different types of background information can be provided: 
raster landcover maps, vectorized information (geographic objects) on
e.g. soil types and land use. Additionally, semantically related (temporal)
information can be prepared such as monitoring sites with attached measured
time series.

\section{Final automatic data processing}

The final data processing requires syntactically and semantically
correct source files of the files ".drn", ".rna", ".dsd", ".cbp" and ".bgd"
as described above. The whole processing with its subsequent steps is managed
by the main program {\em greater-pre-processing.py}.


This part of the data processing is, in principle, not dependent on the
required simulation system, though it does simplify the simulation
system and user
environment developed within the framework of GREAT-ER.
The simple tabular structures allow the easy extension of the
data files for adaptation to any other comparable simulation system.

The data to be generated can be divided into two groups: geographic data
and the corresponding attribute data.
The geographic data form the basis for visualization and geo-referenced
data processing.
These elements (cf. points, lines and polygons) are associated with
attribute data via unique ID's. Their interconnection is based on the
same ID reference.

Processing of the background data (".bgd" file) is not related
with the main processing due to its special task to support the user.
It is not discussed further in this document.

\subsection{Geographic data}

The establishment of geographic data consists of two
processing steps (figure 5.5).
First, the geographic elements (their ID's and coordinates) are
extracted from the files in the pre-defined formats and converted
into another format that can be handled directly by a GIS that
handles the Shapefile format.

\begin{center}
	\Fig{\includegraphics[width=120mm]{./flow_final_geo.pdf}}{
		Command sequence for automatic processing of geographic data}
\end{center}

A straightforward conversion is executed for the two digital maps:
discharge sites ("discharges.shp") and  river network ("rivernet.shp").
For the boundary of the catchment ("catchbound.shp") a further attribute is
appended from the datafile "catchments.txt": the full name of the catchment.

The connection lines between discharges and corresponding river segments
are computed using information from the ".dsd" and ".drn" files.
A special shapefile is created ("disch\_river.shp"), which
provides visual information on where the discharge is connected to
the river network.

\subsection{Attribute data}

While the attribute data are being processed (figure 5.6), the three main
data files (".dsd", ".rna" and ".drn") are first joined and 
then create the primary attribute files for river segments
("river.att") and discharges ("disch.att") as well as the secondary
attribute files for treatment plant types ("wwtp.att") and river
classes ("rivclass.att"). The two secondary files are (currently) static and
basically describe generic treatment plants and rivers.

\begin{center}
	\Fig{\includegraphics[width=120mm]{./flow_final_att.pdf}}{
		Command sequence for automatic preparation of attribute data}
\end{center}

In the final processing step, text based tables are converted into
DBase format. The purpose of this is to optimize data management within
the visualization tool.

\subsection{Core algorithms}

Several data processing steps consist of simple routines (e.g. format
conversions). They are not explicitly explained here.
In this section the core problems of data processing are discussed.

\subsubsection{Creation of River Stretch Sequence}

The information about the logic sequence of single river segments
plays an essential role for simulation models.
A simulation model must be able to {\em walk} down from any source
to the final mouth. Hence, each stretch must offer information (the ID)
about its successor. This purpose is served by a simple two-column table
{\tt fromStretchID,toStretchID}.

The ".drn" files, if semantically and syntactically created as described above,
contain the topology information, but coded in the given coordinates.
The arc-node model is used to convert the topology to the coding based on
StretchIDs. Building the arc-node model from a ".drn" file associates all
segments with IDs of the corresponding "from-node" and "to-node" to describe
their direction. Consequently, the two-column tables
{\tt fnode,StretchID} and
{\tt tnode,}  \linebreak {\tt StretchID} can 
be extracted directly from the arc-node model \linebreak ("topology.awk").
Joining these two tables by the node ID
and then omitting the node ID results in the two-column table
{\tt fromStretchID,toStretchID}. In both columns IDs may now occur multiple
times due to confluences and bifurcations. To create a table where
the {\tt fromStretchID} is a unique identifier, the {\tt toStretchID}s are
joined up into a (space character divided) list. The resulting
two-column table {\tt fromStretchID,toStretchIDlist} is a straightforward
representation of the downstream topology. Analogously, an upstream
topology table {\tt toStretchID,} {\tt fromStretchIDlist}
can be created starting
with the {\tt fromStretchID,toStretchID} table and exchanging
the position of the two columns. 

\subsubsection{Creation of Binary River Network Topology}

Fast and straightforward access for any algorithm walking up
or down the river network is realized using simple data tables.
The minimum number of columns to describe any river network topology is four.
Demanding one column being a unique segment ID and also allowing only exactly
one ID in any field, virtual segments need to be added to grant topologic
consistency (e.g. confluence of three streams at one point).
An additional column finally describes the segment type (listed in table 3.1).
This is needed to correctly interpret the content of the other columns. 
Extending the network by virtual segments will not increase but decrease
the total table size because a further column would rarely be filled for
natural networks where a confluence of two streams is usually the maximum.
For artificial networks (e.g. sewer systems) the situation can be different.

\begin{center}
Table 3.1 {\em Segment Types} \newline
\begin{tabular}{c|c|c|c|l}
\hline
		& Type (if segment is a	& \multicolumn{2}{|c|}{Number of segments}	& \\
Type	& lake; only GREAT-ER 2.0) & Upstream & Downstream & Description \\
\hline
0	& 8		& 0 & 0,1,2	& Source \\
1	& 9		& 1 & 1,2	& Normal segment \\
2	& 10	& 2 & 0,1	& Confluence \\
-2	& 6		& 1$^*$ & 0,1,2	& Bifurcation \\
\hline
\multicolumn{4}{l}{\ $^*$ which must be of type 1 with 2 downstream segments} \\
\hline
\end{tabular}\\[1ex]
\end{center}

Algorithms for walking on binary trees from the leaves to the root
(cf. downstream) are typically implemented recursively starting with
the root (cf. \linebreak mouth).
For exposure assessment in river networks the binary recursion has to
be "post-order" which means performing an action for a segment after all
upstream segments have already undertaken this action.
Hence, the upstream direction is required for the two-column topology table. 

The above-described tables for the river stretch sequence are the basis for
building the binary topology.
During the data loading routine, the highest segment ID is looked up.
This is important to invent further ID's for virtual segments.

When all data are loaded, first bifurcation situations are searched for.
The corresponding segments (type "-2") are associated with their neighboring
segments.
This is important in order to reduce the final topology processing to
a one-pass computation (a single linear processing step).
Next, the segment table is processed straightforwardly. Depending on
the number of upstream and downstream segments, the type is then decided.
Several special cases need further treatment i.e. insertion of virtual
segments:

\paragraph{(I) Confluence of three segments at one point}

When exactly three segments are directed to the same point,
a new (virtual) segment is inserted as shown in figure 5.7.

\begin{center}
	\FigExplanation{\includegraphics[width=100mm]{./triple_confl.pdf}}{
		Resolving confluence of three segments}{
	\begin{minipage}[t]{10cm}
		Resolving confluence of three segments (a)
		into binary representation (b).
		Numbers are stretch IDs. The table of binary network topology
		of (b) is (c).
	\end{minipage}
	}
\end{center}

The virtual segment obtains a length of 0. The ID
does not appear in the geographic data of the river network
and is neutral for the database operation "join".
The simulation software must of course know how to treat segments
with a length of 0: The results (e.g. chemical loads) must be transferred
one-to-one to the next downstream segment.
 
\begin{center}
	\FigExplanation{\includegraphics[width=120mm]{./ocean_segments.pdf}}{
		Resolving multiple end segments (multiple catchments)}{
	\begin{minipage}[t]{10cm}
		Resolving multiple end segments (multiple catchments) (a)
		into one catchment (b). Numbers are stretch IDs.
		Table of binary network topology
		of (b) is (c).
	\end{minipage}
	}
\end{center}

\paragraph{(II) Several end segments}

If several catchments are joined (e.g. to cover an entire island)
the problem of multiple start segments (mouths) occurs and needs to be
incorporated into the topology.
 
The solution is a linear interconnection with virtual segments.
By definition all "ocean segments" are attached with negative ID's.

In the example shown in figure 5.8, the lowest ID is -3 and it is the
new starting segment for the resulting catchment.
The ocean segments influence neither visualization nor simulation.

\paragraph{(III) Bifurcation}

For bifurcation situations both neighbours are of the type "Bifurcation"
(figure 5.9).
Field {\tt up1} will contain the ID of the upstream segment and field
{\tt up2} will contain the ID of the neighboring segment.

\begin{center}
	\Fig{\includegraphics[width=80mm]{./bifurcation.pdf}}{
		Handling of bifurcations (a) in the binary presentation (b).}
\end{center}

\paragraph{(IV) Unconsidered situations}

Some situations may occur in the digitial river network, but are
unlikely to happen in reality for natural catchments. In practice
it is more likely that wrong digitizing or another error may cause
the situation.
Such situations are not considered in the presented
methodology though it is possible to resolve them with further
virtual segments or additional definitions.

\subparagraph{(IV.a) Multiple confluence}

More than three segments are directed to the same point (figure 5.10a).
This situation could be managed similar to the confluence
of three segments using virtual segments, but from a hydrologic
point of view this situation is most unlikely and the chance
of using wrong base data is much more probable.

\begin{center}
	\FigExplanation{\includegraphics[width=80mm]{./unconsidered.pdf}}{
		Unconsidered situations}{
	\begin{minipage}[t]{10cm}
		Unconsidered situations: Multiple confluence (a),
		circle (b), crossing (c) and multiple bifurcation (d).
	\end{minipage}
	}
\end{center}

\subparagraph{(IV.b) Crossing streams}

Both, a confluence and a bifurcation take place at one point (figure 5.10c).
Such a crossing is unlikely. Furthermore, a chemical
fate model would need additional information to adequately deal with
such a situation (what fraction of which upstream segment
goes into which downstream segment?).

\subparagraph{(IV.c) Circular flow}

The segment sequence forms a ring (figure 5.10b). Building a ring, a river
network is in topological terms not a directed graph.
The applied simulation system is not capable of dealing with this situation.
Such circles may occur, for example, in estuaries where the flow direction
changes over time following a certain probability distribution.


\subparagraph{(IV.d) Multiple bifurcation}

Flow separation of one segment into more than two segments (figure 5.10d).
This situation is again unlikely from a hydrological point of view.
An error in the base data is to be supposed.
