\documentstyle[12pt,a4]{article}
\begin{document}

\begin{titlepage}
{

\hoffset=1truein
\hsize=5.25truein
\vsize=10.25truein
\font\small=cmssbx10 at 14.4truept
\font\medium=cmssbx10 at 17.28truept
\font\large=cmssbx10 at 20.74truept
\hrule height 0pt
\parindent=0pt
%\parskip=0pt
\hskip 3.9truein
\large
EDOC085\par
\vskip .5truein
\large
EUROGAM PROJECT\par
\vskip 1.5truein
\hrule height 2pt
\vskip 20pt
\large
NSF DATA ACQUISITION SYSTEM\par
\vskip .5truein
VXI/VME integrated data acquisition and control system for
the EUROGAM array
\vskip 20pt
\hrule height 2pt
\vskip 1truein
\medium
Real Time '91 - Julich
\vskip 5pt
June 25 1991\par
\vfill
\medium
NSF Software Systems Group\par
\vskip 5pt
UK Science and Engineering Research Council\par
\vskip 5pt
Daresbury Laboratory\par
\vskip .5truein

}
\end{titlepage}
 
\title{VXI/VME integrated data acquisition and control system
for the EUROGAM array}
\author{M.M.Aleonard (CENBG, IN2P3, 33175 Gradignan, France) \\
V.F.E.Pucknell (NSF, Daresbury Laboratory, Warrington, WA44AD, UK)}
\date{}
\maketitle

\begin{abstract}
The EUROGAM array data acquisition and control system is
distributed over several VXI and VME crates connected 
by an ethernet local area network. The real time software 
kernel used is VxWorks and graphical user interfaces are
implemented in UNIX based workstations.
Remote Procedure Call techniques are used to implement 
communications between client workstations and the server
data acquisition resources.
\end{abstract}
 
\section{Introduction}
Eurogam is an array of suppressed Germanium detectors used to
detect gamma-rays with high resolving power and total
efficiency. It is being built by a French-UK collaboration
involving IN2P3 in France and SERC in UK. The principal 
laboratories involved are
\begin{itemize}
\item CENBG (Bordeaux)
\item CRN (Strasbourg)
\item IPN and CSNSM   (Orsay)
\item NSF Daresbury Laboratory 
\item Liverpool University 
\end{itemize}
 
Eurogam is a successor to existing devices in Europe such as
Tessa3 and Polytessa (UK), Chateau de Cristal (France) and
Nordball (Denmark).
 
For the first phase which will be operational at the 
Daresbury Laboratory Nuclear Structure Facility for 
12 months starting in January 1992 the array will consist
of 45  Germanium detectors each of which is surrounded 
by 10 BGO crystals to form the suppression shield. For the
second phase starting in Spring 1993 it is hoped to be 
able to use the new Vivitron beam at CRN Strasbourg 
by which time the array should be completed to 70
detectors.
 
The data acquisition and control system for Eurogam is 
implemented within a distributed computing environment. Such 
an environment consists of a number of workstations 
plus file servers, print servers, tape servers ,etc., all 
connected via a local area network (typically ethernet).
This concept is extended by connecting the data acquisition
resources as additional servers onto the local area network.
Integrated electronics using the VXI standard 
and parallel processing of data within VME crates
are used for data acquisition. Each VME and VXI crate has a 
crate controller which is a cpu board having an ethernet 
interface. This cpu runs the server software and provides 
access to the crates from any source on the local area
network (normally a UNIX workstation) which provides
for overall control and monitoring of experiments.
 
\section{Hardware Components}
\subsection{Front-end Electronics}
Since the full Eurogam array consists of 70  GE detectors
and 700 BGO suppression shields a departure from 
conventional electronics using equipment housed in 
NIM modules is required.
 
The data acquisition system needs to provide:
\begin{itemize}
\item a high degree of integration for the electronics components
in order to minimise interconnections   (and cables)
and thus ensure higher reliability for the array;
\item software control of all adjustable parameters;
\item several data buses to allow fast response to control
requests plus high data transfer rates;
\item modularity of hardware to allow a continuous upgrading 
of the equipment as required.
\end{itemize}
 
Using the VXI standard (VME eXtensions for Instrumentation)
electronics modules have been designed which will accept
the analog signals output from the detectors, digitize the
data and output to the data readout bus (DT32 bus).
 
\begin{itemize}
\item GE card---will handle readout from 6 GE detectors
\item BGO card---will handle readout from the BGO suppression
shields associated with 6 GE detectors (i.e. 60 BGO crystals).
\item Master trigger module---handles selection of raw data and
initiation of readout.
\item Readout Controller---one per VXI crate. Initiates readout 
via the VME bus from each instrumentation card and outputs
to the DT32 bus.
\item Resource Manager---provides VXI slot 0 functions
\end{itemize}
 
Software can adjust all variable parameters of the hardware 
(trigger conditions, threshold controls ,etc.,) via the VME 
backplane.
 
Each VXI crate can hold all the electronics for 30 channels
and hence for the first phase  only 2  VXI crates are 
required for data capture and readout. For the
second phase a third VXI crate will be added.
 
Each VXI crate requires a resource manager (RM) in slot 0.
A VXI RM card has been produced which incorporates a 
standard VME processor card (MVME147 or similar) having
an ethernet interface. This card then acts as both  
the VXI crate master and as a crate controller accessible 
over the ethernet LAN which can be programmed as desired.
 
Data from each VXI crate is passed by the Readout Controllers
at rates of up to 4 Mbytes/sec onto the DT32 bus (a 32 bit
bus using ECL logic).
 
\subsection{Histogrammer}
 After all VXI crates the DT32 bus passes through a 
histogrammer unit which is a VME module acting as
a spy onto the data. The unit is software programmable
to select any specific parameters from any choosen 
detectors and build histograms via direct memory
increment into global dual port memory. 
The histograms can be examined using any of the workstations
to check the quality of the data and monitor detector 
performance.
Initially
there will be 32 Mbytes of histogramming memory but this can
be increased by adding more memory cards into the VME crate.
\subsection{Event Builder}
 The DT32 bus terminates in a VME crate containing the Event 
Builder. Input data are received from the DT32 bus via an 
interface module into one or more CES HSM8170 VME modules
controlled by a CES FIC 8232 cpu. A number of processor 
modules (MVME 165) operating in parallel will process
the incoming raw data to reduce the data volume and 
generate blocks of formatted events. The output data will
be passed on via a dedicated fibre optic link using a 
CES FIC 8232 and CES ODL 8142  capable of data rates up to 
6 Mbytes/sec.
 
 For Phase 1 it will be required to process about 1 Mbyte/sec
of raw data. However this construction is very flexible 
and by addition of further hardware modules operating in 
parallel the data rate through the Event Builder can be 
increased to handle the full data rate from the DT32 bus.
 
\subsection{Data Router}
 Data from the optical data link is received by another 
CES FIC 8232/CES ODL 8142 pair. It may then be routed 
to either the data sorter or the tape server or both.
 
 The equipment before the data router is located close to
the detectors while the Data Router and latter components
are located in the control room. At Daresbury these
are some distance apart (100 m) and electrically
isolated from each other. The optical link
provides an excellent means of passing data from one
area to the other.
 
\subsection{Data Sorter}
 This occupies the same VME crate as the Data Router. It uses 
software running in the crate controller (MVME 147) as a
supervisor and a number of processors running in 
parallel to provide online analysis of the data. 
Multi-dimensional histograms are built in global memory. Initially
64 Mbytes of memory will be available but this can simply
be increased by adding further memory modules. By the use
of a suitable number of cpus all data generated by the
Event Builder can be processed.
 
\subsection{Tape Server}
The Tape Server supports a  minimum of four tape streams 
using Exabyte 8500 drives.  Event by event data received 
from the Data Router is written to one or more
of these tapes.
For Phase 1 a total of between 0.5 and 1 Mbyte/sec of data
will be required to be written to tape. Duplicates of all tapes
will be required since post experiment analysis will be 
divided between the collaborators in the experiment.
 
\subsection{Other Components}
 In addition to the above modules used to collect, process,
sort and store
the data other systems provide and control the
environment in which the above operate.
 
\subsubsection{Autofill}
 Another VME crate which provides for the filling and 
monitoring of the liquid nitrogen in the detectors.
\subsubsection{High Voltage control}
 Uses LeCroy high voltage units interfaced via a 
VME  module to provide the necessary bias to the 
GE detectors and BGO crystals.
\subsubsection{Environment Monitoring}
Monitors temperature, electrical supply, fans ,etc., in the
electronics racks.
\subsubsection{UNIX workstations}
Provide  the user  interface to the system.
 
\section{Software Components}
The user interface to the data acquisition system is via
UNIX workstations while all VME cpus run the VxWorks 
real-time kernel.
\subsection{VxWorks}
The VxWorks kernel was choosen for the real time
components because
\begin{itemize}
\item it is designed specifically for real-time 
control  and data acquisition functions within a 
multi-tasking environment;
\item it gives very fast handling of interrupts, task 
scheduling ,etc.;
\item it is UNIX source compatible. All software generation
and compilation is performed using UNIX workstations;
\item it has full support for BSD networking allowing 
complete integration into a UNIX workstation 
environment;
\item source level debugging is available from UNIX
workstations using DBX.
\end{itemize}
 
\subsection{Communications}
Each VXI and VME crate contains a cpu (MVME 147) having an 
ethernet interface which is a node on the local area network
and which also acts as a gateway for other CPU cards 
in the same crate. The use of UNIX and VxWorks allows
for common network software in all components of the system
\begin{itemize}
\item Internet networking  (IP)
\item UDP/IP (User Datagram Protocol/Internet Protocol)
\item RPC (Remote Procedure Calls)
\item XDR (eXternal Data Representation)
\item NFS (Network File System).
\end{itemize}
 
All communications between applications specific to the
Data Acquisition System are based on the Client/Server 
model using Remote Procedure Calls. Using the network
software RPC requests can be made between clients and servers 
across the ethernet, between VME cpus within a crate 
across the VME backplane, and within a single CPU or workstation.
This allows the software components to communication in a
way which is independant of their location.
 
The standard RPC library supplied by Sun Microsystems 
has been extended to be more flexible and suitable
for use in the data acquisition environment. An
application
\begin{itemize}
\item may issue a number of RPC requests in parallel 
to different servers in order to improve 
performance during setup;
\item may adjust the RPC timeouts to be suitable for 
the specific application;
\item must at all times be responsive to the user 
interface.
\end{itemize}
 
Fixed UDP/IP ports are used by the servers in order to 
avoid the delays which the use of dynamic ports and the
Portmapper would involve.
 
Currently ethernet is used as the network medium. This 
would be inadequate for the main event-by-event data path 
and hence non standard solutions are used here. In the future 
FDDI may provide a suitable standard solution.
 
\subsection{The Servers}
The Eurogam data acquisition system has a small number of
server programs which provide access to all the data 
acquisition  resources.
\begin{itemize}
\item register server
\item spectrum server
\item tape server
\item message/error logger
\end{itemize}
 
All access is via the RPC service from client workstations. For
each server an RPC program protocol is defined which is 
designed to provide access to the server facilities and to
conform with the RPC methodology.
 
\subsubsection{Register Server}
The register server is a general purpose communications
core which can be customized for specific data acquisition
applications. It is used for communication with
\begin{itemize}
\item VXI crate setup and control application
\item Histogrammer setup and control application
\item High Voltage control system
\item Nitrogen Autofill control system
\item Environment monitoring
\item Event Builder
\item Data Sorter
\end{itemize}
 
Initially each server has a number of application 
specific procedures which are loaded when the system 
is booted. Registers are then defined dynamically 
by the clients using the server protocol and
specifying the local procedures in the server to be 
invoked when the register is subsequently accessed.
 
The register server allows named items of arbitrary data to be
passed between the user interface programs and the 
server applications. The  register name and the server 
command primitive  select the local procedure to be called.
 
The data can be a single 32 bit word to be written to a 
VXI register or may be up to 8 Kbytes when defining
conditions for the data sorter.
 
Register Server primitives are
\begin{itemize}
\item read
\item write
\item initialise
\item read attribute
\item write attribute
\end{itemize}
 
The register attributes provide additional information as needed
by the application procedures to execute the read, write and
initialise primitives. For example, a simple case would be 
a VXI register and the register attributes would include
the VME address offset used to access the register on the VXI card.
 
The server allows many registers to be accessed by a single
RPC request by the use of wild-card techniques applied to
the names. The available register names can be obtained 
by clients from the server thus  allowing a number
of user interfaces in workstations to access the server 
with all coordination provided by the server.

\subsubsection{Spectrum Server}
Provides a uniform program interface to both on-line 
spectra in the histogrammer and data sorter and to 
off-line spectra held on disc.  Supports various 
high level primitives to allow data to be accessed while
minimizing network traffic.
 
\subsubsection{Tape Server}
Provides access to the tape storage devices.
Supports data striping  
allowing a high rate data stream to be written to a 
number of tape drives accessed in parallel and supports 
data duplication allowing a data stream to be 
duplicated onto a number of tapes.
 
\subsubsection{Message/error logger}
Accepts messages from any component of the system for logging
and will pass on to any client which has expressed an 
interest in the particular message class. 
 
\subsection{Event Builder \& Data Sorter}
In addition to the servers these both run an experiment 
specific application code which performs the 
required transformation and analysis of the data. Computer
languages have been defined to allow the requirements 
for these programs to be expressed in a physics-orientated
language. Using LEX and YACC compilers have been
produced for both languages which converts the user input
into C which is then compiled on the UNIX workstation
and loaded into the VxWorks processors using NFS.
 
\subsection{Visualisation}
Unix workstations are used to setup, control and monitor 
data acquisition. They are also used for off-line 
analysis of the data. A window based system is used 
based on X11 and Open Windows and using the interactive tool
Dev Guide to design the screen layouts.
 
\section{Acknowledgements}
This work is supported by IN2P3 and SERC funding. The 
authors       wish to acknowledge the contributions  
from the many hardware and software engineers in France and
the UK who are
collaborating on the project  and
discussions with the physicists who will eventually
use the array.
 
\end{document}