\documentclass[12pt]{article}

\usepackage{graphicx}

% \oddsidemargin -10mm
% \evensidemargin -10mm
% \textwidth 180mm

\oddsidemargin 0mm
\evensidemargin 0mm
\topmargin -10mm
\textwidth 160mm
\textheight 230mm

\newcommand{\Vec}[1]{\mbox{$\vec{#1}$}}

\newcommand{\Section}[1]{\section{\textmd{\textsf{#1}}}}
\newcommand{\Subsection}[1]{\subsection{\textmd{\textsf{#1}}}}
\newcommand{\Subsubsection}[1]{\subsubsection{\textmd{\textsf{#1}}}}

% #1:width(mm), #2:.eps, #3:caption

\newcommand{\FigureL}[3]{
\vspace*{7mm}
\begin{minipage}{140mm}
\includegraphics[width= #1 mm]{#2}\\
{#3}\\[5mm]
\end{minipage}
}
\newcommand{\Figure}[3]{
\begin{center}
\FigureL{#1}{#2}{#3}
\end{center}
}

\newcounter{bibno}
\newcommand{\mycite}[1]{[\ref{bib:#1}]}
\newcommand{\mybibitem}[2]{
  \refstepcounter{bibno}
  \label{bib:#1}
  \noindent
  [\thebibno]
  \begin{minipage}[t]{150mm}
  #2
  \end{minipage}
  \vskip 5mm
}

\begin{document}

% \gtfamily
\sffamily

\thispagestyle{empty}

% \mcfamily
% \rmfamily

\vspace*{40mm}
\begin{center}
{\Huge Goose Software Package
User's Guide}\\[10mm]

\vspace*{5mm}
{\LARGE for}
\begin{minipage}[t]{50mm}
\vspace*{-10mm}
\includegraphics[width=50mm]{gooselogo.eps}
\end{minipage}
{\LARGE version 1.0.1$\alpha$}\\

\vspace*{10mm}
{\large Last modified at : Sep. 28, 2009
}
\end{center}

\bigskip

\vfill

%
\hfill
\begin{minipage}{80mm}
    {\large
      \includegraphics[width=70mm]{kfcrlogo.eps}\\[3mm]
      ~~K \& F Computing Research Co.\\
      ~~E-mail: support@kfcr.jp
    }
\end{minipage}

\clearpage

\vspace*{-3mm}
\tableofcontents

\clearpage

\Section{Abstract}

This document describes usage of the Goose Software Package.

Goose is an environment for software development that integrates
compilers and other utilities. It helps porting a program described
with a programing language such as C, from PC to a hardware
accelerator that works in SIMD fashion. It is designed to minimize the
amount of modification to the original source code.  The current
version (version 1.0.1$\alpha$) support C language and GRAPE-DR.  In
the near future, it is planned to support Fortran, GPU (both AMD and
NVIDIA), Intel SSE technology, and GRAPE-7.

Goose is a kind of domain-specific developing environment, that is, it
is not designed to support the whole grammar and specification of the
programing language. It handles only descriptions which are suitable
for the hardware accelerator. Other descriptions are passed on to a
conventional compiler, such as {\tt gcc}, to generate an executable
for PC, which serves as the host computer of the accelerator. This
approach would minimize the necessary amount of modification to the
source code.

%
In section \ref{sec:install}, installation procedure of the Goose
Software Package (hereafter "this package") is described.
%
In section \ref{sec:basics}, basic usage of Goose is described via tutorials.
%
In section \ref{sec:pragma}, all Goose directives are listed.
%
In section \ref{sec:cgrammar}, C-language grammar which can be handled
by Goose are shown.
%
Section \ref{sec:advanced} is devoted to detailed description of Goose
compiler inside and architectures of the hardware accelerators.

In the rest of this document, the topmost directory of this package, {\it i.e.},\\
{\tt /{\it path\_to\_the\_directory\_at\_which\_you\_unpacked\_this\_package}/goosepkg{\it the\_version\_number}/},
is denoted as {\tt \$goosepkg}.

\clearpage
\Section{Installation}\label{sec:install}

\Subsection{Required Environment}

Goose runs on 64-bit Linux (x86\_64). Goose internally uses the
following softwares. They need to be installed beforehand.

\begin{itemize}

\item C compiler (gcc version 4.1.0 or higher recommended.)\\
{\tt http://gcc.gnu.org/}

\item Ruby (version 1.8.5 or higher recommended.)\\
{\tt http://www.ruby-lang.org/}

\item Perl (version 5.8.8 or higher recommended.)\\
{\tt http://www.perl.com/}

\item LSUMP (A compiler for the Goose intermediate-representation.
Developed by Naohito Nakasato, University of Aizu.)\\
{\tt http://www.kfcr.jp/goose.html}

\item VSM (An assembler for GRAPE-DR. Developed by Junichiro Makino,
National Astronomical Observatory of Japan.)\\
{\tt http://www.kfcr.jp/goose.html}

\item GRAPE Software Package (version 1.1.0 or higher recommended.)\\
{\tt http://www.kfcr.jp/goose.html}

\end{itemize}
%

\Subsection{Unpacking and Setting up Environment Variables}

Unpack the software package {\tt goosepkg{\it nnn}.tar.gz}, where {\it
  nnn} is the version number. The package includes the following
items:

\vspace*{3mm}
\begin{tabular}{ll}
  00readme                  & Summary of the package.\\
  00readme-j                & 00readme in Japanese.\\
  doc/                      & User's guide and other documents.\\
  scripts/                  & install and backup scripts.\\
  bin/goosecc               & A tool to build executables for hardware accelerators.\\
  include/                  & Header files.\\
  lib/                      & Libraries.\\
  singutil/                 & GRAPE-DR control library.\\
  sample/                   & Examples of application programs which can be compiled with the Goose C Compiler.\\
  init/                     & Data files used by programs in sample/.\\
  misc/                     & Templates for environment variable set up, a program to generate sample data, etc.\\
\end{tabular}\\

\vspace*{3mm}
Set the environment variables listed below:

\vspace*{3mm}
\begin{tabular}{ll}
  LSUMPPATH         & : The path to the directory at which LSUMP is installed.\\
  VSMPATH           & : The path to the directory at which VSM is installed.\\
  GRAPEPKGPATH      & : The path to the directory at which GRAPE Software Package is installed.\\
\end{tabular}\\

\vspace*{5mm}
\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
Example for sh:
kawai@localhost>export LSUMPPATH=/home/kawai/src/lsumppkg
kawai@localhost>export VSMPATH=/home/kawai/src/vsm
kawai@localhost>export GRAPEPKGPATH=/home/kawai/src/grapepkg

Example for csh:
kawai@localhost>setenv LSUMPPATH /home/kawai/src/lsumppkg
kawai@localhost>setenv VSMPATH /home/kawai/src/vsm
kawai@localhost>setenv GRAPEPKGPATH /home/kawai/src/grapepkg
\end{verbatim}
\end{minipage}\\[3mm]

\noindent
You can find template files for above mentioned setting at\\[-3mm]

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
$goosepkg/misc/rc/genv.sh
$goosepkg/misc/rc/genv.csh
\end{verbatim}
\end{minipage}\\

\noindent
You may copy one of them and modify it to suit your environment.
The environment variables below are also available, but not mandatory.

\vspace*{3mm}
\begin{tabular}{ll}
  CC         & : Specify the C compiler to be used.\\
  CFLAGS     & : Specify arguments passed on to the C compiler.\\
\end{tabular}\\
%

\Subsection{Running the Install Script}

\vspace*{5mm} Change directory to {\tt \$goosepkg} and run {\tt
  \$goosepkg/script/install}. It will generate libraries, utilities,
and data used by sample programs.\\

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
kawai@localhost>./script/install 
cc -O3 -c singutil.c -I/home/kawai/grapepkg/include \
-L/home/kawai/grapepkg/lib -fopenmp
ar -r libsing.a singutil.o
ar: creating libsing.a
ranlib libsing.a
cc    -c -o mkdist.o mkdist.c
cc    -c -o util.o util.c
cc  -o mkdist mkdist.o util.o -lm
n: 1024  output file: pl1k
distribution: Plummer

...

done
\end{verbatim}
\end{minipage}\\[5mm]

\noindent
This is all what you need for the installation.  If the package is
successfully installed, all sample programs in {\tt
  \$goosepkg/sample/} directory should be compiled. For example, you
can change directory to {\tt \$goosepkg/sample/s9/} and run {\tt make
  all} to compile a program {\tt s9}, which performs a gravitational
many-body simulation.\\

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
kawai@localhost>pwd
/home/kawai/src/goosepkg/sample/s9
kawai@localhost>make all
../../bin/goosecc -v3 --goose-arch gdr -o s9 sticky9.c  -lm -O3
$LSUMPPATH                           : /home/kawai/src/lsump
$VSMPATH                             : /home/kawai/src/vsm
$GRAPEPKGPATH                        : /home/kawai/src/grapepkg

...

=========================================
 executable file generated successfully.
=========================================
cc -O3 sticky9.c -o s9_host -lm  -fopenmp
kawai@localhost>ls
Makefile  goosetmp  inputpara9  s9  s9_host  sticky9.c
\end{verbatim}
\end{minipage}\\[5mm]

\noindent
On successful build, you will see two executables, namely, {\tt s9}
and {\tt s9\_host}. The former performs the simulation using a
hardware accelerator.  The latter performs the simulation purely on
the host computer.

\clearpage
\Section{Basic Usage}\label{sec:basics}

\Subsection{Programs Handled by Goose}\label{sec:target}

Although Goose handles a program written in C, it does not recognize
all grammars defined by the language specification. It handles only
descriptions suitable for SIMD-type accelerators.  The rest of the
program is passed on to a conventional C compiler, such as {\tt gcc},
and compiled for run on the host computer.
%
Here, a description "suitable for SIMD-type accelerators" is defined as
the one which satisfy the followings:
%
\begin{itemize}

\item[(a)] Can be highly parallelized, and,

\item[(b)] The required amount of data transfer among the accelerator
    and the external memory (or the host computer) is small,
    relatively to the amount of the calculation to be performed on the
    accelerator.

\end{itemize}
%

Goose handles calculations which can be expressed as:
%
\begin{equation}
  \Vec{r_i}  =  f(\Vec{x_i}), \label{eq:simd}
\end{equation}
%
where $\Vec{x_i}$ are input parameters and $\Vec{r_i}$ are calculation
results. Results of calculations for different $i$ must not depend
each other, and they must be able to run in parallel.
%
Among the above-mentioned calculations, Goose is especially optimized
for the ones which can be expressed as:
%
\begin{equation}
  \Vec{r_i} = \sum_j f(\Vec{x_i}, \Vec{y_j}). \label{eq:sumj}
\end{equation}
%
A calculation described in the form of equation (\ref{eq:simd})
satisfies the condition (a), if the different number of $i$ is large
enough.  If it also can be described in the form of equation
(\ref{eq:sumj}), it satisfies the condition (b), too. This is because
the calculation for various different $i$-s can be performed using a
shared set of $\Vec{y_j}$.  Therefore, the amount of data necessary to
be supplied is smaller, compared to a case different $\Vec{y_j}$-s are
required for different $i$-s.
%
Goose also can handle a calculation of the form of equation
(\ref{eq:simd}) but not of equation (\ref{eq:sumj}). In such a case, however,
the potential performance of the accelerator may not fully be utilized.

The necessary number of $i$ and $j$ depends on the architecture of the
accelerator. In order to perform a calculation using all processor
elements of a GRAPE-DR, for example, the number of $i$ and $j$ should
be larger or equal to 128 and 16, respectively ({\it cf.} section
\ref{sec:gdr}).  Otherwise some processor elements are not utilized
and remain idle during the calculation.

\vspace*{5mm}
\hspace*{5mm}
\begin{minipage}{150mm}

    Example : Calculation of gravitational interactions among point
    masses can be expressed by equation (\ref{eq:sumj}). The total
    force $\Vec{f_i}$ on a point mass $i$ from all other point masses
    is given by:
%
\begin{equation}
  \Vec{f_i} = \sum_{j \ne i} \frac{G m_i m_j (\Vec{x_j} - \Vec{x_i})}{|\Vec{x_j} - \Vec{x_i}|^3}
  \label{eq:gravity}
\end{equation}
%
where $G$ is the gravitational constant, $\Vec{x_i}$ and $m_i$ are
position and mass of a particle $i$.
\end{minipage}\\

In C language, equation (\ref{eq:simd}) can be expressed in various
way. For simplicity, however, Goose recognize an expression by 'for'
statement only. In the case of equation (\ref{eq:sumj}), it recognize
an expression by a nested 'for' statement only.  Expressions other
than 'for' statement, such as 'while' statement or 'goto' statement,
are not recognized.

\vspace*{5mm}
\hspace*{5mm}
\begin{minipage}{150mm}
Example : Equation (\ref{eq:gravity}) can be described by a nested 'for' statement:
%
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
for (i = 0; i < n; i++) {
  for (j = 0; j < n; j++) {
    if (i != j) {
      dx[0] = x[j][0] - x[i][0];
      dx[1] = x[j][1] - x[i][1];
      dx[2] = x[j][2] - x[i][2];
      r2 = dx[0]*dx[0] + dx[1]*dx[1] + dx[2]*dx[2];
      mmr3 = G * m[i] * m[j] / (sqrt(r2) * r2);
      f[i][0] += mmr3 * dx[0];
      f[i][1] += mmr3 * dx[1];
      f[i][2] += mmr3 * dx[2];
    }
  }
}
\end{verbatim}
%
\end{minipage}

\Subsection{Design Flow of a Program}

Basically, user write a program in C language and process it with a
command {\tt \$goosepkg/bin/goosecc}. The command generates an
executable for a hardware accelerator.
The design flow of an application program is as follows:

\begin{enumerate}

\item[(step 1)] Write an application program in C language, and test
    it on the host computer (PC). A code fragment which is expected
    to run efficiently on the hardware accelerator ({\it cf.} section
    \ref{sec:target}) should be described with 'for' statements. Do
    not use 'while' or 'do while' statements.

    In the case of calculation which is symmetric for the inner/outer
    loops of the nested 'for' statement ({\it e.g.} particle
    interactions which satisfy the law of reciprocal action), the
    calculation is often described in the following manner, in order
    to save the amount of calculation. However, such a description
    should be avoided since Goose cannot handle it.

\vspace*{5mm}
\hspace*{5mm}
\begin{minipage}{150mm}
    Example : A description of equation (\ref{eq:gravity}) in C
    language. Optimized to save the amount of calculation using the
    symmetric nature of gravity. Such an optimization {\bf should be
      avoided} since Goose cannot handle the inner 'for' statement
    with non-constant loop range.
%
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
for (i = 0; i < n; i++) {
  for (j = i+1; j < n; j++) {
    dx[0] = x[j][0] - x[i][0];
    dx[1] = x[j][1] - x[i][1];
    dx[2] = x[j][2] - x[i][2];
    r2 = dx[0]*dx[0] + dx[1]*dx[1] + dx[2]*dx[2];
    mmr3 = G * m[i] * m[j] / (sqrt(r2) * r2);
    f[i][0] += mmr3 * dx[0];
    f[i][1] += mmr3 * dx[1];
    f[i][2] += mmr3 * dx[2];
    f[j][0] -= mmr3 * dx[0];
    f[j][1] -= mmr3 * dx[1];
    f[j][2] -= mmr3 * dx[2];
  }
}
\end{verbatim}
%
\end{minipage}

\item[(step 2)] Attach an OpenMP directive ({\tt \#pragma omp parallel
      for}) to the 'for' statement mentioned in (step 1). Then test
    the program in a multi-threaded environment. This step may be
    skipped, although it is recommended as a method to check sanity of
    the 'for' statement when it is run in parallel. It is also useful
    to estimate the speed up factor. For the usage of OpenMP, refer to
    other documents such as {\tt http://openmp.org/}.

\item[(step 3)] Attach a Goose directive ({\tt \#pragma goose parallel
      for}) to the 'for' statement mentioned in (step 1) and (step 2).
    Then compile the program with {\tt goosecc} to obtain an
    executable. The executable generated performs the 'for' statement
    on the hardware accelerator. Other parts are performed on the host
    computer.

\end{enumerate}

\Subsection{Tutorial 1 -- A Numerical Integration}\label{sec:tut1}

In this and next sections, basic usage of Goose is shown via two
programming examples.

\vspace*{3mm}

In this section, we describe Goose directives using a sample program\\
{\tt \$goosepkg/sample/midpoint/midpoint.c}.  This program numerically
calculates \mbox{$\int_0^1 4 /(1+x^2) dx$}, which should analytically
be equal to $\pi$.
The primary part of this program is a 'for' statement quoted below:\\

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
  for(i=0;i<n;i++) {
    x = (i+0.5)*dx;
    sum += integrand(x)*dx;        
  }
\end{verbatim}
\end{minipage}

\vspace*{3mm}
\noindent
The domain $[0, 1]$ is divided into {\tt n} segments.  At each center
of the segment, the code multiplies the value of the integrand and {\tt
  dx}, and sums it up to obtain {\tt sum}. The integrand is defined as
a C function, {\tt integrand()}.

\vspace*{3mm}
You can compile the program for the host computer, using a conventional C compiler:\\

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
kawai@localhost>cd $goosepkg/sample/midpoint/
kawai@localhost>cc midpoint.c -o midpoint_host
\end{verbatim}
\end{minipage}

\vspace*{3mm}
\noindent
The generated executable, {\tt midpoint\_host}, outputs the following result:\\

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
kawai@localhost>./midpoint_host
n:16384  sum:3.14159265390022e+00  sum/M_PI-1.0:9.881305e-11
\end{verbatim}
\end{minipage}

\vspace*{5mm}
Now we confirmed that the program works on the host computer as we
expected.  The next step is to put the above mentioned 'for' statement
on a hardware accelerator. Note that the 'for' statement is preceded
by a Goose directive, {\tt \#pragma goose parallel for}, hereafter we
call this directive as a "Goose-for directive".\\

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
#pragma goose parallel for icnt(i) result(sum) ip(i) shared_ro(dx)
  for(i=0;i<n;i++) {
    x = (i+0.5)*dx;
    sum += integrand(x)*dx;        
  }
\end{verbatim}
\end{minipage}

\vspace*{3mm}
\noindent
A Goose-for directive directs the Goose C compiler to put the 'for'
statement on a hardware accelerator. Other parts of the program is
performed on the host computer.

\vspace*{3mm}
If you look inside the Goose 'for' directive, you would notice that
some arguments, such as {\tt icnt(i)}, are given for the directive:

\begin{itemize}

\item The argument {\tt icnt(i)} denotes that a variable {\tt i} is
    used as the loop counter of the 'for' statement.

\item The argument {\tt result(sum)} denotes that a variable {\tt sum}
    need to be sent back from the accelerator to the host computer, as
    the calculation result.

\item The argument {\tt ip(i)} denotes that a variable {\tt i} need to
    be sent from the host computer to the accelerator, as an input
    parameter of the calculation.

\item Similar to {\tt ip()}, the argument {\tt shared\_ro(dx)} also
    denotes that a variable {\tt dx} need to be sent to the
    accelerator.  The difference from the argument {\tt ip()} is that
    {\tt dx}, a variable specified by {\tt shared\_ro()}, does not
    change its value inside the loop, while {\tt i}, a variable
    specified by {\tt ip()}, does.

\end{itemize}

\noindent
As we describe later in section \ref{sec:iotype}, some of these
arguments can be omitted. We recommend, however, that you explicitly
specify the loop counter and the I/O type ({\it cf.} section
\ref{sec:iotype}) of all variables except {\tt private} ones, at least
until you get familiar with behavior of the Goose compiler.

\vspace*{3mm}
Pay your attention to the definition of the integrand : \\[-3mm]

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
#pragma goose func
double integrand(double x)
{
  double s;
  s = 4.0/(1.0+x*x);
  return s;
}
\end{verbatim}
\end{minipage}

\vspace*{3mm}
\noindent
The function is preceded by a Goose directive, {\tt \#pragma goose
  func}, hereafter we call this directive as a "Goose-func directive".
The directive should be attached to a function, if the function is
used inside a 'for' loop, which a Goose-for directive is attached
to. There are some restrictions for a function to be attached by a
Goose-func directive: The function should have exactly one argument;
It should explicitly return a value by a {\tt return} statement; A
Goose-func directive cannot be attached to a function of type {\tt
  void}, or one without {\tt return} statement.

\vspace*{3mm} Now we understood all the Goose directives in the
program.  Next we compile the program to build an executable for a
hardware accelerator. For the compilation, we use a command {\tt
  \$goosepkg/bin/goosecc}. Change directory to {\tt
  \$goosepkg/sample/midpoint/}, and run {\tt goosecc} with arguments
as shown below.
(see section \ref{sec:gooseccopt} for a complete list of the command-line arguments):\\

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
kawai@localhost>../../bin/goosecc midpoint.c -o midpoint
========================
 processing midpoint.c.
========================
Info    : Goose::GforHandler : No j-loop found. The entire i-loop \
body is used as a kernel.
Info    : Goose::GforHandler : Recognized as an ip var : i
Info    : Goose::GforHandler : Recognized as a shared_ro var : dx
...
=========================================
 executable file generated successfully.
=========================================
\end{verbatim}
\end{minipage}

\vspace*{3mm}
\noindent
On successful compilation, an executable {\tt midpoint} will be
generated.  When you set up a hardware accelerator and run {\tt
  midpoint}, the 'for' statement attached by a Goose-for directive
runs on the accelerator. Other parts of the program are performed on
the host computer. The outputs should be the same as that of {\tt midpoint\_host}.

\Subsection{Tutorial 2 -- Gravitational Interactions among Point-Masses}\label{sec:tut2}

In this section, we use a sample program {\tt
  \$goosepkg/sample/gravity/gravity.c}.  This program calculates
acceleration of particles $a_i$ in a many-body system, in which
point-mass particles are interacting via gravity.  The gravitational
force is originally described by equation (\ref{eq:gravity}).  Here,
we adjust the spatial and mass scaling so that the gravitational
constant $G$ equals unity, and introduce a "softening parameter",
$\epsilon$, in order to avoid numerical inaccuracy:
%
\begin{equation}
  \Vec{a_i} = \sum_{j \ne i} \frac{m_j (\Vec{x_j} - \Vec{x_i})}
{(|\Vec{x_j} - \Vec{x_i}|^2 + \epsilon^2)^{3/2}}
\label{eq:gravity2}
\end{equation}
%
In the sample program, you can find a nested 'for' statement, that describes the equation above.\\

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
#pragma goose parallel for icnt(i) jcnt(j) \
  ip(x[i][0..2]) jp(x[j][0..2],m[j]) result(a[i][0..2],pot[i])\
  precision ("double") 
    for(i=0;i<n;i++) {
      for(k=0;k<3;k++) a[i][k] = 0.0;
      pot[i] = 0.0;
      for(j=0;j<n;j++) {
        for(k=0;k<3;k++) dx[k] = x[j][k] - x[i][k];
        r2 = dx[0]*dx[0] + dx[1]*dx[1] + dx[2]*dx[2] + eps2;
        rinv = rsqrt(r2);
        mrinv = m[j]*rinv;
        mr3inv = mrinv*rinv*rinv;
        for(k=0;k<3;k++) a[i][k] += mr3inv * dx[k];
        pot[i] += -mrinv;      
      }
    }
    get_cputime(&lt,&st);
    total_time += lt;
    for(i=0;i<n;i++) pot[i] += m[i]/sqrt(eps2);
\end{verbatim}
\end{minipage}

\vspace*{3mm}
\noindent
Equation (\ref{eq:gravity2}) is in the form of equation
(\ref{eq:sumj}). In such a case, the program should be described using
a nested 'for' statement ({\it cf.} section \ref{sec:target}).  A
Goose-for directive directs the Goose C compiler to put the nested
'for' statement on a hardware accelerator. Other parts of the program
is performed on the host computer.

\vspace*{3mm} The followings are arguments to the Goose-for directive
(some of them are the ones described in section \ref{sec:tut1}):

\begin{itemize}

\item The argument {\tt icnt(i)} denotes that a variable {\tt i} is
    used as the loop counter of the outer 'for' statement, while {\tt
      jcnt(j)} denotes {\tt j} is the counter of the inner 'for'.

\item The argument {\tt ip(x[i][0..2])} denotes that a variable {\tt
      x[i][0..2]} need to be sent from the host computer to the
    accelerator, as an input parameter of the calculation. Here, {\tt
      0..2} is interpreted as three indices, {\tt 0, 1,} and {\tt 2}.
    That is, an expression {\tt x[i][0..2]} is equivalent to {\tt
      x[i][0], x[i][1], x[i][2]}. The two-dot notation, '{\tt ..}', is
    valid only in the Goose directives.

\item The argument {\tt jp(x[j][0..2],m[j])} denotes that variables
    {\tt x[j][0..2]} and {\tt m[j]} need to be sent from the host
    computer to the accelerator, as input parameters of the
    calculation. The difference from the argument {\tt ip()} is that
    variables specified by {\tt jp()} change their value inside the
    inner loop, while variables specified by {\tt ip()} do not.

\item The argument {\tt result(a[i][0..2],pot[i])} denotes that
    variables {\tt a[i][0..2]} and {\tt pot[i]} need to be sent back
    from the accelerator to the host computer, as the calculation
    result.

\item The argument {\tt precision("double")} denotes that variables
    are expressed in 64-bit floating-point format. Types specified in
    variable declarations of C language, if any, would be ignored.
    valid values for the {\tt precision()} argument are listed in
    section \ref{sec:foropt}.

\end{itemize}

\vspace*{3mm}
{\bf Notice:}

\begin{itemize}

\item If the name of a function attached by a Goose-func directive is
    predefined as Goose-precompiled function, the user definition is
    ignored ({\it cf.} section \ref{sec:pragmafunc}).  For example,
    {\tt rsqrt()} is one of a Goose-precompiled function.
    Thus, the definition at the top of the sample program:\\

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
double rsqrt(double r2)
{
  return 1.0/sqrt(r2);
}
\end{verbatim}
\end{minipage}

\vspace*{3mm}
\noindent
is skipped by {\tt goosecc}.

\item A self-assignment operator {\tt -=} is not recognized.  Use {\tt
      +=} instead.

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
Example : pot[i] += -mrinv
\end{verbatim}
\end{minipage}

\item 'For' statements nested further inside a nested 'for' loop, would be unrolled.
    For example,\\[-3mm]

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
for(k=0;k<3;k++) dx[k] = x[j][k] - x[i][k];
\end{verbatim}
\end{minipage}

\vspace*{3mm}
is equivalent to\\

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
dx[0] = x[j][0] - x[i][0];
dx[1] = x[j][1] - x[i][1];
dx[2] = x[j][2] - x[i][2];
\end{verbatim}
\end{minipage}

\vspace*{3mm}

{\tt Goosecc} does not handle a 'for' statement whose loop-range is
infinite or not determined at the compilation time. Such a statement
is ignored, and {\tt goosecc} warns about it ({\it cf.} section
\ref{sec:cgrammarfor}).

\item Since the sample program uses a mathematical function {\tt
      sqrt()}, a command-line option {\tt -lm} is required.\\

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
kawai@localhost>cd $goosepkg/sample/gravity/
kawai@localhost>../../bin/goosecc gravity.c -o gravity -lm
\end{verbatim}
\end{minipage}

\vspace*{3mm}

\end{itemize}

\Subsection{Sample Programs}

A directory {\tt \$goosepkg}/sample/ contains examples of application
programs which can be compiled by {\tt goosecc}. These would be useful
as reference designs to help understanding of the usage of the Goose
directives.

\vspace*{3mm}
\begin{tabular}{ll}
\hline
\multicolumn{2}{l}{Nested 'for' statement (equation (\ref{eq:sumj}))}\\
\hline
  gravity/              & Calculate gravitational interactions\\
                        & (used in the tutorial in section \ref{sec:tut2}).\\[3mm]

  gravity\_cutoff/      & Calculate gravitational interactions with P${}^3$M cutoff, \\
                        & under a periodic boundary condition.\\[3mm]
  hermite/              & Calculate gravitational interactions and its time derivatives.\\[3mm]
  tree/                 & Calculate gravitational interactions using the Barnes-Hut Tree algorithm.\\[3mm]
  s9/                   & Performs a gravitational many-body simulation\\
                        & (the leaf-frog integrator, shared timestep).\\[3mm]
  s8/                   & Performs a gravitational many-body simulation\\
                        & (the Hermite integrator of the 4th order, individual timestep).\\[3mm]
  vdw/                  & Calculate van der Waals interactions\\
                        & (the Lennard-Jones potential).\\[3mm]
  sph/                  & Calculate the accelerations for SPH particles\\
                        & (Spline kernel, no artificial viscosity).\\[3mm]
\hline\\[3mm]

\hline
\multicolumn{2}{l}{A single 'for' statement (equation (\ref{eq:simd}))}\\
\hline
  midpoint/             & Numerically calculate $\int_0^1 4 /(1+x^2) dx$ using the Midpoint method\\
                        & (used in the tutorial in section \ref{sec:tut1}).\\[3mm]
  mesh1d/               & On each grid-point in a one-dimensional domain space, calculate\\
                        & a weighted average of a given function.\\[3mm]
\hline
\end{tabular}\\

\vspace*{3mm}
\noindent
In each directory, type {\tt make all} to generate an executable for a
hardware accelerator. An executable that performs the calculation
without the accelerator, is also generated.

\clearpage
\Section{Goose Directives}\label{sec:pragma}

Goose recognizes two directives below:

\paragraph{Goose-for directive}:
{\tt \#pragma goose parallel for [{\it optional-arguments}]}\\
Inserted just before a 'for' statement to make the statement run on a hardware accelerator.

\paragraph{Goose-func directive}:
{\tt \#pragma goose func}\\
Inserted just before a function definition, so that the function can
be used inside a 'for' statement which a Goose-for directive is attached to.

\vspace*{5mm}
\hspace*{5mm}
\begin{minipage}{150mm}
Notice :
Goose directive must be written in a single line. Otherwise the 
line-feed character must be escaped by a backslash.\\

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
  Expample : #pragma goose \
             parallel for
\end{verbatim}
\end{minipage}
\end{minipage}

\Subsection{Goose-for Directive}\label{sec:pragmafor}

A Goose-for directive is inserted just before a 'for' statement.  It
directs the Goose to make the statement run on a hardware accelerator.
It is usually attached to a nested 'for' statement, although it can be
attached to a single 'for'. 'For' statements further inside the nested
'for' loop, would be unrolled.\\

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
Example :
  #pragma goose parallel for
    for (i = 0; i < ni; i++) {        // outer 'for'.
      for (j = 0; j < nj; j++) {      // inner 'for'.
        for (k = 0; k < 3; k++) {     // 'for' to be unrolled.
          dx[k] = x[j][k] - x[i][k];
        }
        r2 = dx[0]*dx[0] + dx[1]*dx[1] + dx[2]*dx[2] + eps2;
        rinv = rsqrt(r2);
        mf = m[j]*rinv*rinv*rinv;
        for (k = 0; k < 3; k++) {     // 'for' to be unrolled.
          a[i][k] += mf * dx[k];
        }
        p[i] -= m[j] * rinv;
      }
    }
\end{verbatim}
\end{minipage}\\[1mm]

\noindent
Types of all variables used by the accelerator are {\tt double}, or
the one specified by {\tt \#pragma goose parallel for precision} ({\it
  cf.} section \ref{sec:foropt}).  Types specified in variable
declarations of C language, if any, would be ignored.

\Subsubsection{I/O Type of Variables}\label{sec:iotype}

The host computer and the hardware accelerator communicate the values
of variables used inside a 'for' statement attached by a Goose-for
directive.  Depending on the communication required, each variable is
assigned an "I/O type" attribute.\\

\noindent
{\bf I/O Types}\\[3mm]
\begin{tabular}{ll}
\hline
type       & description \\
\hline
\hline
ip         & Input parameters which depend on $i$, that is, $\Vec{x_i}$ in equation (\ref{eq:sumj}).\\
           & Sent from the host computer to the accelerator.\\
jp         & Input parameters which depend on $j$, that is, $\Vec{x_j}$ in equation (\ref{eq:sumj}).\\
           & Sent from the host computer to the accelerator.\\
shared\_ro & Input parameters which depend neither on $i$ nor $j$.\\
           & Sent from the host computer to the accelerator.\\
private    & Intermediate variables temporarily used during the calculation.\\
           & Used only on the accelerator, and not sent back to the host computer.\\
result     & Results of the calculation, that is, $\Vec{r_i}$ in equation (\ref{eq:sumj}).\\
           & Sent back from the accelerator to the host computer.\\
\hline
\end{tabular}

\vspace*{3mm}
\noindent
The I/O type of a variable is automatically inferred by {\tt goosecc}
following the rules shown below. However, you can explicitly specify
the I/O type by giving an argument to the Goose-for directive ({\it
  cf.}  section \ref{sec:foropt}).

\vspace*{5mm}
\noindent
{\bf I/O Type Inference Rules:}\\

Letting the loop counter of the outer and inner 'for' being
{\tt i} and {\tt j},

\begin{itemize}

\item A variable indexed by {\tt i} is inferred as a variable of I/O
    type {\tt ip}. If the variable is used as an lvalue, however, it
    is inferred as a {\tt result} type.

\item A variable indexed by {\tt j} is inferred as a {\tt jp} type.

\item A variable used as an lvalue is inferred as a {\tt private}
    type, if none of above mentioned conditions are met. It can be
    recognized as a {\tt result} type, if explicitly specified by an
    argument to the Goose-for directive. A variable {\tt sum} in \\
    {\tt \$goosepkg/sample/midpoint/midpoint.c} is such an example
    ({\it cf.} section \ref{sec:tut1}).

\item A variable never used as an lvalue is inferred as a {\tt shared\_ro} type.

\end{itemize}


\clearpage
\Subsubsection{Optional Arguments}\label{sec:foropt}

A Goose-for directive may take various optional arguments including
I/O type specifiers. Below is the complete list of valid arguments.

\vspace*{5mm}
\noindent
\begin{minipage}{160mm}
{\bf Optional Arguments:}\\[3mm]
\begin{tabular}{ll}
\hline
synopsis               & description \\
\hline
\hline
{\tt icnt({\it var})}  &                 Use a variable {\it var} as the loop counter of the outer 'for' statement.\\
                       &                 Default value is {\tt i}.\\[3mm]
%
{\tt jcnt({\it var})}  &                 Use a variable {\it var} as the loop counter of the inner 'for' statement.\\
                       &                 Default value is {\tt j}.\\[3mm]
%
{\tt result({\it var, ...})}  &          Set the I/O type of a variable {\it var} to {\tt result}.\\[3mm]
%
{\tt ip({\it var, ...})}  &              Set the I/O type of a variable {\it var} to {\tt ip}.\\[3mm]
%
{\tt jp({\it var, ...})}  &              Set the I/O type of a variable {\it var} to {\tt jp}.\\[3mm]
%
{\tt private({\it var, ...})}  &         Set the I/O type of a variable {\it var} to {\tt private}.\\[3mm]
%
{\tt shared\_ro({\it var, ...})}  &      Set the I/O type of a variable {\it var} to {\tt shared\_ro}.\\[3mm]
%
{\tt precision("{\it prec}")}  &
\begin{minipage}[t]{120mm}
    Set the numerical format of variables to "{\it prec}".  Valid
    values as "{\it prec}" are: "{\tt double}" for 64-bit
    floating-point and "{\tt double-single}" for 64-bit floating-point
    addition and 32-bit floating-point multiplication.  Two more
    values, "{\tt single}" (32-bit floating-point) and "{\tt
      quadruple}" (128-bit floating-point) will be supported soon.
    The default value is "{\tt double}".\\[3mm]
\end{minipage}\\
%
{\tt asmfile("{\it filename}")}  &
\begin{minipage}[t]{120mm}
    Specify an assembly code passed on to the assembler.  If given,
    the code is used to build an executable, instead of the one
    generated by {\tt goosecc} ({\it e.g.,} {\tt foo\_0.vsm}~~{\it
      cf.} section \ref{sec:gooseccinside}). This argument is used
    when you need to hand-tune the assembly code.\\
\end{minipage}\\
%
\multicolumn{2}{l}{~~(the list continues to the next page...)} \\[3mm]
%
\hline
%
\end{tabular}
\end{minipage}

\clearpage
\vspace*{3mm}
\noindent
\begin{minipage}{160mm}
{\bf Optional Arguments (...continued from the previous page):}\\[3mm]
\begin{tabular}{ll}
\hline
synopsis               & description \\
\hline
\hline
%
{\tt njp\_write({\it var})}  &
\begin{minipage}[t]{120mm}
    When given, only the first {\it var} variables of {\tt jp} type is
    sent to the accelerator. Variables with index {\tt j} larger than,
    or equal to {\it var} are not sent to the accelerator. Their
    values remain unchanged from the previous calculation and reused.
    By doing this, the amount of data transfer would be reduced ({\it
      cf.} {\tt \$goosepkg/sample/s8/}).  All {\tt jp}-type variables
    are sent to the accelerator, if this argument is not given.\\
\end{minipage}\\
%
{\tt nip\_pack({\it uint})}  &
\begin{minipage}[t]{120mm}
    Set the amount of calculation processed by a single processor
    element (PE). If this argument is not given, one PE processes
    calculations for {\it nvec} indices ({\tt i}-s) (Here, {\it nvec}
    is the vector loop length of the PE). When given, one PE processes
    {\it uint $\times$ nvec} indices. Effective performance of a
    calculation may be improved when {\it uint} is set to around
    2$\sim$5, since the communication overhead between the host
    computer and the accelerator is hidden.\\
\end{minipage}\\
\hline
%
\end{tabular}
\end{minipage}

\clearpage
\noindent
Notice:

\begin{itemize}

\item In order to specify the I/O types of multiple variables, give a
    comma-separated list of variables to the
    type specifier.\\[-5mm]

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
Example :  ip(a, b, c)
\end{verbatim}
\end{minipage}

\item Strictly speaking, an I/O-type attribute is assigned not to a
    variable but to a value referenced by a variable. Therefore, the
    attribute is assigned to, for example, an array element,
    dereference of a pointer, and a member of a structure.\\[-3mm]

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
Example :
     double x[255];
     int index[255], j;
     struct a_struct_t a_struct, *a_struct_p;
     ip(x[0]);                 // array element.
     ip(x[i]);                 // array element with variable index.
     ip(x[index[i]]);          // array element with index expressed
                               // using another array element.
     jp(*(x+j));               // dereference of a pointer.
     jp(a_struct.a_member);    // a member of a structure.
     jp(a_struct_p->a_member); // a member of a structure refered
                               // by a pointer.
\end{verbatim}
\end{minipage}

\item A range of an array can be specified by a two-dot notation, '{\tt ..}'.\\[-5mm]

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
Example : ip(x[i][0..2])
\end{verbatim}
\end{minipage}

\noindent
is equivalent to the below.\\[-5mm]

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
ip(x[i][0], x[i][1], x[i][2])
\end{verbatim}
\end{minipage}

\item The hardware accelerator and the host computer do not share any
    address space.  Therefore, the address of variables cannot be
    passed on to each other.
    The I/O-type specifier is not valid for reference to an address.\\[-3mm]

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
Example:
     double x, y[255];
     int i;
     ip(&x);                // NG
     ip(x);                 // OK
     ip(y);                 // NG
     ip(y[0]);              // OK
     ip(y[i]);              // OK
     ip(*(y + i));          // OK
\end{verbatim}
\end{minipage}

\end{itemize}

\Subsection{Goose-func Directive}\label{sec:pragmafunc}

Goose-func directive, {\tt \#pragma goose func},
should be attached to a function, if the function is
used inside a 'for' loop, which a Goose-for directive is attached
to. There are some restrictions for a function to be attached by a
Goose-func directive: The function should have exactly one argument;
It should explicitly return a value by a {\tt return} statement; A
Goose-func directive cannot be attached to a function of type {\tt
  void}, or one without {\tt return} statement.

The value of argument and the returning value are treated as {\tt
  double}, or the numerical format specified by {\tt \#pragma goose
  parallel for precision} ({\it cf.} section \ref{sec:foropt}).  Types
specified in variable
declarations of C language, if any, would be ignored.\\

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
Example :
     #pragma goose func
     static double rsqrt(double r2)
     {
        int i;
        double x0 = 1.0;
        for (i = 0; i < 4; i++) {
            x0 = 0.5*x0*(3.0 - r2*x0*x0);
        }
        return x0;
     }
\end{verbatim}
\end{minipage}

\vspace*{3mm}
If the name of a function attached by a Goose-func directive is
predefined as Goose-precompiled function, the user definition is
ignored and the precompiled one is used. Below is the complete list of
Goose-precompiled functions:

\begin{itemize}

\item {\tt rsqrt(x)} : Returns the inverse square root of the argument {\tt x}, that is, $1/\sqrt{x}$.

\end{itemize}

\clearpage
\Section{C-Language Grammar Handled by Goose}\label{sec:cgrammar}

Although {\tt goosecc} handles a program written in C, it does not
recognize all grammars defined by the language specification.  In this
section, C-language grammars which can be handled by {\tt goosecc} are
described.

\Subsection{Assign Statement}

{\tt Goosecc} can compile most of assign statements.\\[-3mm]

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
Example :
  r2 = dx[0]*dx[0] + dx[1]*dx[1] + dx[2]*dx[2];
  a[i][k] += mf * dx[k]; // a self assignment.
  double eps = 0.01;     // a variable declaration with an initializer.
  rinv = rsqrt(r2);      // an assignment of a returning value of a function.
\end{verbatim}
\end{minipage}

\vspace*{3mm}
\noindent
An assign expression can be evaluated as a value.\\[-3mm]

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
Example :
  x = y = z = 0.0;
  rinv = rsqrt(r2 = dx[0]*dx[0] + dx[1]*dx[1] + dx[2]*dx[2]);
\end{verbatim}
\end{minipage}

\Subsection{If Statement}

An 'if' statement can be compiled if its conditional expression matches
the following form:\\[-3mm]

\hspace*{5mm}
\begin{minipage}{150mm}
{\tt
{\it var0} {\it rel\_op} {\it var1}
}
\end{minipage}

\vspace*{3mm}
\noindent
Here, {\it var0} and {\it var1} are variables or number literals,\\
{\it rel\_op} is a relational operator one of : $<, <=, >, >=, ==$, and $!=$.\\[-3mm]

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
Example :
  if (i != j) {
    f[i] += fij;
  }
\end{verbatim}
\end{minipage}

\vspace*{3mm}
\noindent
The statements can be nested.\\[-3mm]

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
Example :
  if (q > 2.0) {
    wkernel = 0.0;
  }
  else if (q > 1.0) {
    wkernel = 0.25*(2.0-q)*(2.0-q)*(2.0-q);
  }
  else {
    wkernel = 1.0-1.5*q*q+0.75*q*q*q;
  }
\end{verbatim}
\end{minipage}

\Subsection{Conditional Operator}

A conditional operator can be compiled if its conditional expression matches
the following form:\\[-3mm]

\hspace*{5mm}
\begin{minipage}{150mm}
{\tt
{\it var0} {\it rel\_op} {\it var1}
}
\end{minipage}

\vspace*{3mm}
\noindent
Here, {\it var0} and {\it var1} are variables or number literals,\\
{\it rel\_op} is a relational operator one of : $<, <=, >, >=, ==$, and $!=$.\\[-3mm]

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
Example : wkernel = q > 2.0 ? 0.0 : 0.25*(2.0-q)*(2.0-q)*(2.0-q);
\end{verbatim}
\end{minipage}

\vspace*{3mm}
\noindent
The operators can be nested.\\[-3mm]

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
Example :
  wkernel = q > 2.0 ? 0.0 :
                      q > 1.0 ? 0.25*(2.0-q)*(2.0-q)*(2.0-q) :
                                1.0-1.5*q*q+0.75*q*q*q;
\end{verbatim}
\end{minipage}

\Subsection{For Statement}\label{sec:cgrammarfor}


A 'For' statement further inside a nested 'for' loop attached by a
Goose-for directive, or one inside a function attached by a Goose-func
directive, would be unrolled. If the 'for' statement is nested, all
the loops are recursively unrolled.

A 'for' statement can be unrolled only if its loop-range and stride is
fixed at the compilation time. Otherwise the statement is ignored, and
{\tt goosecc} warns about it. More precisely, a 'for' statement is
unrolled, if it matches the following form:\\[-3mm]

\hspace*{5mm}
\begin{minipage}{150mm}
{\tt
for ({\it var} = {\it init\_val} ; {\it var} {\it rel\_op} {\it final\_val} ; {\it var} = {\it var} + {\it inc\_val})
}
\end{minipage}

\vspace*{3mm}
\noindent
Here, {\it var} is a variable, {\it init\_val, final\_val}, and {\it inc\_val} are number literals, and
{\it rel\_op} is a relational operator one of : $<$ and $<=$.
The equation {\it var} = {\it var} + {\it inc\_val} can be written as
{\it var} += {\it inc\_val}. If
{\it inc\_val} is unity, it can be written as {\it var}++, too.\\

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
Example :
  for (k = 0; k < 3; k++) {
    a[i][k] += mf * dx[k];
  }
\end{verbatim}
\noindent
This 'for' statement is unrolled as:
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
     a[i][0] += mf * dx[0];
     a[i][1] += mf * dx[1];
     a[i][2] += mf * dx[2];
\end{verbatim}
\end{minipage}

\Subsection{Return Statement}

A 'return' statement can be used only inside a function attached by a
Goose-func directive.  It cannot be used inside a 'for' statement
attached by a Goose-for directive.

\Subsection{Statements Not Handled by Goose}\label{sec:invalidgrammar}

When {\tt goosecc} find a statement that it cannot handle, it warns
about it and continue the compilation process. The statement is just
ignored. Therefore, for example, a function call to {\tt printf()}
which was introduced just for debugging purpose, may not affect the
calculation on the hardware accelerator.

\clearpage
\Section{Goose Inside}\label{sec:advanced}

\Subsection{{\tt Goosecc} Command-Line Arguments}\label{sec:gooseccopt}

Synopsis:\\[-3mm]

\hspace*{5mm}
\begin{minipage}{150mm}
{\tt
  goosecc [{\it options}] {\it inputfile(s)...}
}
\end{minipage}

\vspace*{3mm}
\noindent
{\it inputfile(s)...} are source files written in C.
Valid {\it options} are as follows:

\vspace*{3mm}
\noindent
{\bf Command-Line Arguments:}\\[3mm]
\begin{tabular}{ll}
\hline
arguments                        & description (the default value is denoted by [~~].) \\
\hline
\hline
--goose-arch $<$ {\it arch} $>$  & Architecture type of the hardware accelerator. [gdr] \\[3mm]
%
--goose-backannotate             & Suppress generation of assembly codes.\\
                                 & Used to avoid hand-tuned assembly codes are over written.\\[3mm]
%
-o $<${\it outputfile} $>$       & Name of executable file to be generated. [a.out] \\[3mm]
%
-I $<${\it headerpath}$>$        & Search path for header files.\\[3mm]
%
--verbose [ = {\it level} ]      & Be verbose. The level may optionally be given.\\
-v [ {\it level} ]               & The higher level gives the more verbose messages.\\
                                 & The minimum value is 0, the default is 2.\\[3mm]
%
--help                           & Print help message.\\
-h                               & \\[3mm]
%
-Wcc ``{\it options}''           & Explicitly specify options passed on to the C compiler.\\
\hline
\end{tabular}

\vspace*{5mm}
\noindent
All options not listed above are not recognized by {\tt goosecc}, and
implicitly passed on to the C compiler.

\Subsection{External Commands Used by {\tt goosecc}}\label{sec:gooseccinside}

{\tt Goosecc} invokes several external commands to build an
executable.  Here we briefly describe this internal procedure,
although a user do not need to take care about it.

At the beginning, {\tt goosecc} scans files given by command-line
arguments and have file extension '.c'. If {\tt goosecc} finds one or
more Goose directives in a file, the file is passed to a C-language
front end ({\tt goosec2q}). Next the output of {\tt goosec2q} is fed
to a compiler for the Goose intermediate-representation ({\tt LSUMP}),
and then processed by an assembler ({\tt vsm}) and by a C compiler ({\tt gcc}).

If no Goose directive is found in an input file, {\tt goosecc} passes
it directly to the C compiler. Files with extensions other than '.c'
are also passed to the C compiler.

The C compiler takes in all inputs from other programs, and
compiles and links them to generate an executable.

%
\Figure{140}{comp_flow.eps}
{Figure : {\tt Goosecc}-internal compilation procedure\\
(For the case GRAPE-DR is chosen as the hardware accelerator).}

\Subsection{Architecture of the Hardware Accelerators}

In this section, architectures of the hardware accelerators are
briefly described. The description would help Goose users to develop
efficient program targeting a given accelerator.

\Subsubsection{GRAPE-DR}\label{sec:gdr}

GRAPE-DR is a hardware accelerator developed by K\&F Computing
Research. Co.  It houses GRAPE-DR processor chip, a custom LSI
developed by the University of Tokyo and National Astronomical
Observatory of Japan. A GRAPE-DR model450 board houses one chip per
board, while a model1800 houses four.

In a single GRAPE-DR chip, 512 vector-processor elements (hereafter
PE) are integrated. Each PE has a multiplier, an adder, register
files, and a local memory. It processes an instruction stream in SIMD
fashion.  The maximum loop length of the vector processor is 4.

Each 32 PEs are grouped into 16 broadcast blocks. Each broadcast block
has a broadcast memory, whose contents can be broadcasted to all 32
PEs in the same block.

Outputs of the 16 broadcast blocks are connected each other, via
result-reduction tree network. Each block outputs 128 ($= 32$ PEs
$\times$ loop length 4) calculation results. Results from all 16
blocks, 2048 (128 $\times$ 16) in total, are reduced into 128 by
adders (or logical operators), and written out to outside of the chip.

%
\Figure{140}{gdrarch.eps}
{Figure : A block diagram of the GRAPE-DR chip.}

\noindent
{\bf A Nested 'for' Statement (equation (\ref{eq:sumj})) on GRAPE-DR:}\\

In the following, we use a nested 'for' statement below (the same
one used in section \ref{sec:tut2}) to describe a calculation
procedure on GRAPE-DR.\\

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
#pragma goose parallel for icnt(i) jcnt(j) \
  ip(x[i][0..2]) jp(x[j][0..2],m[j]) result(a[i][0..2],pot[i])\
  precision ("double") 
    for(i=0;i<n;i++) {
      for(k=0;k<3;k++) a[i][k] = 0.0;
      pot[i] = 0.0;
      for(j=0;j<n;j++) {
        for(k=0;k<3;k++) dx[k] = x[j][k] - x[i][k];
        r2 = dx[0]*dx[0] + dx[1]*dx[1] + dx[2]*dx[2] + eps2;
        rinv = rsqrt(r2);
        mrinv = m[j]*rinv;
        mr3inv = mrinv*rinv*rinv;
        for(k=0;k<3;k++) a[i][k] += mr3inv * dx[k];
        pot[i] += -mrinv;      
      }
    }
    get_cputime(&lt,&st);
    total_time += lt;
    for(i=0;i<n;i++) pot[i] += m[i]/sqrt(eps2);
\end{verbatim}
\end{minipage}

\vspace*{3mm}
\noindent
\begin{enumerate}

\item[(step 1)] The host computer writes {\tt n} variables of I/O type
    {\tt jp}, namely {\tt x[j][0..2]} and {\tt m[j]}, into the
    on-board memory of GRAPE-DR.

\item[(step 2)] The host computer writes variables of I/O type {\tt
      ip}, {\it i.e.}, {\tt x[i][0..2]}, into the local memory of each
    PE. Here, among {\tt n} sets of {\tt x[i][0..2]} for different
    {\tt i}-s, only 128 sets are written.  They are broadcasted to all
    16 broadcast blocks.  For each block, the 128 sets are split into
    32 pieces, each contains 4 sets ({\it i.e.}, loop-length of the
    vector processor), and each 4 sets are written to one of the 32
    PEs (PE0 $\sim$ PE31).  Note that the same 128 sets are written to
    all broadcast blocks.

\item[(step 3)] Before starting the calculation, 16 {\tt x[j][0..2]}
    and {\tt m[j]} are read out from the on-board memory and each of
    them is stored into each broadcast memory. Different {\tt x[j][0..2]}
    and {\tt m[j]} are written to different blocks.

\item[(step 4)] When the calculation starts, each broadcast memory
    broadcasts {\tt x[j][0..2]} and {\tt m[j]} to its 32 PEs.  each PE
    uses common {\tt x[j][0..2]} and {\tt m[j]} and its own {\tt
      x[i][0..2]} to execute the calculation. Different PEs in the
    same block perform calculations of gravitational forces from the
    same {\tt x[j][0..2], m[j]} to different {\tt x[i][0..2]}.
    Meanwhile, another 32 PEs in a different broadcast block perform
    calculations from different {\tt x[j][0..2], m[j]} to the same
    {\tt x[i][0..2]}.

\item[(step 5)] Each of the 16 broadcast blocks outputs calculation
    results {\tt a[i][0..2]} and {\tt pot[i]} for 128 different {\tt
      i}-s.  Results from 16 blocks, 128 $\times$ 16 in total, are
    summed up via the result-reduction tree network and reduced to 128
    results.

\item[(step 6)] Apply (step 2) to (step 5) for another 128 sets of
    {\tt x[i][0..2]}, until {\tt a[i][0..2]} and {\tt pot[i]} for all
    {\tt i}-s are obtained.

\end{enumerate}

\noindent
This procedure utilize all 512 PEs in the chip, if the input has 128
or more {\tt i}-s and 16 or more {\tt j}-s. Otherwise some of the PEs
are not utilized. They remain idle during the calculation.

\vspace*{5mm}
\noindent
{\bf A Single 'for' Statement (equation (\ref{eq:simd})) on GRAPE-DR:}\\


In the following, we use a single 'for' statement below (the same
one used in section \ref{sec:tut1}) to describe a calculation
procedure on GRAPE-DR.\\

\hspace*{5mm}
\begin{minipage}{150mm}
\baselineskip=14pt \lineskiplimit=-\maxdimen
\begin{verbatim}
#pragma goose parallel for icnt(i) result(sum) ip(i) shared_ro(dx)
  for(i=0;i<n;i++) {
    x = (i+0.5)*dx;
    sum += integrand(x)*dx;        
  }
\end{verbatim}
\end{minipage}

\vspace*{3mm}
\noindent
\begin{enumerate}

\item[(step 1)] The host computer writes variables {\tt i} of I/O type
    {\tt ip} and {\tt dx} of type {\tt shared\_ro} into the local
    memory of each PE. Among {\tt n} sets of {\tt i} and {\tt dx},
    only 128 sets are written. They are broadcasted to all 16
    broadcast blocks. For each block, the 128 sets are split into 32
    pieces, each consists of 4 sets ({\it i.e.}, loop-length of the
    vector processor), and each 4 sets are written to one of the 32
    PEs (PE0 $\sim$ PE31).  Note that the same 128 sets are written to
    all broadcast blocks.

\item[(step 2)] When the calculation starts, Each PE uses {\tt i} and
    {\tt dx} to execute the calculation. All 16 broadcast block
    perform the same calculation.

\item[(step 3)] Each of the 16 broadcast blocks outputs calculation
    results for 128 different {\tt i}-s.  Results from 16 blocks, 128
    $\times$ 16 in total, are summed up via the result-reduction tree
    network to 128. On the host computer, the results are divided by
    16 and the final results are obtained.

\item[(step 4)] Apply (step 1) to (step 3) for another 128 sets of
    {\tt i}-s, until results for all {\tt i}-s are obtained.

\end{enumerate}

\vspace*{5mm}
\noindent
This procedure has at least two rooms for improvement:

\begin{itemize}

\item All 16 broadcast blocks are performing the same calculation.
    This behavior can be modified so that different block perform
    calculation for different {\tt i}-s. With this modification,
    calculations for 128 $\times$ 16 different {\tt i}-s would be
    parallelized.

\item A value of a {\tt shared\_ro}-type variable does not depend on
    {\tt i}. Therefore, the host computer need to send it to the
    accelerator only once per calculation. But actually, it is resent
    at the beginning of each outer 'for' statement. This useless data
    transfer can be removed to reduce the communication time.

\end{itemize}

\noindent
These two improvements are going to be implemented soon.  Even after
the improvements, however, calculation described in the form of
equation (\ref{eq:simd}) has a potential disadvantage to that of
equation (\ref{eq:sumj}), due to the amount of data transfer ({\it
  cf.}  section \ref{sec:target}).

\clearpage

\Section{License}\label{sec:license}

Permission for use of the Goose Software Package (hereafter the
"Software") is granted only to owners of a copy of the Software.  The
Software may not be redistributed. The Software may be modified by the
owner, as long as the modified ones subject to this license agreement.

The copyright of the Software belongs to K\&F Computing Research. Co.
Programs and libraries which the Software relies on have their own
licenses, copyrights, and restrictions.

\Section{Acknowledgement}\label{sec:acknowledge}

K\&F Computing Research Co. would like to thank the following people for
help in development of the Goose Software Package:\\
%
Naohito Nakasato (University of Aizu) gave us a permission to use
LSUMP as a compiler for the Goose intermediate-representation.
Also, he kindly and quickly responded to a lot of our request for
modification and extension to LSUMP.
%
Junichiro Makino (National Astronomical Observatory of Japan) gave 
us a permission to use VSM, an assembler for GRAPE-DR.
%
Satoshi Katsuyama designed the logo of Goose.
%
We used {\tt racc}, a fast and powerful parser generator developed by
Mineroh Aoki, for development of {\tt goosec2q}, the C-language front
end of Goose ({\tt http://i.loveruby.net/ja/projects/racc/}).

\Section{Modification History}\label{sec:history}
\vspace*{5mm}

\begin{tabular}{llll}
\hline
version & date & description & author(s)\\
\hline
1.0.1$\alpha$  & 28-Sep-2009 & Document translated from the Japanese version.   & A. Kawai\\
\hline
\end{tabular}\\[5mm]

\noindent
Contact address for questions and bug reports:\\
K\&F Computing Research Co. ({\tt support@kfcr.jp})

\end{document}
