mirror of https://gitlab.com/QEF/q-e.git
1582 lines
70 KiB
TeX
1582 lines
70 KiB
TeX
\documentclass[12pt,a4paper]{article}
|
|
\def\version{5.0}
|
|
\def\qe{{\sc Quantum ESPRESSO}}
|
|
|
|
\usepackage{html}
|
|
|
|
% BEWARE: don't revert from graphicx for epsfig, because latex2html
|
|
% doesn't handle epsfig commands !!!
|
|
\usepackage{graphicx}
|
|
|
|
\textwidth = 17cm
|
|
\textheight = 24cm
|
|
\topmargin =-1 cm
|
|
\oddsidemargin = 0 cm
|
|
|
|
\def\pwx{\texttt{pw.x}}
|
|
\def\cpx{\texttt{cp.x}}
|
|
\def\phx{\texttt{ph.x}}
|
|
\def\nebx{\texttt{neb.x}}
|
|
\def\configure{\texttt{configure}}
|
|
\def\PWscf{\texttt{PWscf}}
|
|
\def\PHonon{\texttt{PHonon}}
|
|
\def\CP{\texttt{CP}}
|
|
\def\PostProc{\texttt{PostProc}}
|
|
\def\NEB{\texttt{PWneb}} % to be decided
|
|
\def\make{\texttt{make}}
|
|
|
|
\begin{document}
|
|
\author{}
|
|
\date{}
|
|
|
|
\def\qeImage{quantum_espresso.pdf}
|
|
\def\democritosImage{democritos.pdf}
|
|
|
|
\begin{htmlonly}
|
|
\def\qeImage{quantum_espresso.png}
|
|
\def\democritosImage{democritos.png}
|
|
\end{htmlonly}
|
|
|
|
\title{
|
|
\includegraphics[width=5cm]{\qeImage} \hskip 2cm
|
|
\includegraphics[width=6cm]{\democritosImage}\\
|
|
\vskip 1cm
|
|
% title
|
|
\Huge User's Guide for \qe\ \smallskip
|
|
\Large (version \version)
|
|
}
|
|
%\endhtmlonly
|
|
|
|
%\latexonly
|
|
%\title{
|
|
% \epsfig{figure=quantum_espresso.png,width=5cm}\hskip 2cm
|
|
% \epsfig{figure=democritos.png,width=6cm}\vskip 1cm
|
|
% % title
|
|
% \Huge User's Guide for \qe \smallskip
|
|
% \Large (version \version)
|
|
%}
|
|
%\endlatexonly
|
|
|
|
\maketitle
|
|
|
|
\tableofcontents
|
|
|
|
\section{Introduction}
|
|
|
|
This guide gives a general overview of the contents and of the installation
|
|
of \qe\ (opEn-Source Package for Research in Electronic Structure, Simulation,
|
|
and Optimization), version \version.
|
|
|
|
The \qe\ distribution contains the core packages \PWscf\ (Plane-Wave
|
|
Self-Consistent Field) and \CP\ (Car-Parrinello) for the calculation
|
|
of electronic-structure properties within
|
|
Density-Functional Theory (DFT), using a Plane-Wave (PW) basis set
|
|
and pseudopotentials. It also includes other packages for
|
|
more specialized calculations:
|
|
\begin{itemize}
|
|
\item \NEB:
|
|
energy barriers and reaction pathways through the Nudged Elastic Band
|
|
(NEB) method.
|
|
\item \PHonon:
|
|
vibrational properties with Density-Functional Perturbation Theory.
|
|
\item \PostProc:
|
|
codes and utilities for data postprocessing.
|
|
\item \texttt{PWcond}:
|
|
ballistic conductance.
|
|
\item \texttt{XSPECTRA}:
|
|
K-edge X-ray adsorption spectra.
|
|
\item \texttt{GWW}:
|
|
GW calculations using Wannier functions
|
|
\item \texttt{TD-DFPT}:
|
|
spectra from Time-Dependent
|
|
Density-Functional Perturbation Theory.
|
|
\end{itemize}
|
|
The following auxiliary packages are included as well:
|
|
\begin{itemize}
|
|
\item \texttt{PWgui}:
|
|
a Graphical User Interface, producing input data files for
|
|
\PWscf\ and some \PostProc\ codes.
|
|
\item \texttt{atomic}:
|
|
atomic calculations and pseudopotential generation.
|
|
\item \texttt{QHA}:
|
|
utilities for the calculation of projected density of states (PDOS)
|
|
and of the free energy in the Quasi-Harmonic Approximation (to be
|
|
used in conjunction with \PHonon).
|
|
\item \texttt{PlotPhon}:
|
|
phonon dispersion plotting utility (to be
|
|
used in conjunction with \PHonon).
|
|
\end{itemize}
|
|
A copy of required external libraries is also included.
|
|
Finally, several additional packages that exploit data produced by \qe\
|
|
or patch some \qe\ routines can be installed as {\em plug-ins}:
|
|
\begin{itemize}
|
|
\item \texttt{Wannier90}:
|
|
maximally localized Wannier functions.
|
|
\item \texttt{WanT}:
|
|
quantum transport properties with Wannier functions.
|
|
\item \texttt{YAMBO}:
|
|
electronic excitations within Many-Body Perturbation Theory:
|
|
GW and Bethe-Salpeter equation.
|
|
\item \texttt{PLUMED}:
|
|
calculation of free-energy surface through metadynamics.
|
|
\item \texttt{GIPAW} (Gauge-Independent Projector Augmented Waves):
|
|
NMR chemical shifts and EPR g-tensor.
|
|
\end{itemize}
|
|
Documentation on single packages can be found in the \texttt{Doc/} or
|
|
\texttt{doc/} directory of each package. A detailed description of input
|
|
data is available for most packages in files \texttt{INPUT\_*.txt} and
|
|
\texttt{INPUT\_*.html}.
|
|
|
|
The \qe\ codes work on many different types of Unix machines,
|
|
including parallel machines using both OpenMP and MPI
|
|
(Message Passing Interface).
|
|
Running \qe\ on Mac OS X and MS-Windows is also possible:
|
|
see section \ref{Sec:Installation}.
|
|
|
|
Further documentation, beyond what is provided in this guide, can be found in:
|
|
\begin{itemize}
|
|
\item the \texttt{Doc/} directory of the \qe\ distribution;
|
|
\item the \qe\ web site \texttt{http://www.quantum-espresso.org};
|
|
\item the archives of the mailing list:
|
|
See section \ref{SubSec:Contacts}, ``Contacts'', for more info.
|
|
\end{itemize}
|
|
People who want to contribute to \qe\ should read the
|
|
Developer Manual: \texttt{Doc/developer\_man.pdf}.
|
|
|
|
This guide does not explain the basic Unix concepts (shell, execution
|
|
path, directories etc.) and utilities needed to run \qe; it does not
|
|
explain either solid state physics and its computational methods.
|
|
If you want to learn the latter, you should first read a good textbook,
|
|
such as e.g. the book by Richard Martin:
|
|
{\em Electronic Structure: Basic Theory and Practical Methods},
|
|
Cambridge University Press (2004); or:
|
|
{\em Density functional theory: a practical introduction},
|
|
D. S. Sholl, J. A. Steckel (Wiley, 2009); or
|
|
{\em Electronic Structure Calculations for Solids and Molecules:
|
|
Theory and Computational Methods},
|
|
J. Kohanoff (Cambridge University Press, 2006). Then you should consult
|
|
the documentation of the package you want to use for more specific references.
|
|
|
|
All trademarks mentioned in this guide belong to their respective owners.
|
|
|
|
\subsection{People}
|
|
|
|
The maintenance and further development of the \qe\ distribution
|
|
is promoted by the DEMOCRITOS National Simulation Center
|
|
of IOM-CNR under the coordination of
|
|
Paolo Giannozzi (Univ.Udine, Italy) and Layla Martin-Samos
|
|
(Univ.Nova Gorica) with the strong support
|
|
of the CINECA National Supercomputing Center in Bologna under
|
|
the responsibility of Carlo Cavazzoni.
|
|
|
|
Main contributors to \qe, in addition to the authors of the paper
|
|
mentioned in Sect.\ref{SubSec:Terms}, are acknowledged in the
|
|
documentation of each package. An alphabetic list of further
|
|
contributors who answered questions on the mailing list, found
|
|
bugs, helped in porting to new architectures, wrote some code,
|
|
contributed in some way or another at some stage, follows:
|
|
\begin{quote}
|
|
Dario Alf\`e, Audrius Alkauskas, Alain Allouche, Francesco Antoniella,
|
|
Uli Aschauer, Francesca Baletto, Gerardo Ballabio, Mauro Boero,
|
|
Claudia Bungaro, Paolo Cazzato, Gabriele Cipriani, Jiayu Dai,
|
|
Cesar Da Silva, Alberto Debernardi, Gernot Deinzer, Yves Ferro,
|
|
Martin Hilgeman, Yosuke Kanai, Axel Kohlmeyer, Konstantin Kudin,
|
|
Nicolas Lacorne, Stephane Lefranc, Sergey Lisenkov, Kurt Maeder,
|
|
Andrea Marini, Giuseppe Mattioli, Nicolas Mounet, William Parker,
|
|
Pasquale Pavone, Mickael Profeta, Guido Roma, Kurt Stokbro,
|
|
Sylvie Stucki, Paul Tangney, Pascal Thibaudeau, Antonio Tilocca,
|
|
Jaro Tobik, Malgorzata Wierzbowska, Vittorio Zecca,
|
|
Silviu Zilberman, Federico Zipoli,
|
|
\end{quote}
|
|
and let us apologize to everybody we have forgotten.
|
|
|
|
\subsection{Contacts}
|
|
\label{SubSec:Contacts}
|
|
|
|
The web site for \qe\ is \texttt{http://www.quantum-espresso.org/}.
|
|
Releases and patches can be downloaded from this
|
|
site or following the links contained in it. The main entry point for
|
|
developers is the QE-forge web site:
|
|
\texttt{http://qe-forge.org/}.
|
|
|
|
The recommended place where to ask questions about installation
|
|
and usage of \qe, and to report problems, is the \texttt{pw\_forum}
|
|
mailing list: \texttt{pw\_forum@pwscf.org}.
|
|
Here you can obtain help from the developers and from
|
|
knowledgeable users. Please read the guidelines for posting,
|
|
section \ref{SubSec:Guidelines}!
|
|
|
|
You have to be subscribed (see ``Contacts'' section of the web site)
|
|
in order to post to the \texttt{pw\_forum} list.
|
|
NOTA BENE: only messages that appear to come from the
|
|
registered user's e-mail address, in its {\em exact form}, will be
|
|
accepted. Messages "waiting for moderator approval" are
|
|
automatically deleted with no further processing (sorry, too
|
|
much spam). In case of trouble, carefully check that your return
|
|
e-mail is the correct one (i.e. the one you used to subscribe).
|
|
|
|
Since \texttt{pw\_forum} has a sizable traffic, an alternative
|
|
low-traffic mailing list, \texttt{pw\_users@pwscf.org}, is provided for
|
|
those interested only in \qe-related news, such as e.g. announcements
|
|
of new versions, tutorials, etc.. You can subscribe (but not post) to
|
|
this list from the web site, ``Contacts'' section.
|
|
|
|
If you need to contact the developers for {\em specific} questions
|
|
about coding, proposals, offers of help, etc., please send a message
|
|
to the developers' mailing list: \texttt{q-e-developers@qe-forge.org}.
|
|
Inappropriate questions will be ignored.
|
|
|
|
\subsection{Guidelines for posting to the mailing list}
|
|
\label{SubSec:Guidelines}
|
|
Life for subscribers of \texttt{pw\_forum} will be easier if everybody
|
|
complies with the following guidelines:
|
|
\begin{itemize}
|
|
\item Before posting, {\em please}: browse or search the archives --
|
|
links are available in the ``Contacts'' section of the web site.
|
|
Most questions are asked over and over again. Also: make an attempt
|
|
to search the
|
|
available documentation, notably the FAQs and the User Guide(s).
|
|
The answer to most questions is already there.
|
|
\item Reply to both the mailing list and the author or the post, using
|
|
``Reply to all'' (not ``Reply'': the Reply-To: field no longer points
|
|
to the mailing list).
|
|
\item Sign your post with your name and affiliation.
|
|
\item Choose a meaningful subject. Do not use "reply" to start a new
|
|
thread:
|
|
it will confuse the ordering of messages into threads that most mailers
|
|
can do. In particular, do not use "reply" to a Digest!!!
|
|
\item Be short: no need to send 128 copies of the same error message just
|
|
because you this is what came out of your 128-processor run. No need to
|
|
send the entire compilation log for a single error appearing at the end.
|
|
\item Avoid excessive or irrelevant quoting of previous messages. Your
|
|
message must be immediately visible and easily readable, not hidden
|
|
into a sea of quoted text.
|
|
\item Remember that even experts cannot guess where a problem lies in
|
|
the absence of sufficient information. One piece of information that
|
|
must {\em always} be provided is the version number of \qe.
|
|
\item Remember that the mailing list is a voluntary endeavor: nobody is
|
|
entitled to an answer, even less to an immediate answer.
|
|
\item Finally, please note that the mailing list is not a replacement
|
|
for your own work, nor is it a replacement for your thesis director's
|
|
work.
|
|
\end{itemize}
|
|
|
|
\subsection{Terms of use}
|
|
\label{SubSec:Terms}
|
|
|
|
\qe\ is free software, released under the
|
|
GNU General Public License. See
|
|
\texttt{http://www.gnu.org/licenses/old-licenses/gpl-2.0.txt},
|
|
or the file License in the distribution).
|
|
|
|
We shall greatly appreciate if scientific work done using this code will
|
|
contain an explicit acknowledgment and the following reference:
|
|
\begin{quote}
|
|
P. Giannozzi, S. Baroni, N. Bonini, M. Calandra, R. Car, C. Cavazzoni,
|
|
D. Ceresoli, G. L. Chiarotti, M. Cococcioni, I. Dabo, A. Dal Corso,
|
|
S. Fabris, G. Fratesi, S. de Gironcoli, R. Gebauer, U. Gerstmann,
|
|
C. Gougoussis, A. Kokalj, M. Lazzeri, L. Martin-Samos, N. Marzari,
|
|
F. Mauri, R. Mazzarello, S. Paolini, A. Pasquarello, L. Paulatto,
|
|
C. Sbraccia, S. Scandolo, G. Sclauzero, A. P. Seitsonen, A. Smogunov,
|
|
P. Umari, R. M. Wentzcovitch, J.Phys.:Condens.Matter 21, 395502 (2009),
|
|
http://arxiv.org/abs/0906.2569
|
|
\end{quote}
|
|
Note the form \qe\ for textual citations of the code.
|
|
Pseudopotentials should be cited as (for instance)
|
|
\begin{quote}
|
|
[ ] We used the pseudopotentials C.pbe-rrjkus.UPF
|
|
and O.pbe-vbc.UPF from\\
|
|
\texttt{http://www.quantum-espresso.org}.
|
|
\end{quote}
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Installation}
|
|
|
|
\subsection{Download}
|
|
|
|
Presently, \qe\ is distributed in source form; some precompiled
|
|
executables (binary files) are provided for \texttt{PWgui}.
|
|
Packages for the Debian Linux distribution are however
|
|
made available by \texttt{debichem} developers.
|
|
Stable releases of the \qe\ source package (current version
|
|
is \version) can be downloaded from the Download section
|
|
of \texttt{http://www.quantum-espresso.org}.
|
|
|
|
Uncompress and unpack the core distribution using the command:
|
|
\begin{verbatim}
|
|
tar zxvf espresso-X.Y.Z.tar.gz
|
|
\end{verbatim}
|
|
(a hyphen before "zxvf" is optional) where \texttt{X.Y.Z} stands for the
|
|
version number. If your version of \texttt{tar}
|
|
doesn't recognize the "z" flag:
|
|
\begin{verbatim}
|
|
gunzip -c espresso-X.Y.Z.tar.gz | tar xvf -
|
|
\end{verbatim}
|
|
A directory \texttt{espresso-X.Y.Z/} will be created. Given the size
|
|
of the complete distribution, you may need to download more packages
|
|
and to unpack them following the same procedure (they will unpack into
|
|
the same directory).
|
|
|
|
Plug-ins \texttt{YAMBO}, \texttt{PLUMED}, \texttt{W90}, \texttt{WanT} should be downloaded into
|
|
subdirectory \texttt{archive} but {\em not unpacked or uncompressed}:
|
|
command \texttt{make} will take care of this during installation.
|
|
\texttt{GIPAW} is instead automatically downloaded on demand.
|
|
|
|
Occasionally, patches for the current version, fixing some errors and bugs,
|
|
may be distributed as a "diff" file. In order to install a patch (for
|
|
instance):
|
|
\begin{verbatim}
|
|
cd espresso-X.Y.Z/
|
|
patch -p1 < /path/to/the/diff/file/patch-file.diff
|
|
\end{verbatim}
|
|
If more than one patch is present, they should be applied in the correct order.
|
|
|
|
% Daily snapshots of the development version can be downloaded from the
|
|
%developers' site \texttt{qe-forge.org}: follow the link ''Quantum ESPRESSO'',
|
|
%then ''SCM''.
|
|
The bravest
|
|
may access the development version via anonymous access to the
|
|
Subversion (SVN) repository : see the Developer Manual
|
|
(\texttt{Doc/developer\_man.pdf}), section ''Using SVN''.
|
|
Beware: the development version
|
|
is, well, under development: use at your own risk!
|
|
|
|
The \qe\ distribution contains several directories. Some of them are
|
|
common to all packages:
|
|
|
|
\begin{tabular}{ll}
|
|
\texttt{Modules/} & source files for modules that are common to all programs\\
|
|
\texttt{include/} & files *.h included by fortran and C source files\\
|
|
\texttt{clib/} & external libraries written in C\\
|
|
\texttt{flib/} & external libraries written in Fortran\\
|
|
\texttt{install/} & installation scripts and utilities\\
|
|
\texttt{pseudo}/ & pseudopotential files used by examples\\
|
|
\texttt{upftools/}& converters to unified pseudopotential format (UPF)\\
|
|
\texttt{Doc/} & general documentation\\
|
|
\texttt{archive/} & contains plug-ins in .tar.gz form\\
|
|
\end{tabular}
|
|
\\
|
|
while others are specific to a single package:
|
|
|
|
\begin{tabular}{ll}
|
|
\texttt{PW/} & \PWscf\ package\\
|
|
\texttt{NEB/} & \NEB\ package\\
|
|
\texttt{PP/} & \PostProc\ package\\
|
|
\texttt{PHonon/} & \PHonon\ package\\
|
|
\texttt{PWCOND/} & \texttt{PWcond}\ package\\
|
|
\texttt{CPV/} & \CP\ package\\
|
|
\texttt{atomic/} & \texttt{atomic} package\\
|
|
\texttt{GUI/} & \texttt{PWGui} package\
|
|
\end{tabular}
|
|
|
|
\subsection{Prerequisites}
|
|
\label{Sec:Installation}
|
|
|
|
To install \qe\ from source, you need first of all a minimal Unix
|
|
environment: basically, a command shell (e.g.,
|
|
bash or tcsh) and the utilities \make, \texttt{awk}, \texttt{sed}.
|
|
MS-Windows users need to have Cygwin (a UNIX environment which
|
|
runs under Windows) installed:
|
|
see \texttt{http://www.cygwin.com/}. Note that the scripts contained
|
|
in the distribution assume that the local language is set to the
|
|
standard, i.e. "C"; other settings
|
|
may break them. Use \texttt{export LC\_ALL=C} (sh/bash) or
|
|
\texttt{setenv LC\_ALL C} (csh/tcsh) to prevent any problem
|
|
when running scripts (including installation scripts).
|
|
|
|
Second, you need C and Fortran-95 compilers. For parallel
|
|
execution, you will also need MPI libraries and a parallel
|
|
(i.e. MPI-aware) compiler. For massively parallel machines, or
|
|
for simple multicore parallelization, an OpenMP-aware compiler
|
|
and libraries are also required.
|
|
|
|
Big machines with
|
|
specialized hardware (e.g. IBM SP, CRAY, etc) typically have a
|
|
Fortran-95 compiler with MPI and OpenMP libraries bundled with
|
|
the software. Workstations or ``commodity'' machines, using PC
|
|
hardware, may or may not have the needed software. If not, you need
|
|
either to buy a commercial product (e.g Portland) or to install
|
|
an open-source compiler like gfortran or g95.
|
|
Note that several commercial compilers are available free of charge
|
|
under some license for academic or personal usage (e.g. Intel, Sun).
|
|
|
|
\subsection{\configure}
|
|
|
|
To install the \qe\ source package, run the \configure\
|
|
script. This is actually a wrapper to the true \configure,
|
|
located in the \texttt{install/} subdirectory. \configure\
|
|
will (try to) detect compilers and libraries available on
|
|
your machine, and set up things accordingly. Presently it is expected
|
|
to work on most Linux 32- and 64-bit PCs (all Intel and AMD CPUs) and
|
|
PC clusters, SGI Altix, IBM SP and BlueGene machines, NEC SX, Cray XT
|
|
machines, Mac OS X, MS-Windows PCs. A GPU-enabled version is currently
|
|
available as a separate branch of the SVN repository. See the following
|
|
message and instructions:\\
|
|
\texttt{http://www.democritos.it/pipermail/pw\_forum/2011-May/020399.html}.
|
|
|
|
Instructions for the impatient:
|
|
\begin{verbatim}
|
|
cd espresso-X.Y.Z/
|
|
./configure
|
|
make all
|
|
\end{verbatim}
|
|
Symlinks to executable programs will be placed in the
|
|
\texttt{bin/}
|
|
subdirectory. Note that both C and Fortran compilers must be in your execution
|
|
path, as specified in the PATH environment variable.
|
|
Additional instructions for special machines:
|
|
|
|
\begin{tabular}{ll}
|
|
\texttt{./configure ARCH=crayxt4}& for CRAY XT machines \\
|
|
\texttt{./configure ARCH=necsx} & for NEC SX machines \\
|
|
\texttt{./configure ARCH=ppc64-mn}& PowerPC Linux + xlf (Marenostrum) \\
|
|
\texttt{./configure ARCH=ppc64-bg}& IBM BG/P (BlueGene)
|
|
\end{tabular}
|
|
|
|
\noindent \configure\ generates the following files:
|
|
|
|
\begin{tabular}{ll}
|
|
\texttt{make.sys} & compilation rules and flags (used by \texttt{Makefile})\\
|
|
\texttt{install/configure.msg} & a report of the configuration run (not needed for compilation)\\
|
|
\texttt{install/config.log} & detailed log of the configuration run (may be needed for debugging)\\
|
|
\texttt{include/fft\_defs.h} & defines fortran variable for C pointer (used only by FFTW)\\
|
|
\texttt{include/c\_defs.h} & defines C to fortran calling convention\\
|
|
& and a few more definitions used by C files\\
|
|
\end{tabular}\\
|
|
NOTA BENE: unlike previous versions, \configure\ no longer runs the
|
|
\texttt{makedeps.sh} shell script that updates dependencies. If you modify the
|
|
sources, run \texttt{./install/makedeps.sh} or type \texttt{make depend}
|
|
to update files \texttt{make.depend} in the various subdirectories.
|
|
|
|
You should always be able to compile the \qe\ suite
|
|
of programs without having to edit any of the generated files. However you
|
|
may have to tune \configure\ by specifying appropriate environment variables
|
|
and/or command-line options. Usually the tricky part is to get external
|
|
libraries recognized and used: see Sec.\ref{Sec:Libraries}
|
|
for details and hints.
|
|
|
|
Environment variables may be set in any of these ways:
|
|
\begin{verbatim}
|
|
export VARIABLE=value; ./configure # sh, bash, ksh
|
|
setenv VARIABLE value; ./configure # csh, tcsh
|
|
./configure VARIABLE=value # any shell
|
|
\end{verbatim}
|
|
Some environment variables that are relevant to \configure\ are:
|
|
|
|
\begin{tabular}{ll}
|
|
\texttt{ARCH}& label identifying the machine type (see below)\\
|
|
\texttt{F90, F77, CC} &names of Fortran 95, Fortran 77, and C compilers\\
|
|
\texttt{MPIF90} & name of parallel Fortran 95 compiler (using MPI)\\
|
|
\texttt{CPP} & source file preprocessor (defaults to \$CC -E)\\
|
|
\texttt{LD} & linker (defaults to \$MPIF90)\\
|
|
\texttt{(C,F,F90,CPP,LD)FLAGS}& compilation/preprocessor/loader flags\\
|
|
\texttt{LIBDIRS}& extra directories where to search for libraries\\
|
|
\end{tabular}\\
|
|
For example, the following command line:
|
|
\begin{verbatim}
|
|
./configure MPIF90=mpf90 FFLAGS="-O2 -assume byterecl" \
|
|
CC=gcc CFLAGS=-O3 LDFLAGS=-static
|
|
\end{verbatim}
|
|
instructs \configure\ to use \texttt{mpf90} as Fortran 95 compiler
|
|
with flags \texttt{-O2 -assume byterecl}, \texttt{gcc} as C compiler with
|
|
flags \texttt{-O3}, and to link with flag \texttt{-static}.
|
|
Note that the value of \texttt{FFLAGS} must be quoted, because it contains
|
|
spaces. NOTA BENE: do not pass compiler names with the leading path
|
|
included. \texttt{F90=f90xyz} is ok, \texttt{F90=/path/to/f90xyz} is not.
|
|
Do not use
|
|
environmental variables with \configure\ unless they are needed! try
|
|
\configure\ with no options as a first step.
|
|
|
|
If your machine type is unknown to \configure, you may use the
|
|
\texttt{ARCH}
|
|
variable to suggest an architecture among supported ones. Some large
|
|
parallel machines using a front-end (e.g. Cray XT) will actually
|
|
need it, or else \configure\ will correctly recognize the front-end
|
|
but not the specialized compilation environment of those
|
|
machines. In some cases, cross-compilation requires to specify the target machine with the
|
|
\texttt{--host} option. This feature has not been extensively
|
|
tested, but we had at least one successful report (compilation
|
|
for NEC SX6 on a PC). Currently supported architectures are:
|
|
|
|
\begin{tabular}{ll}
|
|
\texttt{ia32}& Intel 32-bit machines (x86) running Linux\\
|
|
\texttt{ia64}& Intel 64-bit (Itanium) running Linux\\
|
|
\texttt{x86\_64}& Intel and AMD 64-bit running Linux - see note below\\
|
|
\texttt{aix}& IBM AIX machines\\
|
|
\texttt{solaris}& PC's running SUN-Solaris\\
|
|
\texttt{sparc}& Sun SPARC machines\\
|
|
\texttt{crayxt4}& Cray XT4/5 machines\\
|
|
\texttt{mac686}& Apple Intel machines running Mac OS X\\
|
|
\texttt{cygwin}& MS-Windows PCs with Cygwin\\
|
|
\texttt{necsx}& NEC SX-6 and SX-8 machines\\
|
|
\texttt{ppc64}& Linux PowerPC machines, 64 bits\\
|
|
\texttt{ppc64-mn}&as above, with IBM xlf compiler\\
|
|
\texttt{ppc64-bg}&IBM BlueGene
|
|
\end{tabular}\\
|
|
{\em Note}: \texttt{x86\_64} replaces \texttt{amd64} since v.4.1.
|
|
Cray Unicos machines, SGI
|
|
machines with MIPS architecture, HP-Compaq Alphas are no longer supported
|
|
since v.4.2; PowerPC Macs are no longer
|
|
supported since v.5.0.
|
|
Finally, \configure\ recognizes the following command-line options:\\
|
|
\begin{tabular}{ll}
|
|
\texttt{--enable-parallel}& compile for parallel (MPI) execution if possible (default: yes)\\
|
|
\texttt{--enable-openmp}& compile for OpenMP execution if possible (default: no)\\
|
|
\texttt{--enable-shared}& use shared libraries if available (default: yes;\\
|
|
& "no" is implemented, untested, in only a few cases)\\
|
|
\texttt{--enable-debug}& compile with debug flags (only for selected cases; default: no)\\
|
|
\texttt{--disable-wrappers}& disable C to fortran wrapper check (default: enabled)\\
|
|
\texttt{--enable-signals}& enable signal trapping (default: disabled)\\
|
|
\end{tabular}\\
|
|
and the following optional packages:\\
|
|
\begin{tabular}{ll}
|
|
\texttt{--with-internal-blas}& compile with internal BLAS (default: no)\\
|
|
\texttt{--with-internal-lapack}& compile with internal LAPACK (default: no)\\
|
|
\texttt{--with-scalapack}& use ScaLAPACK if available (default: yes)\\
|
|
\end{tabular}\\
|
|
If you want to modify the \configure\ script (advanced users only!),
|
|
see the Developer Manual.
|
|
|
|
\subsubsection{Manual configuration}
|
|
\label{SubSec:manconf}
|
|
If \configure\ stops before the end, and you don't find a way to fix
|
|
it, you have to write working \texttt{make.sys}, \texttt{include/fft\_defs.h}
|
|
and \texttt{include/c\_defs.h} files.
|
|
For the latter two files, follow the explanations in
|
|
\texttt{include/defs.h.README}.
|
|
|
|
If \configure\ has run till the end, you should need only to
|
|
edit \texttt{make.sys}. A few sample \texttt{make.sys} files
|
|
are provided in \texttt{install/Make.}{\em system}. The template used
|
|
by \configure\ is also found there as \texttt{install/make.sys.in}
|
|
and contains explanations of the meaning
|
|
of the various variables. Note that you may need
|
|
to select appropriate preprocessing flags
|
|
in conjunction with the desired or available
|
|
libraries (e.g. you need to add \texttt{-D\_\_FFTW} to \texttt{DFLAGS}
|
|
if you want to link internal FFTW). For a correct choice of preprocessing
|
|
flags, refer to the documentation in \texttt{include/defs.h.README}.
|
|
|
|
NOTA BENE: If you change any settings (e.g. preprocessing,
|
|
compilation flags)
|
|
after a previous (successful or failed) compilation, you must run
|
|
\texttt{make clean} before recompiling, unless you know exactly which
|
|
routines are affected by the changed settings and how to force their recompilation.
|
|
|
|
\subsection{Libraries}
|
|
\label{Sec:Libraries}
|
|
|
|
\qe\ makes use of the following external libraries:
|
|
\begin{itemize}
|
|
\item BLAS (\texttt{http://www.netlib.org/blas/}) and
|
|
\item LAPACK (\texttt{http://www.netlib.org/lapack/}) for linear algebra
|
|
\item FFTW (\texttt{http://www.fftw.org/}) for Fast Fourier Transforms
|
|
\end{itemize}
|
|
A copy of the needed routines is provided with the distribution. However,
|
|
when available, optimized vendor-specific libraries should be used: this
|
|
often yields huge performance gains.
|
|
|
|
\paragraph{BLAS and LAPACK}
|
|
\qe\ can use the following architecture-specific replacements for BLAS and LAPACK:\\
|
|
\begin{quote}
|
|
MKL for Intel Linux PCs\\
|
|
ACML for AMD Linux PCs\\
|
|
ESSL for IBM machines\\
|
|
SCSL for SGI Altix\\
|
|
SUNperf for Sun
|
|
\end{quote}
|
|
If none of these is available, we suggest that you use the optimized ATLAS library: see \\
|
|
\texttt{http://math-atlas.sourceforge.net/}. Note that ATLAS is not
|
|
a complete replacement for LAPACK: it contains all of the BLAS, plus the
|
|
LU code, plus the full storage Cholesky code. Follow the instructions in the
|
|
ATLAS distributions to produce a full LAPACK replacement.
|
|
|
|
Sergei Lisenkov reported success and good performances with optimized
|
|
BLAS by Kazushige Goto. They can be freely downloaded,
|
|
but not redistributed. See the "GotoBLAS2" item at\\
|
|
\texttt{http://www.tacc.utexas.edu/tacc-projects/}.
|
|
|
|
\paragraph{FFT}
|
|
\qe\ has an internal copy of an old FFTW version, and it
|
|
can use the following vendor-specific FFT libraries:
|
|
\begin{quote}
|
|
IBM ESSL\\
|
|
SGI SCSL\\
|
|
SUN sunperf\\
|
|
NEC ASL
|
|
\end{quote}
|
|
\configure\ will first search for vendor-specific FFT libraries;
|
|
if none is found, it will search for an external FFTW v.3 library;
|
|
if none is found, it will fall back to the internal copy of FFTW.
|
|
|
|
If you have recent versions (v.10 or later) of MKL installed, you
|
|
may use the FFTW3 interface provided with MKL. This can be directly
|
|
linked in MKL distributed with v.12 of the Intel compiler. In earlier
|
|
versions, only sources are distributed: you have to compile them and
|
|
to modify file \texttt{make.sys} accordingly
|
|
(MKL must be linked {\em after} the FFTW-MKL interface).
|
|
|
|
\paragraph{MPI libraries}
|
|
MPI libraries are usually needed for parallel execution
|
|
(unless you are happy with OpenMP multicore parallelization).
|
|
In well-configured machines, \configure\ should find the appropriate
|
|
parallel compiler for you, and this should find the appropriate
|
|
libraries. Since often this doesn't
|
|
happen, especially on PC clusters, see Sec.\ref{SubSec:LinuxPCMPI}.
|
|
|
|
\paragraph{Other libraries}
|
|
\qe\ can use the MASS vector math
|
|
library from IBM, if available (only on AIX).
|
|
|
|
\paragraph{If optimized libraries are not found}
|
|
The \configure\ script attempts to find optimized libraries, but may fail
|
|
if they have been installed in non-standard places. You should examine
|
|
the final value of \texttt{BLAS\_LIBS, LAPACK\_LIBS, FFT\_LIBS, MPI\_LIBS} (if needed),
|
|
\texttt{MASS\_LIBS} (IBM only), either in the output of \configure\ or in the generated
|
|
\texttt{make.sys}, to check whether it found all the libraries that you intend to use.
|
|
|
|
If some library was not found, you can specify a list of directories to search
|
|
in the environment variable \texttt{LIBDIRS},
|
|
and rerun \configure; directories in the
|
|
list must be separated by spaces. For example:
|
|
\begin{verbatim}
|
|
./configure LIBDIRS="/opt/intel/mkl70/lib/32 /usr/lib/math"
|
|
\end{verbatim}
|
|
If this still fails, you may set some or all of the \texttt{*\_LIBS} variables manually
|
|
and retry. For example:
|
|
\begin{verbatim}
|
|
./configure BLAS_LIBS="-L/usr/lib/math -lf77blas -latlas_sse"
|
|
\end{verbatim}
|
|
Beware that in this case, \configure\ will blindly accept the specified value,
|
|
and won't do any extra search.
|
|
|
|
\subsection{Compilation}
|
|
|
|
There are a few adjustable parameters in \texttt{Modules/parameters.f90}.
|
|
The
|
|
present values will work for most cases. All other variables are dynamically
|
|
allocated: you do not need to recompile your code for a different system.
|
|
|
|
At your choice, you may compile the complete \qe\ suite of programs
|
|
(with \texttt{make all}), or only some specific programs. \texttt{make} with no arguments yields a list of valid compilation targets:
|
|
\begin{itemize}
|
|
\item \texttt{make pw} compiles the self-consistent-field package \PWscf
|
|
\item \texttt{make cp} compiles the Car-Parrinello package \CP
|
|
\item \texttt{make neb} compiles the \NEB\ package
|
|
\item \texttt{make ph} compiles the \PHonon\ package
|
|
\item \texttt{make pp} compiles the postprocessing package \PostProc
|
|
\item \texttt{make pwcond} compiles the ballistic conductance package
|
|
\texttt{PWcond}
|
|
\item \texttt{make pwall} produces all of the above.
|
|
\item \texttt{make ld1} compiles the pseudopotential package \texttt{atomic}
|
|
\item \texttt{make upf} produces utilities for pseudopotential conversion in
|
|
directory \texttt{upftools/}
|
|
\item \texttt{make all} produces all of the above
|
|
\item \texttt{make plumed} unpacks \texttt{PLUMED}, patches several routines
|
|
in \texttt{PW/}, \texttt{CPV/} and \texttt{clib/},
|
|
recompiles \PWscf\ and \CP\ with \texttt{PLUMED}
|
|
support
|
|
\item \texttt{make w90} unpacks \texttt{wannier90}, copies an appropriate
|
|
\texttt{make.sys} file, produces all executables
|
|
in \texttt{W90/wannier90.x} and in \texttt{bin/}
|
|
\item \texttt{make want} unpacks \texttt{WanT}, runs its \configure,
|
|
produces all executables for \texttt{WanT} in
|
|
\texttt{WANT/bin}.
|
|
\item \texttt{make yambo} unpacks \texttt{yambo}, runs its \configure,
|
|
produces all \texttt{yambo} executables in
|
|
\texttt{YAMBO/bin}
|
|
\item \texttt{make gipaw} downloads \texttt{GIPAW} from \texttt{qe-forge},
|
|
unpacks it, runs its \configure,
|
|
produces all \texttt{GIPAW} executables in
|
|
\texttt{GIPAW/bin} and in \texttt{bin}.
|
|
\end{itemize}
|
|
For the setup of the GUI, refer to the \texttt{PWgui-X.Y.Z /INSTALL} file, where
|
|
X.Y.Z stands for the version number of the GUI (should be the same as the
|
|
general version number). If you are using the SVN sources, see
|
|
the \texttt{GUI/README} file instead.
|
|
|
|
\subsection{Running tests and examples}
|
|
\label{SubSec:Examples}
|
|
|
|
% You should first of all ensure that you have downloaded
|
|
% and correctly unpacked the package containing examples (since v.4.1 in a
|
|
% separate package):
|
|
% \begin{verbatim}
|
|
% tar -zxvf /path/to/package/espresso-X.Y.Z-examples.tar.gz
|
|
% \end{verbatim}
|
|
% will unpack several subdirectories into \texttt{espresso-X.Y.Z/}.
|
|
|
|
As a final check that compilation was successful, you may want to run some or
|
|
all of the examples. There are two different types of examples:
|
|
\begin{itemize}
|
|
\item automated tests. Quick and exhaustive, but not
|
|
meant to be realistic, implemented only for \PWscf\ and \CP.
|
|
\item examples.
|
|
Cover many more programs and features of the \qe\ distribution,
|
|
but they require manual inspection of the results.
|
|
\end{itemize}
|
|
Instructions for the impatient:
|
|
\begin{verbatim}
|
|
cd PW/tests/
|
|
./check_pw.x.j
|
|
\end{verbatim}
|
|
for \PWscf;
|
|
\texttt{PW/tests/README} contains a list of what is tested.
|
|
For \CP:
|
|
\begin{verbatim}
|
|
cd CPV/tests/
|
|
./check_cp.x.j
|
|
\end{verbatim}
|
|
Instructions for all others: edit file \texttt{environment\_variables},
|
|
setting the following variables as needed.
|
|
\begin{quote}
|
|
BIN\_DIR: directory where executables reside\\
|
|
PSEUDO\_DIR: directory where pseudopotential files reside\\
|
|
TMP\_DIR: directory to be used as temporary storage area
|
|
\end{quote}
|
|
The default values of BIN\_DIR and PSEUDO\_DIR should be fine,
|
|
unless you have installed things in nonstandard places. TMP\_DIR
|
|
must be a directory where you have read and write access to, with
|
|
enough available space to host the temporary files produced by the
|
|
example runs, and possibly offering high I/O performance (i.e., don't
|
|
use an NFS-mounted directory). NOTA BENE: do not use a
|
|
directory containing other data: the examples will clean it!
|
|
|
|
If you have compiled the parallel version of \qe\ (this
|
|
is the default if parallel libraries are detected), you will usually
|
|
have to specify a launcher program (such as \texttt{mpirun} or
|
|
\texttt{mpiexec}) and the number of processors: see Sec.\ref{SubSec:para} for
|
|
details. In order to do that, edit again the \texttt{environment\_variables}
|
|
file
|
|
and set the PARA\_PREFIX and PARA\_POSTFIX variables as needed.
|
|
Parallel executables will be run by a command like this:
|
|
\begin{verbatim}
|
|
$PARA_PREFIX pw.x $PARA_POSTFIX -in file.in > file.out
|
|
\end{verbatim}
|
|
For example, if the command line is like this (as for an IBM SP):
|
|
\begin{verbatim}
|
|
poe pw.x -procs 4 -in file.in > file.out
|
|
\end{verbatim}
|
|
you should set PARA\_PREFIX="poe", PARA\_POSTFIX="-procs
|
|
4". Furthermore, if your machine does not support interactive use, you
|
|
must run the commands specified above through the batch queuing
|
|
system installed on that machine. Ask your system administrator for
|
|
instructions. For execution using OpenMP on N threads,
|
|
you should set PARA\_PREFIX to \texttt{"env OMP\_NUM\_THREADS=N ... "}.
|
|
|
|
To run an example, go to the corresponding directory (e.g.
|
|
\texttt{PW/examples/example01}) and execute:
|
|
\begin{verbatim}
|
|
./run_example
|
|
\end{verbatim}
|
|
This will create a subdirectory \texttt{results/}, containing the input and
|
|
output files generated by the calculation. Some examples take only a
|
|
few seconds to run, while others may require several minutes depending
|
|
on your system.
|
|
|
|
In each example's directory, the \texttt{reference/} subdirectory contains
|
|
verified output files, that you can check your results against. They
|
|
were generated on a Linux PC using the Intel compiler. On different
|
|
architectures the precise numbers could be slightly different, in
|
|
particular if different FFT dimensions are automatically selected. For
|
|
this reason, a plain diff of your results against the reference data
|
|
doesn't work, or at least, it requires human inspection of the results.
|
|
|
|
The example scripts stop if an error is detected. You should look {\em inside}
|
|
the last written output file to understand why.
|
|
|
|
\subsection{Installation tricks and problems}
|
|
|
|
\subsubsection{All architectures}
|
|
\begin{itemize}
|
|
\item
|
|
Working Fortran-95 and C compilers are needed in order
|
|
to compile \qe. Most ``Fortran-90'' compilers actually
|
|
implement the Fortran-95 standard, but older versions
|
|
may not be Fortran-95 compliant. Moreover,
|
|
C and Fortran compilers must be in your PATH.
|
|
If \configure\ says that you have no working compiler, well,
|
|
you have no working compiler, at least not in your PATH, and
|
|
not among those recognized by \configure.
|
|
\item
|
|
If you get {\em Compiler Internal Error} or similar messages: your
|
|
compiler version is buggy. Try to lower the optimization level, or to
|
|
remove optimization just for the routine that has problems. If it
|
|
doesn't work, or if you experience weird problems at run time, try to
|
|
install patches for your version of the compiler (most vendors release
|
|
at least a few patches for free), or to upgrade to a more recent
|
|
compiler version.
|
|
\item
|
|
If you get error messages at the loading phase that look like
|
|
{\em file XYZ.o: unknown / not recognized/ invalid / wrong
|
|
file type / file format / module version},
|
|
one of the following things have happened:
|
|
\begin{enumerate}
|
|
\item you have leftover object files from a compilation with another
|
|
compiler: run \texttt{make clean} and recompile.
|
|
\item \make\ did not stop at the first compilation error (it may
|
|
happen in some software configurations). Remove the file *.o
|
|
that triggers the error message, recompile, look for a
|
|
compilation error.
|
|
\end{enumerate}
|
|
If many symbols are missing in the loading phase: you did not specify the
|
|
location of all needed libraries (LAPACK, BLAS, FFTW, machine-specific
|
|
optimized libraries), in the needed order.
|
|
If only symbols from \texttt{clib/} are missing, verify that
|
|
you have the correct C-to-Fortran bindings, defined in
|
|
\texttt{include/c\_defs.h}.
|
|
Note that \qe\ is self-contained (with the exception of MPI libraries for
|
|
parallel compilation): if system libraries are missing, the problem is in
|
|
your compiler/library combination or in their usage, not in \qe.
|
|
\item
|
|
If you get mysterious errors in the provided tests and examples:
|
|
your compiler, or your mathematical libraries, or MPI libraries,
|
|
or a combination thereof, is very likely buggy. Although the
|
|
presence of subtle bugs in \qe\ that are not revealed during
|
|
the testing phase can never be ruled out, it is very unlikely
|
|
that this happens on the provided tests and examples.
|
|
\end{itemize}
|
|
|
|
\subsubsection{Cray XT machines}
|
|
|
|
Use \texttt{./configure ARCH=crayxt4} or else \configure\ will
|
|
not recognize the Cray-specific software environment. Older Cray
|
|
machines: T3D, T3E, X1, are no longer supported.
|
|
|
|
\subsubsection{IBM AIX}
|
|
|
|
v.4.3.1 of the CP code, Wannier-function dynamics, crashes with
|
|
``segmentation violation'' on some AIX v.6 machines.
|
|
Workaround: compile it with \texttt{mpxlf95} instead of
|
|
\texttt{mpxlf90}. (Info by Roberto Scipioni, June 2011)
|
|
|
|
On IBM machines with ESSL libraries installed, there is a
|
|
potential conflict between a few LAPACK routines that are also part of ESSL,
|
|
but with a different calling sequence. The appearance of run-time errors like {\em
|
|
ON ENTRY TO ZHPEV PARAMETER NUMBER 1 HAD AN ILLEGAL VALUE}
|
|
is a signal that you are calling the bad routine. If you have defined
|
|
\texttt{-D\_\_ESSL} you should load ESSL before LAPACK: see
|
|
variable LAPACK\_LIBS in make.sys.
|
|
|
|
\subsubsection{IBM BlueGene}
|
|
|
|
The current \configure\ is tested and works on the machines at CINECA
|
|
and at J\"ulich. For other sites, you may need something like
|
|
\begin{verbatim}
|
|
./configure ARCH=ppc64-bg BLAS_LIBS=... LAPACK_LIBS=... \
|
|
SCALAPACK_DIR=... BLACS_DIR=..."
|
|
\end{verbatim}
|
|
where the various *\_LIBS and *\_DIR "suggest" where the various libraries
|
|
are located.
|
|
|
|
\subsubsection{Linux PC}
|
|
|
|
Both AMD and Intel CPUs, 32-bit and 64-bit, are supported and work,
|
|
either in 32-bit emulation and in 64-bit mode. 64-bit executables
|
|
can address a much larger memory space than 32-bit executable, but
|
|
there is no gain in speed.
|
|
Beware: the default integer type for 64-bit machine is typically
|
|
32-bit long. You should be able to use 64-bit integers as well,
|
|
but it is not guaranteed to work and will not give
|
|
any advantage anyway.
|
|
|
|
Currently the following compilers are supported by \configure:
|
|
Intel (ifort), Portland (pgf90), gfortran, g95, Pathscale (pathf95),
|
|
Sun Studio (sunf95), AMD Open64 (openf95). The ordering approximately
|
|
reflects the quality of support. Both Intel MKL and AMD acml mathematical
|
|
libraries are supported. Some combinations of compilers and of libraries
|
|
may however require manual editing of \texttt{make.sys}.
|
|
|
|
It is usually convenient to create semi-statically linked executables (with only
|
|
libc, libm, libpthread dynamically linked). If you want to produce a binary
|
|
that runs on different machines, compile it on the oldest machine you have
|
|
(i.e. the one with the oldest version of the operating system).
|
|
|
|
If you get errors like {\em IPO Error: unresolved : \_\_svml\_cos2}
|
|
at the linking stage, your compiler is optimized to use the SSE
|
|
version of sine, cosine etc. contained in the SVML library. Append
|
|
\texttt{-lsvml} to the list of libraries in your \texttt{make.sys} file (info by Axel
|
|
Kohlmeyer, oct.2007).
|
|
|
|
\paragraph{Linux PCs with Portland compiler (pgf90)}
|
|
|
|
\qe\ does not work reliably, or not at all, with many old
|
|
versions ($< 6.1$) of the Portland Group compiler (pgf90).
|
|
Use the latest version of each
|
|
release of the compiler, with patches if available (see
|
|
the Portland Group web site, \texttt{http://www.pgroup.com/}).
|
|
|
|
\paragraph{Linux PCs with Pathscale compiler}
|
|
|
|
Version 2.99 of the Pathscale EKO compiler (web site
|
|
\texttt{http://www.pathscale.com/})
|
|
works and is recognized by
|
|
\configure, but the preprocessing command, \texttt{pathcc -E},
|
|
causes a mysterious error in compilation of iotk and should be replaced by
|
|
\begin{verbatim}
|
|
/lib/cpp -P --traditional
|
|
\end{verbatim}
|
|
The MVAPICH parallel environment with Pathscale compilers also works
|
|
(info by Paolo Giannozzi, July 2008).
|
|
|
|
Version 3.1 and version 4 (open source!) of the Pathscale EKO compiler
|
|
also work (info by Cezary Sliwa, April 2011, and Carlo Nervi, June 2011).
|
|
In case of mysterious errors while compiling \texttt{iotk},
|
|
remove all lines like:
|
|
\begin{verbatim}
|
|
# 1 "iotk_base.spp"
|
|
\end{verbatim}
|
|
from all \texttt{iotk} source files.
|
|
|
|
\paragraph{Linux PCs with gfortran}
|
|
|
|
Old gfortran versions often produce nonfunctional
|
|
phonon executables (segmentation faults and the like); other versions
|
|
miscompile iotk (the executables work but crash with a mysterious iotk
|
|
error when reading from data files). Recent versions should be fine.
|
|
|
|
If you experience problems in reading files produced by previous versions
|
|
of \qe: ``gfortran used 64-bit record markers to allow writing of records
|
|
larger than 2 GB. Before with 32-bit record markers only records $<$2GB
|
|
could be written. However, this caused problems with older files and
|
|
inter-compiler operability. This was solved in GCC 4.2 by using 32-bit
|
|
record markers but such that one can still store $>$2GB records (following
|
|
the implementation of Intel). Thus this issue should be gone. See 4.2
|
|
release notes (item ``Fortran") at
|
|
\texttt{http://gcc.gnu.org/gcc-4.2/changes.html}."
|
|
(Info by Tobias Burnus, March 2010).
|
|
|
|
``Using gfortran v.4.4 (after May 27, 2009) and 4.5 (after May 5, 2009) can
|
|
produce wrong results, unless the environment variable
|
|
GFORTRAN\_UNBUFFERED\_ALL=1 is set. Newer 4.4/4.5 versions
|
|
(later than April 2010) should be OK. See\\
|
|
\texttt{http://gcc.gnu.org/bugzilla/show\_bug.cgi?id=43551}."
|
|
(Info by Tobias Burnus, March 2010).
|
|
|
|
\paragraph{Linux PCs with g95}
|
|
|
|
g95 v.0.91 and later versions (\texttt{http://www.g95.org}) work.
|
|
The executables that produce are however slower (let us say 20\% or so)
|
|
that those produced by gfortran, which in turn are slower
|
|
(by another 20\% or so) than those produced by ifort.
|
|
|
|
\paragraph{Linux PCs with Sun Studio compiler}
|
|
|
|
``The Sun Studio compiler, sunf95, is free (web site:
|
|
\texttt{http://developers.sun.com/sunstudio/} and comes
|
|
with a set of algebra libraries that can be used in place of the slow
|
|
built-in libraries. It also supports OpenMP, which g95 does not. On the
|
|
other hand, it is a pain to compile MPI with it. Furthermore the most
|
|
recent version has a terrible bug that totally miscompiles the iotk
|
|
input/output library (you'll have to compile it with reduced optimization).''
|
|
(info by Lorenzo Paulatto, March 2010).
|
|
|
|
\paragraph{Linux PCs with AMD Open64 suite}
|
|
|
|
The AMD Open64 compiler suite, openf95 (web site:
|
|
\texttt{http://developer.amd.com/cpu/open64/pages/default.aspx})
|
|
can be freely downloaded from the AMD site.
|
|
It is recognized by \configure\ but little tested. It sort of works
|
|
but it fails to pass several tests (info by Paolo Giannozzi, March 2010).
|
|
"I have configured for Pathscale, then switched to the Open64 compiler by
|
|
editing make.sys. "make pw" succeeded and pw.x did process my file, but with
|
|
"make all" I get an internal compiler error [in CPV/wf.f90]" (info by Cezary
|
|
Sliwa, April 2011).
|
|
|
|
\paragraph{Linux PCs with Intel compiler (ifort)}
|
|
|
|
The Intel compiler, ifort, is available for free for personal
|
|
usage (\texttt{http://software.intel.com/}). It seem to produce the faster executables,
|
|
at least on Intel CPUs, but not all versions work as expected.
|
|
ifort versions $<9.1$ are not recommended, due to the presence of subtle
|
|
and insidious bugs. In case of trouble, update your version with
|
|
the most recent patches,
|
|
available via Intel Premier support (registration free of charge for Linux):
|
|
\texttt{http://software.intel.com/en-us/articles/intel-software-developer-support}.
|
|
Since each major release of ifort
|
|
differs a lot from the previous one, compiled objects from different
|
|
releases may be incompatible and should not be mixed.
|
|
|
|
If \configure\ doesn't find the compiler, or if you get
|
|
{\em Error loading shared libraries} at run time, you may have
|
|
forgotten to execute the script that
|
|
sets up the correct PATH and library path. Unless your system manager has
|
|
done this for you, you should execute the appropriate script -- located in
|
|
the directory containing the compiler executable -- in your
|
|
initialization files. Consult the documentation provided by Intel.
|
|
|
|
The warning: {\em feupdateenv is not implemented and will always fail},
|
|
showing up in recent versions, can be safely ignored. Warnings on
|
|
"bad preprocessing option" when compiling iotk
|
|
and complains about ``recommanded formats''
|
|
should also be ignored.
|
|
|
|
{\bf ifort v.12}: release 12.0.0 miscompiles iotk, leading to
|
|
mysterious errors when reading data files. Workaround: increase
|
|
the parameter BLOCKSIZE to e.g. 131072*1024 when opening files in
|
|
\texttt{iotk/src/iotk\_files.f90} (info by Lorenzo Paulatto,
|
|
Nov. 2010). Release 12.0.2 seems to work and to produce faster executables
|
|
than previous versions on 64-bit CPUs (info by P. Giannozzi, March 2011).
|
|
|
|
{\bf ifort v.11}: Segmentation faults were reported for the combination
|
|
ifort 11.0.081, MKL 10.1.1.019, OpenMP 1.3.3. The problem disappeared
|
|
with ifort 11.1.056 and MKL 10.2.2.025 (Carlo Nervi, Oct. 2009).
|
|
|
|
{\bf ifort v.10}: On 64-bit AMD CPUs, at least some versions of ifort 10.1
|
|
miscompile subroutine \texttt{write\_rho\_xml} in
|
|
\texttt{Module/xml\_io\_base.f90} with -O2
|
|
optimization. Using -O1 instead solves the problem (info by Carlo
|
|
Cavazzoni, March 2008).
|
|
|
|
"The intel compiler version 10.1.008 miscompiles a lot of codes (I have proof
|
|
for CP2K and CPMD) and needs to be updated in any case" (info by Axel
|
|
Kohlmeyer, May 2008).
|
|
|
|
{\bf ifort v.9}: The latest (July 2006) 32-bit version of ifort 9.1
|
|
works. Earlier versions yielded {\em Compiler Internal Error}.
|
|
|
|
\paragraph{Linux PCs with MKL libraries}
|
|
On Intel CPUs it is very convenient to use Intel MKL libraries. They can be
|
|
also used for AMD CPU, selecting the appropriate machine-optimized
|
|
libraries, and also together with non-Intel compilers. Note however
|
|
that recent versions of MKL (10.2 and following) do not perform
|
|
well on AMD machines.
|
|
|
|
\configure\ should recognize properly installed MKL libraries.
|
|
By default the non-threaded version of MKL is linked, unless option
|
|
\texttt{configure --with-openmp} is specified. In case of trouble,
|
|
refer to the following web page to find the correct way to link MKL:\\
|
|
\texttt{http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/}.
|
|
|
|
MKL contains optimized FFT routines and a FFTW interface, to be separately
|
|
compiled. For 64-bit Intel Core2 processors, they are slightly faster than
|
|
FFTW (MKL v.10, FFTW v.3 fortran interface, reported by P. Giannozzi,
|
|
November 2008).
|
|
|
|
For parallel (MPI) execution on multiprocessor (SMP) machines, set the
|
|
environmental variable OMP\_NUM\_THREADS to 1 unless you know what you
|
|
are doing. See Sec.\ref{Sec:para} for more info on this
|
|
and on the difference between MPI and OpenMP parallelization.
|
|
|
|
\paragraph{Linux PCs with ACML libraries}
|
|
For AMD CPUs, especially recent ones, you may find convenient to
|
|
link AMD acml libraries (can be freely downloaded from AMD web site).
|
|
\configure\ should recognize properly installed acml libraries,
|
|
together with the compilers most frequently used on AMD systems:
|
|
pgf90, pathscale, openf95, sunf95.
|
|
|
|
\subsubsection{Linux PC clusters with MPI}
|
|
\label{SubSec:LinuxPCMPI}
|
|
PC clusters running some version of MPI are a very popular
|
|
computational platform nowadays. \qe\ is known to work
|
|
with at least two of the major MPI implementations (MPICH, LAM-MPI),
|
|
plus with the newer MPICH2 and OpenMPI implementation.
|
|
\configure\ should automatically recognize a properly installed
|
|
parallel environment and prepare for parallel compilation.
|
|
Unfortunately this not always happens. In fact:
|
|
\begin{itemize}
|
|
\item \configure\ tries to locate a parallel compiler in a logical
|
|
place with a logical name, but if it has a strange names or it is
|
|
located in a strange location, you will have to instruct \configure\
|
|
to find it. Note that in many PC clusters (Beowulf), there is no
|
|
parallel Fortran-95 compiler in default installations: you have to
|
|
configure an appropriate script, such as mpif90.
|
|
\item \configure\ tries to locate libraries (both mathematical and
|
|
parallel libraries) in the usual places with usual names, but if
|
|
they have strange names or strange locations, you will have to
|
|
rename/move them, or to instruct \configure\ to find them. If MPI
|
|
libraries are not found,
|
|
parallel compilation is disabled.
|
|
\item \configure\ tests that the compiler and the libraries are
|
|
compatible (i.e. the compiler may link the libraries without
|
|
conflicts and without missing symbols). If they aren't and the
|
|
compilation fails, \configure\ will revert to serial compilation.
|
|
\end{itemize}
|
|
|
|
Apart from such problems, \qe\ compiles and works on all non-buggy, properly
|
|
configured hardware and software combinations. You may have to
|
|
recompile MPI libraries: not all MPI installations contain support for
|
|
the fortran-90 compiler of your choice (or for any fortran-90 compiler
|
|
at all!). Useful step-by-step instructions for MPI compilation can be
|
|
found in the following post by Javier Antonio Montoya:\\
|
|
\texttt{http://www.democritos.it/pipermail/pw\_forum/2008April/008818.htm}.
|
|
|
|
If \qe\ does not work for some reason on a PC cluster,
|
|
try first if it works in serial execution. A frequent problem with parallel
|
|
execution is that \qe\ does not read from standard input,
|
|
due to the configuration of MPI libraries: see Sec.\ref{SubSec:para}.
|
|
|
|
If you are dissatisfied with the performances in parallel execution,
|
|
see Sec.\ref{Sec:para} and in particular Sec.\ref{SubSec:badpara}.
|
|
See also the following post from Axel Kohlmeyer:\\
|
|
\texttt{http://www.democritos.it/pipermail/pw\_forum/2008-April/008796.html}
|
|
|
|
\subsubsection{Intel Mac OS X}
|
|
|
|
Newer Mac OS-X machines (10.4 and later) with Intel CPUs are supported
|
|
by \configure,
|
|
with gcc4+g95, gfortran, and the Intel compiler ifort with MKL libraries.
|
|
Parallel compilation with OpenMPI also works.
|
|
|
|
\paragraph{Intel Mac OS X with ifort}
|
|
|
|
"Uninstall darwin ports, fink and developer tools. The presence of all of
|
|
those at the same time generates many spooky events in the compilation
|
|
procedure. I installed just the developer tools from apple, the intel
|
|
fortran compiler and everything went on great" (Info by Riccardo Sabatini,
|
|
Nov. 2007)
|
|
|
|
\paragraph{Intel Mac OS X 10.4 with g95 and gfortran}
|
|
|
|
An updated version of Developer Tools (XCode 2.4.1 or 2.5), that can be
|
|
downloaded from Apple, may be needed. Some tests fails with mysterious
|
|
errors, that disappear if
|
|
fortran BLAS are linked instead of system Atlas libraries. Use:
|
|
\begin{verbatim}
|
|
BLAS_LIBS_SWITCH = internal
|
|
BLAS_LIBS = /path/to/espresso/BLAS/blas.a -latlas
|
|
\end{verbatim}
|
|
(Info by Paolo Giannozzi, jan.2008, updated April 2010)
|
|
|
|
\paragraph{Detailed installation instructions for Mac OS X 10.6}
|
|
|
|
(Instructions for 10.6.3 by Osman Baris Malcioglu, tested as of May 2010)
|
|
Summary for the hasty:
|
|
\begin{itemize}
|
|
\item GNU fortran:
|
|
Install macports compilers,
|
|
Install MPI environment,
|
|
Configure \qe\ using
|
|
\begin{verbatim}
|
|
./configure CC=gcc-mp-4.3 CPP=cpp-mp-4.3 CXX=g++-mp-4.3 F77=g95 FC=g95
|
|
\end{verbatim}
|
|
\item Intel compiler:
|
|
Use Version $>11.1.088$,
|
|
Use 32 bit compilers,
|
|
Install MPI environment,
|
|
install macports provided cpp (optional),
|
|
Configure \qe\ using
|
|
\begin{verbatim}
|
|
./configure CC=icc CXX=icpc F77=ifort F90=ifort FC=ifort CPP=cpp-mp-4.3
|
|
\end{verbatim}
|
|
\end{itemize}
|
|
|
|
\paragraph{Compilation with GNU compilers}.
|
|
The following instructions use macports version of gnu compilers due to some
|
|
issues in mixing gnu supplied fortran compilers with apple modified gnu compiler
|
|
collection. For more information regarding macports please refer to:
|
|
\texttt{http://www.macports.org/}
|
|
|
|
First install necessary compilers from macports
|
|
\begin{verbatim}
|
|
port install gcc43
|
|
port install g95
|
|
\end{verbatim}
|
|
The apple supplied MPI environment has to be overridden since there is
|
|
a new set of compilers now (and Apple provided mpif90 is just an empty
|
|
placeholder since Apple does not provide fortran compilers). I have used
|
|
OpenMPI for this case. Recommended minimum configuration line is:
|
|
\begin{verbatim}
|
|
./configure CC=gcc-mp-4.3 CPP=cpp-mp-4.3 CXX=g++-mp-4.3 F77=g95 FC=g95
|
|
\end{verbatim}
|
|
of course, installation directory should be set accordingly if a multiple
|
|
compiler environment is desired. The default installation directory of
|
|
OpenMPI overwrites apple supplied MPI permanently!\\
|
|
Next step is \qe\ itself. Sadly, the Apple supplied optimized BLAS/LAPACK
|
|
libraries tend to misbehave under different tests, and it is much safer to
|
|
use internal libraries. The minimum recommended configuration
|
|
line is (presuming the environment is set correctly):
|
|
\begin{verbatim}
|
|
./configure CC=gcc-mp-4.3 CXX=g++-mp-4.3 F77=g95 F90=g95 FC=g95 \
|
|
CPP=cpp-mp-4.3 --with-internal-blas --with-internal-lapack
|
|
\end{verbatim}
|
|
\paragraph{Compilation with Intel compilers}.
|
|
Newer versions of Intel compiler (>11.1.067) support Mac OS X 10.6, and furthermore they are
|
|
bundled with intel MKL. 32 bit binaries obtained using 11.1.088 are tested and no problems
|
|
have been encountered so far. Sadly, as of 11.1.088 the 64 bit binary misbehave
|
|
under some tests. Any attempt to compile 64 bit binary using v.$<11.1.088$ will result in
|
|
very strange compilation errors.
|
|
|
|
Like the previous section, I would recommend installing macports compiler suite.
|
|
First, make sure that you are using the 32 bit version of the compilers,
|
|
i.e.
|
|
\begin{verbatim}
|
|
. /opt/intel/Compiler/11.1/088/bin/ifortvars.sh ia32
|
|
\end{verbatim}
|
|
\begin{verbatim}
|
|
. /opt/intel/Compiler/11.1/088/bin/iccvars.sh ia32
|
|
\end{verbatim}
|
|
will set the environment for 32 bit compilation in my case.
|
|
|
|
Then, the MPI environment has to be set up for Intel compilers similar to previous
|
|
section.
|
|
|
|
The recommended configuration line for \qe\ is:
|
|
\begin{verbatim}
|
|
./configure CC=icc CXX=icpc F77=ifort F90=ifort FC=ifort CPP=cpp-mp-4.3
|
|
\end{verbatim}
|
|
MKL libraries will be detected automatically if they are in their default locations.
|
|
Otherwise, mklvars32 has to be sourced before the configuration script.
|
|
|
|
Security issues:
|
|
MacOs 10.6 comes with a disabled firewall. Preparing a ipfw based firewall is recommended.
|
|
Open source and free GUIs such as "WaterRoof" and "NoobProof" are available that may help
|
|
you in the process.
|
|
|
|
\newpage
|
|
|
|
\section{Parallelism}
|
|
\label{Sec:para}
|
|
|
|
\subsection{Understanding Parallelism}
|
|
|
|
Two different parallelization paradigms are currently implemented
|
|
in \qe:
|
|
\begin{enumerate}
|
|
\item {\em Message-Passing (MPI)}. A copy of the executable runs
|
|
on each CPU; each copy lives in a different world, with its own
|
|
private set of data, and communicates with other executables only
|
|
via calls to MPI libraries. MPI parallelization requires compilation
|
|
for parallel execution, linking with MPI libraries, execution using
|
|
a launcher program (depending upon the specific machine). The number of CPUs used
|
|
is specified at run-time either as an option to the launcher or
|
|
by the batch queue system.
|
|
\item {\em OpenMP}. A single executable spawn subprocesses
|
|
(threads) that perform in parallel specific tasks.
|
|
OpenMP can be implemented via compiler directives ({\em explicit}
|
|
OpenMP) or via {\em multithreading} libraries ({\em library} OpenMP).
|
|
Explicit OpenMP require compilation for OpenMP execution;
|
|
library OpenMP requires only linking to a multithreading
|
|
version of mathematical libraries, e.g.:
|
|
ESSLSMP, ACML\_MP, MKL (the latter is natively multithreading).
|
|
The number of threads is specified at run-time in the environment
|
|
variable OMP\_NUM\_THREADS.
|
|
\end{enumerate}
|
|
|
|
MPI is the well-established, general-purpose parallelization.
|
|
In \qe\ several parallelization levels, specified at run-time
|
|
via command-line options to the executable, are implemented
|
|
with MPI. This is your first choice for execution on a parallel
|
|
machine.
|
|
|
|
Library OpenMP is a low-effort parallelization suitable for
|
|
multicore CPUs. Its effectiveness relies upon the quality of
|
|
the multithreading libraries and the availability of
|
|
multithreading FFTs. If you are using MKL,\footnote{Beware:
|
|
MKL v.10.2.2 has a buggy \texttt{dsyev} yielding wrong results
|
|
with more than one thread; fixed in v.10.2.4}
|
|
you may want to select FFTW3 (set \texttt{CPPFLAGS=-D\_\_FFTW3...}
|
|
in \texttt{make.sys}) and to link with the MKL interface to FFTW3.
|
|
You will get a decent speedup ($\sim 25$\%) on two cores.
|
|
|
|
Explicit OpenMP is a recent addition, still under
|
|
development, devised to increase scalability on
|
|
large multicore parallel machines. Explicit OpenMP can be used
|
|
together with MPI and also together with library OpenMP. Beware
|
|
conflicts between the various kinds of parallelization!
|
|
If you don't know how to run MPI processes
|
|
and OpenMP threads in a controlled manner, forget about mixed
|
|
OpenMP-MPI parallelization.
|
|
|
|
\subsection{Running on parallel machines}
|
|
\label{SubSec:para}
|
|
|
|
Parallel execution is strongly system- and installation-dependent.
|
|
Typically one has to specify:
|
|
\begin{enumerate}
|
|
\item a launcher program (not always needed),
|
|
such as \texttt{poe}, \texttt{mpirun}, \texttt{mpiexec},
|
|
with the appropriate options (if any);
|
|
\item the number of processors, typically as an option to the launcher
|
|
program, but in some cases to be specified after the name of the
|
|
program to be
|
|
executed;
|
|
\item the program to be executed, with the proper path if needed;
|
|
\item other \qe-specific parallelization options, to be
|
|
read and interpreted by the running code.
|
|
\end{enumerate}
|
|
Items 1) and 2) are machine- and installation-dependent, and may be
|
|
different for interactive and batch execution. Note that large
|
|
parallel machines are often configured so as to disallow interactive
|
|
execution: if in doubt, ask your system administrator.
|
|
Item 3) also depend on your specific configuration (shell, execution path, etc).
|
|
Item 4) is optional but it is very important
|
|
for good performances. We refer to the next
|
|
section for a description of the various
|
|
possibilities.
|
|
|
|
\subsection{Parallelization levels}
|
|
|
|
In \qe\ several MPI parallelization levels are
|
|
implemented, in which both calculations
|
|
and data structures are distributed across processors.
|
|
Processors are organized in a hierarchy of groups,
|
|
which are identified by different MPI communicators level.
|
|
The groups hierarchy is as follow:
|
|
\begin{itemize}
|
|
\item {\bf world}: is the group of all processors (MPI\_COMM\_WORLD).
|
|
\item
|
|
{\bf images}: Processors can then be divided into different "images", each corresponding to a
|
|
different self-consistent or linear-response
|
|
calculation, loosely coupled to others.
|
|
\item
|
|
{\bf pools}: each image can be subpartitioned into
|
|
"pools", each taking care of a group of k-points.
|
|
\item
|
|
{\bf bands}: each pool is subpartitioned into
|
|
"band groups", each taking care of a group
|
|
of Kohn-Sham orbitals (also called bands, or
|
|
wavefunctions) (still experimental)
|
|
\item
|
|
{\bf PW}: orbitals in the PW basis set,
|
|
as well as charges and density in either
|
|
reciprocal or real space, are distributed
|
|
across processors.
|
|
This is usually referred to as "PW parallelization".
|
|
All linear-algebra operations on array of PW /
|
|
real-space grids are automatically and effectively parallelized.
|
|
3D FFT is used to transform electronic wave functions from
|
|
reciprocal to real space and vice versa. The 3D FFT is
|
|
parallelized by distributing planes of the 3D grid in real
|
|
space to processors (in reciprocal space, it is columns of
|
|
G-vectors that are distributed to processors).
|
|
\item
|
|
{\bf tasks}:
|
|
In order to allow good parallelization of the 3D FFT when
|
|
the number of processors exceeds the number of FFT planes,
|
|
FFTs on Kohn-Sham states are redistributed to
|
|
"task" groups so that each group
|
|
can process several wavefunctions at the same time.
|
|
\item
|
|
{\bf linear-algebra group}:
|
|
A further level of parallelization, independent on
|
|
PW or k-point parallelization, is the parallelization of
|
|
subspace diagonalization / iterative orthonormalization.
|
|
Both operations required the diagonalization of
|
|
arrays whose dimension is the number of Kohn-Sham states
|
|
(or a small multiple of it). All such arrays are distributed block-like
|
|
across the ``linear-algebra group'', a subgroup of the pool of processors,
|
|
organized in a square 2D grid. As a consequence the number of processors
|
|
in the linear-algebra group is given by $n^2$, where $n$ is an integer;
|
|
$n^2$ must be smaller than the number of processors in the PW group.
|
|
The diagonalization is then performed
|
|
in parallel using standard linear algebra operations.
|
|
(This diagonalization is used by, but should not be confused with,
|
|
the iterative Davidson algorithm). The preferred option is to use
|
|
ScaLAPACK; alternative built-in algorithms are anyway available.
|
|
\end{itemize}
|
|
Note however that not all parallelization levels
|
|
are implemented in all codes!
|
|
|
|
\paragraph{About communications}
|
|
Images and pools are loosely coupled and processors communicate
|
|
between different images and pools only once in a while, whereas
|
|
processors within each pool are tightly coupled and communications
|
|
are significant. This means that Gigabit ethernet (typical for
|
|
cheap PC clusters) is ok up to 4-8 processors per pool, but {\em fast}
|
|
communication hardware (e.g. Mirynet or comparable) is absolutely
|
|
needed beyond 8 processors per pool.
|
|
|
|
\paragraph{Choosing parameters}:
|
|
To control the number of processors in each group,
|
|
command line switches:
|
|
\texttt{-nimage}, \texttt{-npools}, \texttt{-nband},
|
|
\texttt{-ntg}, \texttt{-northo} or \texttt{-ndiag}
|
|
are used.
|
|
As an example consider the following command line:
|
|
\begin{verbatim}
|
|
mpirun -np 4096 ./neb.x -nimage 8 -npool 2 -ntg 8 -ndiag 144 -input my.input
|
|
\end{verbatim}
|
|
This executes a NEB calculation on 4096 processors, 8 images (points in the configuration
|
|
space in this case) at the same time, each of
|
|
which is distributed across 512 processors.
|
|
k-points are distributed across 2 pools of 256 processors each,
|
|
3D FFT is performed using 8 task groups (64 processors each, so
|
|
the 3D real-space grid is cut into 64 slices), and the diagonalization
|
|
of the subspace Hamiltonian is distributed to a square grid of 144
|
|
processors (12x12).
|
|
|
|
Default values are: \texttt{-nimage 1 -npool 1 -ntg 1} ;
|
|
\texttt{ndiag} is set to 1 if ScaLAPACK is not compiled,
|
|
it is set to the square integer smaller than or equal to half the number
|
|
of processors of each pool.
|
|
|
|
\paragraph{Massively parallel calculations}
|
|
For very large jobs (i.e. O(1000) atoms or more) or for very long jobs,
|
|
to be run on massively parallel machines (e.g. IBM BlueGene) it is
|
|
crucial to use in an effective way all available parallelization levels. Without a judicious choice of
|
|
parameters, large jobs will find a stumbling block in either memory or
|
|
CPU requirements.
|
|
|
|
Since v.4.1, ScaLAPACK can be used to diagonalize block distributed
|
|
matrices, yielding better speed-up than the internal algorithms for
|
|
large ($ > 1000\times 1000$) matrices, when using a large number of processors
|
|
($> 512$). You need to have \texttt{-D\_\_SCALAPACK} added to DFLAGS
|
|
in \texttt{make.sys}, LAPACK\_LIBS set to something like:
|
|
\begin{verbatim}
|
|
LAPACK_LIBS = -lscalapack -lblacs -lblacsF77init -lblacs -llapack
|
|
\end{verbatim}
|
|
The repeated \texttt{-lblacs} is not an error, it is needed!
|
|
\configure\ tries to find a ScaLAPACK library, unless
|
|
\texttt{configure --with-scalapack=no} is specified.
|
|
If it doesn't, inquire with your system manager
|
|
on the correct way to link it.
|
|
|
|
A further possibility to expand scalability, especially on machines
|
|
like IBM BlueGene, is to use mixed MPI-OpenMP. The idea is to have
|
|
one (or more) MPI process(es) per multicore node, with OpenMP
|
|
parallelization inside a same node. This option is activated by \texttt{configure --with-openmp},
|
|
which adds preprocessing flag \texttt{-D\_\_OPENMP}
|
|
and one of the following compiler options:
|
|
|
|
\begin{tabular}{ll}
|
|
ifort& \texttt{-openmp}\\
|
|
xlf& \texttt{-qsmp=omp}\\
|
|
PGI& \texttt{-mp}\\
|
|
ftn& \texttt{-mp=nonuma}\\
|
|
\end{tabular}
|
|
|
|
OpenMP parallelization is currently implemented and tested for the following combinations of FFTs
|
|
and libraries:
|
|
|
|
\begin{tabular}{ll}
|
|
internal FFTW copy &requires \texttt{-D\_\_FFTW}\\
|
|
ESSL& requires \texttt{-D\_\_ESSL} or \texttt{-D\_\_LINUX\_ESSL}, link
|
|
with \texttt{-lesslsmp}\\
|
|
\end{tabular}
|
|
|
|
Currently, ESSL (when available) are faster than internal FFTW.
|
|
|
|
\subsubsection{Understanding parallel I/O}
|
|
In parallel execution, each processor has its own slice of data
|
|
(Kohn-Sham orbitals, charge density, etc), that have to be written
|
|
to temporary files during the calculation,
|
|
or to data files at the end of the calculation.
|
|
This can be done in two different ways:
|
|
\begin{itemize}
|
|
\item ``distributed'': each processor
|
|
writes its own slice to disk in its internal
|
|
format to a different file.
|
|
\item ``collected'': all slices are
|
|
collected by the code to a single processor
|
|
that writes them to disk, in a single file,
|
|
using a format that doesn't depend upon
|
|
the number of processors or their distribution.
|
|
\end{itemize}
|
|
|
|
The ``distributed'' format is fast and simple,
|
|
but the data so produced is readable only by
|
|
a job running on the same number of processors,
|
|
with the same type of parallelization, as the
|
|
job who wrote the data, and if all
|
|
files are on a file system that is visible to all
|
|
processors (i.e., you cannot use local scratch
|
|
directories: there is presently no way to ensure
|
|
that the distribution of processes across
|
|
processors will follow the same pattern
|
|
for different jobs).
|
|
|
|
Currently, \CP\ uses the ``collected'' format;
|
|
\PWscf\ uses the ``distributed'' format, but
|
|
has the option to write the final data file in
|
|
``collected'' format (input variable \texttt{wf\_collect})
|
|
so that it can be easily read by \CP\ and by other
|
|
codes running on a different number of processors.
|
|
|
|
In addition to the above, other restrictions to file
|
|
interoperability apply: e.g., \CP\ can read only files
|
|
produced by \PWscf\ for the $k=0$ case.
|
|
|
|
The directory for data is specified in input variables
|
|
\texttt{outdir} and \texttt{prefix} (the former can be specified
|
|
as well in environment variable ESPRESSO\_TMPDIR):
|
|
\texttt{outdir/prefix.save}. A copy of pseudopotential files
|
|
is also written there. If some processor cannot access the
|
|
data directory, the pseudopotential files are read instead
|
|
from the pseudopotential directory specified in input data.
|
|
Unpredictable results may follow if those files
|
|
are not the same as those in the data directory!
|
|
|
|
{\em IMPORTANT:}
|
|
Avoid I/O to network-mounted disks (via NFS) as much as you can!
|
|
Ideally the scratch directory \texttt{outdir} should be a modern
|
|
Parallel File System. If you do not have any, you can use local
|
|
scratch disks (i.e. each node is physically connected to a disk
|
|
and writes to it) but you may run into trouble anyway if you
|
|
need to access your files that are scattered in an unpredictable
|
|
way across disks residing on different nodes.
|
|
|
|
You can use input variable \texttt{disk\_io='minimal'}, or even
|
|
\texttt{'none'}, if you run
|
|
into trouble (or into angry system managers) with excessive I/O with \pwx.
|
|
The code will store wavefunctions into RAM during the calculation.
|
|
Note however that this will increase your memory usage and may limit
|
|
or prevent restarting from interrupted runs.
|
|
|
|
\subsection{Tricks and problems}
|
|
|
|
\paragraph{Trouble with input files}
|
|
Some implementations of the MPI library have problems with input
|
|
redirection in parallel. This typically shows up under the form of
|
|
mysterious errors when reading data. If this happens, use the option
|
|
\texttt{-in} (or \texttt{-inp} or \texttt{-input}), followed by the input file name.
|
|
Example:
|
|
\begin{verbatim}
|
|
pw.x -in inputfile -npool 4 > outputfile
|
|
\end{verbatim}
|
|
Of course the
|
|
input file must be accessible by the processor that must read it
|
|
(only one processor reads the input file and subsequently broadcasts
|
|
its contents to all other processors).
|
|
|
|
Apparently the LSF implementation of MPI libraries manages to ignore or to
|
|
confuse even the \texttt{-in/inp/input} mechanism that is present in all
|
|
\qe\ codes. In this case, use the \texttt{-i} option of \texttt{mpirun.lsf}
|
|
to provide an input file.
|
|
|
|
\paragraph{Trouble with MKL and MPI parallelization}
|
|
If you notice very bad parallel performances with MPI and MKL libraries,
|
|
it is very likely that the OpenMP parallelization performed by the latter
|
|
is colliding with MPI. Recent versions of MKL enable autoparallelization
|
|
by default on multicore machines. You must set the environmental variable
|
|
OMP\_NUM\_THREADS to 1 to disable it.
|
|
Note that if for some reason the correct setting of variable
|
|
OMP\_NUM\_THREADS
|
|
does not propagate to all processors, you may equally run into trouble.
|
|
Lorenzo Paulatto (Nov. 2008) suggests to use the \texttt{-x} option to \texttt{mpirun} to
|
|
propagate OMP\_NUM\_THREADS to all processors.
|
|
Axel Kohlmeyer suggests the following (April 2008):
|
|
"(I've) found that Intel is now turning on multithreading without any
|
|
warning and that is for example why their FFT seems faster than
|
|
FFTW. For serial and OpenMP based runs this makes no difference (in
|
|
fact the multi-threaded FFT helps), but if you run MPI locally, you
|
|
actually lose performance. Also if you use the 'numactl' tool on linux
|
|
to bind a job to a specific cpu core, MKL will still try to use all
|
|
available cores (and slow down badly). The cleanest way of avoiding
|
|
this mess is to either link with
|
|
\begin{quote}
|
|
\texttt{-lmkl\_intel\_lp64 -lmkl\_sequential -lmkl\_core} (on 64-bit:
|
|
x86\_64, ia64)\\
|
|
\texttt{-lmkl\_intel -lmkl\_sequential -lmkl\_core} (on 32-bit, i.e. ia32 )
|
|
\end{quote}
|
|
or edit the \texttt{libmkl\_'platform'.a} file. I'm using now a file
|
|
\texttt{libmkl10.a} with:
|
|
\begin{verbatim}
|
|
GROUP (libmkl_intel_lp64.a libmkl_sequential.a libmkl_core.a)
|
|
\end{verbatim}
|
|
It works like a charm". UPDATE: Since v.4.2, \configure\ links by
|
|
default MKL without multithreaded support.
|
|
|
|
\paragraph{Trouble with compilers and MPI libraries}
|
|
Many users of \qe, in particular those working on PC clusters,
|
|
have to rely on themselves (or on less-than-adequate system managers) for
|
|
the correct configuration of software for parallel execution. Mysterious and
|
|
irreproducible crashes in parallel execution are sometimes due to bugs
|
|
in \qe, but more often than not are a consequence of buggy
|
|
compilers or of buggy or miscompiled MPI libraries. Very useful step-by-step
|
|
instructions to compile and install MPI libraries
|
|
can be found in the following post by Javier Antonio Montoya:\\
|
|
\texttt{http://www.democritos.it/pipermail/pw\_forum/2008-April/008818.htm}.
|
|
|
|
\end{document}
|
|
|