Merge branch 'develop' into update_afqmctools_options

.github/pull_request_template.md

@@ -7,30 +7,29 @@ Describe what this PR changes and why.  If it closes an issue, link to it here
 with [a supported keyword](https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword).
 
 ## What type(s) of changes does this code introduce?
-_Put an `x` in the boxes that apply._
+_Delete the items that do not apply_
 
-- [ ] Bugfix
-- [ ] New feature
-- [ ] Code style update (formatting, renaming)
-- [ ] Refactoring (no functional changes, no api changes)
-- [ ] Build related changes
-- [ ] Documentation content changes
-- [ ] Other (please describe):
+- Bugfix
+- New feature
+- Code style update (formatting, renaming)
+- Refactoring (no functional changes, no api changes)
+- Build related changes
+- Documentation content changes
+- Other (please describe):
 
 ### Does this introduce a breaking change?
 
-- [ ] Yes
-- [ ] No
+- Yes
+- No
 
 ## What systems has this change been tested on?
 
 ## Checklist
 
-_Put an x in the boxes that apply. You can also fill these out after creating
-the PR. If you're unsure about any of them, don't hesitate to ask.  This is
+_Update the following with a yes where the items apply. If you're unsure about any of them, don't hesitate to ask.  This is
 simply a reminder of what we are going to look for before merging your code._
 
-- [ ] this PR is up to date with current the current state of 'develop'
-- [ ] code added or changed in the PR has been clang-formatted
-- [ ] this PR adds tests to cover any new code, or to catch a bug that is being fixed
-- [ ] documentation has been added (if appropriate)
+- Yes/No. This PR is up to date with current the current state of 'develop'
+- Yes/No. Code added or changed in the PR has been clang-formatted
+- Yes/No. This PR adds tests to cover any new code, or to catch a bug that is being fixed
+- Yes/No. Documentation has been added (if appropriate)

CHANGELOG.md

@@ -2,60 +2,73 @@
 
 Notable changes to QMCPACK are documented in this file.
 
-## [Unreleased] 
+## [3.9.1] - 2020-02-11
 
 ### Notes
 
-Work is ongoing to support dynamical spin variables and implement spin-orbit. Prototype limited functionality universal VMC and DMC drivers are available that
-support GPU execution with OpenMP offload.
+This release is the same as v3.9.0 except that the version number of QMCPACK is reported correctly. See the v3.9.0 part of the CHANGELOG for important changes compared to v3.8.0.
 
-- All uses of Python updated to Python 3, which is now required. Python 2 was retired at the end of 2019, and many packages
+## [3.9.0] - 2020-02-11 
+
+### Notes
+
+This release includes a large number of refinements to improve or extend the functionality of QMCPACK and NEXUS. Importantly, this
+release supports and requires Python 3. After this release we plan to remove the array-of-structures build configuration and also
+the legacy CUDA implementation for GPUs. If any needed functionality is not supported by the now-default structures-of-arrays
+configuration, users should contact the developers via the QMCPACK Google Groups or via an issue on the QMCPACK GitHub repository.
+Work is ongoing to support dynamical spin variables, implement spin-orbit, and to develop new support for accelerators via a new
+framework that will consistently support CPUs and GPUs from the same codebase. 
+
+* All uses of Python updated to Python 3, which is now required. Python 2 was retired at the end of 2019, and many packages
   already only support Python 3.
-- Greatly expanded selection of effective core potentials is available at [https://pseudopotentiallibrary.org/](https://pseudopotentiallibrary.org/).    
-- All major functionality is now supported by the default structures-of-arrays (SoA) build. This release is the last to support the legacy array-of-structures (AoS)
+* A greatly expanded selection of effective core potentials is available at
+  [https://pseudopotentiallibrary.org/](https://pseudopotentiallibrary.org/) in formats suitable for QMCPACK and common DFT and
+  quantum chemistry codes.    
+* All major functionality is now supported by the default structures-of-arrays (SoA) build. This release is the last to support the legacy array-of-structures (AoS)
   build. See [\#861](https://github.com/QMCPACK/qmcpack/issues/861).
-- Major bug identified and fixed in the periodic Coulomb evaluation (Optimized breakup method of Natoli-Ceperley). Many thanks to Jan Brndiar
+* Major bug identified and fixed in the periodic Coulomb evaluation (Optimized breakup method of Natoli-Ceperley). Many thanks to Jan Brndiar
   and coworkers for reporting this. For large anisotropic supercells such as a graphene layer with substantial vacuum, the ion-ion potential was
   incorrectly computed. Results in all bulk-like supercells tested so far have been accurate. An independent Ewald check of the ion-ion potential evaluation has been added. See
   [\#2137](https://github.com/QMCPACK/qmcpack/pull/2137). The Coulomb potential evaluation has also been found to converge very slowly for certain
   anisotropic supercells, particularly for quasi-2D cells where huge errors can result. The new independent Coulomb check will
   abort if a tolerance is not reached and provide guidance. Research is ongoing to develop an improved methodology
   [\#2185](https://github.com/QMCPACK/qmcpack/issues/2185).  
-- Support for periodic gaussian-based trial wavefunctions at complex k-points
+* Support for periodic gaussian-based trial wavefunctions at complex k-points
   [\#1988](https://github.com/QMCPACK/qmcpack/issues/1988).
-- Determinant-localization approximation (DLA) of Zen et al. [J. Chem. Phys. 151, 134105
+* Determinant-localization approximation (DLA) of Zen et al. [J. Chem. Phys. 151, 134105
   (2019)](https://doi.org/10.1063/1.5119729) for DMC non-local pseudopotential evaluation implemented.
-- Improved force implementation [\#1769](https://github.com/QMCPACK/qmcpack/issues/1769), [\#1768](https://github.com/QMCPACK/qmcpack/issues/1768).
-- Non-local pseudopotential derivatives are supported in the SoA build and recommended for all optimizations
+* Improved force implementation [\#1769](https://github.com/QMCPACK/qmcpack/issues/1769), [\#1768](https://github.com/QMCPACK/qmcpack/issues/1768).
+* Non-local pseudopotential derivatives are supported in the SoA build and recommended for all optimizations
   [\#2083](https://github.com/QMCPACK/qmcpack/issues/2083).
-- Above 192 electrons in a spin determinant, delayed updating with delay 32 is enabled by default for higher performance,
+* Above 192 electrons in a spin determinant, delayed updating with delay 32 is enabled by default for higher performance,
   [\#2027](https://github.com/QMCPACK/qmcpack/pull/2027). Rank-1 updating is used by default for smaller determinants.
-- Improved configuration and detection of Intel MKL and vector MKL when used with non-Intel compilers.
-- QMCPACK will now run with wavefunctions where only electrons of a single spin are specified.
+* Improved configuration and detection of Intel MKL and vector MKL when used with non-Intel compilers.
+* QMCPACK will now run with wavefunctions where only electrons of a single spin are specified.
   [\#2148](https://github.com/QMCPACK/qmcpack/pull/2148).
-- AFQMC estimators now include 1 and 2 body reduced density matrices (1RRM, 2RDM) and on-top pair density.
+* AFQMC estimators now include 1 and 2 body reduced density matrices (1RRM, 2RDM) and on-top pair density.
   [\#2097](https://github.com/QMCPACK/qmcpack/pull/2097).
-- [QMCPACK spack package](https://spack.readthedocs.io/en/latest/package_list.html#qmcpack)  supports the latest release as well
+* Dense real hamiltonian added for AFQMC allowing for GPU acceleration for chemistry applications. [\#2131](https://github.com/QMCPACK/qmcpack/pull/2131). 
+* [QMCPACK spack package](https://spack.readthedocs.io/en/latest/package_list.html#qmcpack)  supports the latest release as well
   as the development version. This package can also install and patch
   Quantum Espresso. 
-- Support for Blue Gene removed due to retirement of this architecture.
-- Many minor bugfixes, expanded testing, and small feature improvements.
+* Support for Blue Gene removed due to retirement of this architecture.
+* Many minor bug fixes, expanded testing, and small feature improvements.
 
 ### Known bugs
 
 See [list of open bugs](https://github.com/QMCPACK/qmcpack/issues?q=is%3Aissue+is%3Aopen+label%3Abug).
 
-- Use of reconfiguration in DMC is disabled since it is incorrect. [\#2254](https://github.com/QMCPACK/qmcpack/pull/2254)
+* Use of reconfiguration in DMC is disabled since it is incorrect. [\#2254](https://github.com/QMCPACK/qmcpack/pull/2254)
 
 ### NEXUS
 
-- NEXUS version is increased to 2.0.0 due to major updates in this release.
-- NEXUS has been transitioned to Python 3 and now requires it.
-- Significantly expanded test system to cover all major functionality.
-- Full support for PySCF to QMCPACK and AFQMC workflows [\#1970](https://github.com/QMCPACK/qmcpack/pull/1970).
-- Support for DLA [\#2061](https://github.com/QMCPACK/qmcpack/pull/2061).
-- VMC optimization performed with NLPP derivatives by default [\#2128](https://github.com/QMCPACK/qmcpack/pull/2128).
-- Many minor bugfixes and feature improvements.
+* NEXUS version is increased to 2.0.0 due to major updates in this release.
+* NEXUS has been transitioned to Python 3 and now requires it.
+* Significantly expanded test system to cover all major functionality.
+* Full support for PySCF to QMCPACK and AFQMC workflows [\#1970](https://github.com/QMCPACK/qmcpack/pull/1970).
+* Support for DLA [\#2061](https://github.com/QMCPACK/qmcpack/pull/2061).
+* VMC optimization performed with NLPP derivatives by default [\#2128](https://github.com/QMCPACK/qmcpack/pull/2128).
+* Many minor bugfixes and feature improvements.
 
 ## [3.8.0] - 2019-07-23
 

CMake/CheckSincos.cmake

@@ -1,21 +1,6 @@
 INCLUDE(${CMAKE_ROOT}/Modules/CheckCXXSourceCompiles.cmake)
 
-SET(SINCOS_INCLUDE cmath)
-
-SET(ENABLE_MASS FALSE CACHE BOOL "ENABLE MASS math libraries for Power architecture")
-# This needs to go before HAVE_SINCOS
-IF(ENABLE_MASS)
-  INCLUDE(CMake/FindIBMMASS.cmake)
-  IF(HAVE_MASS)
-    SET(CMAKE_REQUIRED_INCLUDES ${MASS_INCLUDE_DIRECTORIES} ${CMAKE_REQUIRED_INCLUDES})
-    SET(CMAKE_REQUIRED_LINK_OPTIONS ${MASS_LINKER_FLAGS})
-    SET(CMAKE_REQUIRED_LIBRARIES ${MASS_LIBRARIES})
-  ENDIF(HAVE_MASS)
-ENDIF(ENABLE_MASS)
-
-MESSAGE("CMAKE_REQUIRED_INCLUDES: ${CMAKE_REQUIRED_INCLUDES}")
-MESSAGE("CMAKE_REQUIRED_LINK_OPTIONS: ${CMAKE_REQUIRED_LINK_OPTIONS}")
-MESSAGE("CMAKE_REQUIRED_LIBRARIES: ${CMAKE_REQUIRED_LIBRARIES}")
+SET(CMAKE_REQUIRED_LIBRARIES Math::scalar_vector_functions)
 
 SET( SINCOS_TEST_SRC
   "#include \"${SINCOS_INCLUDE}\"

CMake/FindFFTW.cmake

@@ -20,6 +20,11 @@ IF(FFTW_INCLUDE_DIR AND FFTW_LIBRARIES)
   MESSAGE(STATUS "FFTW_INCLUDE_DIR=${FFTW_INCLUDE_DIR}")
   MESSAGE(STATUS "FFTW_LIBRARIES=${FFTW_LIBRARIES}")
   SET(FFTW_FOUND TRUE)
+  #create FFTW3 target
+  ADD_LIBRARY(Math::FFTW3 INTERFACE IMPORTED)
+  SET_TARGET_PROPERTIES(Math::FFTW3 PROPERTIES INTERFACE_INCLUDE_DIRECTORIES "${FFTW_INCLUDE_DIR}"
+                                               INTERFACE_COMPILE_DEFINITIONS "HAVE_LIBFFTW"
+                                               INTERFACE_LINK_LIBRARIES "${FFTW_LIBRARIES}")
 ENDIF()
 
 MARK_AS_ADVANCED(

CMake/FindIBMMASS.cmake

@@ -7,9 +7,9 @@ set(SUFFIXES include)
 find_path(MASS_INCLUDE_DIRECTORIES name "mass.h" HINTS ${MASS_ROOT}
   PATH_SUFFIXES ${SUFFIXES})
 if (NOT MASS_INCLUDE_DIRECTORIES)
-  message(FATAL_ERROR "MASS_INCLUDE_DIRECTORIES not set. \"mass.h\" not found in MASS_ROOT/(${SUFFIXES})")
+  MESSAGE(FATAL_ERROR "MASS_INCLUDE_DIRECTORIES not set. \"mass.h\" not found in MASS_ROOT/(${SUFFIXES})")
 endif (NOT MASS_INCLUDE_DIRECTORIES)
-message("MASS_INCLUDE_DIRECTORIES: ${MASS_INCLUDE_DIRECTORIES}")
+MESSAGE(STATUS "MASS_INCLUDE_DIRECTORIES: ${MASS_INCLUDE_DIRECTORIES}")
 
   # Finding and setting the MASS_LINK_DIRECTORIES
   # the directory organization varies with platform and targets
@@ -19,10 +19,10 @@ set(SUFFIXES lib lib64)
 find_path(MASS_LINK_DIRECTORIES name "${MASS_FIND_LIB}" HINTS ${MASS_ROOT}
   PATH_SUFFIXES ${SUFFIXES})
 if (NOT MASS_LINK_DIRECTORIES)
-  message(FATAL_ERROR "MASS_LINK_DIRECTORIES not set. ${MASS_FIND_LIB} "
+  MESSAGE(FATAL_ERROR "MASS_LINK_DIRECTORIES not set. ${MASS_FIND_LIB} "
     "not found in MASS_ROOT/(${SUFFIXES})")
 endif (NOT MASS_LINK_DIRECTORIES)
-message("MASS_LINK_DIRECTORIES: ${MASS_LINK_DIRECTORIES}")
+MESSAGE(STATUS "MASS_LINK_DIRECTORIES: ${MASS_LINK_DIRECTORIES}")
 
 set(MASS_LINKER_FLAGS -L${MASS_LINK_DIRECTORIES} -Wl,-rpath,${MASS_LINK_DIRECTORIES})
 set(MASS_LIBRARIES "-lmass -lmassv")
@@ -45,6 +45,7 @@ for( int i = 0; i < in_size; ++i)
 vlog10(resultv, inputv, &in_size);
 }
 ")
+
 try_compile(HAVE_MASS ${CMAKE_BINARY_DIR}
   ${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/CMakeTmp/src_mass.cxx
   CMAKE_FLAGS
@@ -58,17 +59,17 @@ if ( NOT HAVE_MASS )
 endif ( NOT HAVE_MASS )
 
 IF ( HAVE_MASS )
-  SET( MASS_FOUND 1 )
-  SET( MASS_FLAGS ${MASS_COMPILE_DEFINITIONS} )
-  include_directories( ${MASS_INCLUDE_DIRECTORIES} )
-  set( HAVE_VECTOR_MATH 1 )
-  set( HAVE_MASSV 1 )
-  set( SINCOS_INCLUDE mass.h )
+  SET( MASS_FOUND TRUE )
+  SET( HAVE_MASSV TRUE )
   MESSAGE(STATUS "MASS found: HAVE_MASS=${HAVE_MASS}, HAVE_MASSV=${HAVE_MASSV}")
+
+  #create scalar_vector_functions target
+  ADD_LIBRARY(Math::scalar_vector_functions INTERFACE IMPORTED)
+  SET_TARGET_PROPERTIES(Math::scalar_vector_functions PROPERTIES INTERFACE_COMPILE_DEFINITIONS "HAVE_MASS;HAVE_MASSV"
+                                                                 INTERFACE_INCLUDE_DIRECTORIES "${MASS_INCLUDE_DIRECTORIES}"
+                                                                 INTERFACE_LINK_LIBRARIES "${MASS_LINKER_FLAGS} ${MASS_LIBRARIES}")
+  set( SINCOS_INCLUDE mass.h )
 ELSE( HAVE_MASS )
-  SET( MASS_FOUND 0 )
-  SET( MASS_FLAGS )
-  SET( MASS_LIBRARIES )
-  SET( MASS_LINKER_FLAGS )
-  MESSAGE("MASS not found")
+  SET( MASS_FOUND FALSE )
+  MESSAGE(STATUS "MASS not found")
 ENDIF( HAVE_MASS )

CMake/FindMKL.cmake

@@ -20,21 +20,21 @@ else (NOT MKL_ROOT)
   find_path(MKL_INCLUDE_DIRECTORIES "mkl.h" HINTS ${MKL_ROOT}
     PATH_SUFFIXES ${SUFFIXES})
   if (NOT MKL_INCLUDE_DIRECTORIES)
-    message(FATAL_ERROR "MKL_INCLUDE_DIRECTORIES not set. \"mkl.h\" not found in MKL_ROOT/(${SUFFIXES})")
+    MESSAGE(FATAL_ERROR "MKL_INCLUDE_DIRECTORIES not set. \"mkl.h\" not found in MKL_ROOT/(${SUFFIXES})")
   endif ()
 endif (NOT MKL_ROOT)
 
 if (NOT MKL_ROOT)
   if (CMAKE_CXX_COMPILER_ID MATCHES "Intel")
-    message(FATAL_ERROR "Intel's standard compilervar.sh sets the env variable MKLROOT.\n"
+    MESSAGE(FATAL_ERROR "Intel's standard compilervar.sh sets the env variable MKLROOT.\n"
             "If you are invoking icc without the customary environment\n"
             "you must set the the environment variable or pass cmake MKL_ROOT.")
   else(CMAKE_CXX_COMPILER_ID MATCHES "Intel")
-    message (FATAL_ERROR "ENABLE_MKL is TRUE and mkl directory not found. Set MKL_ROOT." )
+    MESSAGE(FATAL_ERROR "ENABLE_MKL is TRUE and mkl directory not found. Set MKL_ROOT." )
   endif(CMAKE_CXX_COMPILER_ID MATCHES "Intel")
 endif (NOT MKL_ROOT)
 
-message("MKL_INCLUDE_DIRECTORIES: ${MKL_INCLUDE_DIRECTORIES}")
+MESSAGE(STATUS "MKL_INCLUDE_DIRECTORIES: ${MKL_INCLUDE_DIRECTORIES}")
 
 if ( NOT CMAKE_CXX_COMPILER_ID MATCHES "Intel" )
   # Finding and setting the MKL_LINK_DIRECTORIES
@@ -45,11 +45,11 @@ if ( NOT CMAKE_CXX_COMPILER_ID MATCHES "Intel" )
   find_library(MKL_LINK_DIRECTORIES "${MKL_FIND_LIB}" HINTS ${MKL_ROOT}
     PATH_SUFFIXES ${SUFFIXES})
   if (NOT MKL_LINK_DIRECTORIES)
-    message(FATAL_ERROR "MKL_LINK_DIRECTORIES not set. ${MKL_FIND_LIB} "
+    MESSAGE(FATAL_ERROR "MKL_LINK_DIRECTORIES not set. ${MKL_FIND_LIB} "
       "not found in MKL_ROOT/(${SUFFIXES})")
   endif ()
   string(REPLACE "/${MKL_FIND_LIB}" "" MKL_LINK_DIRECTORIES ${MKL_LINK_DIRECTORIES})
-  message("MKL_LINK_DIRECTORIES: ${MKL_LINK_DIRECTORIES}")
+  MESSAGE(STATUS "MKL_LINK_DIRECTORIES: ${MKL_LINK_DIRECTORIES}")
   set(MKL_LINKER_FLAGS "-L${MKL_LINK_DIRECTORIES} -Wl,-rpath,${MKL_LINK_DIRECTORIES}")
   set(MKL_LIBRARIES "-lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl")
 else ( NOT CMAKE_CXX_COMPILER_ID MATCHES "Intel" )
@@ -91,13 +91,12 @@ find_path(MKL_FFTW3 "fftw3.h"
   HINTS ${MKL_INCLUDE_DIRECTORIES}
   PATH_SUFFIXES fftw)
 if(MKL_FFTW3)
-  list(APPEND MKL_INCLUDE_DIRECTORIES ${MKL_FFTW3})
   FILE( WRITE "${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/CMakeTmp/src_mkl_fftw3.cxx"
     "#include <iostream>\n #include <fftw3.h>\n int main() { return 0; }\n" )
   try_compile(HAVE_MKL_FFTW3 ${CMAKE_BINARY_DIR}
     ${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/CMakeTmp/src_mkl_fftw3.cxx
     CMAKE_FLAGS
-    "-DINCLUDE_DIRECTORIES=${MKL_INCLUDE_DIRECTORIES} "
+    "-DINCLUDE_DIRECTORIES=${MKL_FFTW3} "
     "-DLINK_DIRECTORIES=${MKL_LINK_DIRECTORIES}"
     LINK_LIBRARIES "${MKL_LIBRARIES}"
     COMPILE_DEFINITIONS "${MKL_COMPILE_DEFINITIONS}"
@@ -107,14 +106,46 @@ else(MKL_FFTW3)
 endif(MKL_FFTW3)
 
 IF ( HAVE_MKL )
-  SET( MKL_FOUND 1 )
-  SET( MKL_FLAGS ${MKL_COMPILE_DEFINITIONS} )
-  include_directories( ${MKL_INCLUDE_DIRECTORIES} )
+  SET( MKL_FOUND TRUE )
   MESSAGE(STATUS "MKL found: HAVE_MKL=${HAVE_MKL}, HAVE_MKL_VML=${HAVE_MKL_VML}, HAVE_MKL_FFTW3=${HAVE_MKL_FFTW3}")
+
+  #create BLAS_LAPACK target
+  ADD_LIBRARY(Math::BLAS_LAPACK INTERFACE IMPORTED)
+  SET_TARGET_PROPERTIES(Math::BLAS_LAPACK PROPERTIES INTERFACE_COMPILE_DEFINITIONS "HAVE_MKL")
+  IF (NOT CMAKE_CXX_COMPILER_ID MATCHES "Intel" )
+    SET_TARGET_PROPERTIES(Math::BLAS_LAPACK PROPERTIES INTERFACE_INCLUDE_DIRECTORIES "${MKL_INCLUDE_DIRECTORIES}"
+                                                       INTERFACE_LINK_LIBRARIES "${MKL_LINKER_FLAGS} ${MKL_LIBRARIES}")
+  ELSE (NOT CMAKE_CXX_COMPILER_ID MATCHES "Intel" )
+    SET_TARGET_PROPERTIES(Math::BLAS_LAPACK PROPERTIES INTERFACE_COMPILE_OPTIONS "-mkl"
+                                                       INTERFACE_LINK_LIBRARIES "-mkl")
+  ENDIF (NOT CMAKE_CXX_COMPILER_ID MATCHES "Intel" )
+
+  IF( HAVE_MKL_VML )
+    #create scalar_vector_functions target
+    ADD_LIBRARY(Math::scalar_vector_functions INTERFACE IMPORTED)
+    SET_TARGET_PROPERTIES(Math::scalar_vector_functions PROPERTIES INTERFACE_COMPILE_DEFINITIONS "HAVE_MKL;HAVE_MKL_VML")
+    IF (NOT CMAKE_CXX_COMPILER_ID MATCHES "Intel" )
+      SET_TARGET_PROPERTIES(Math::scalar_vector_functions PROPERTIES INTERFACE_INCLUDE_DIRECTORIES "${MKL_INCLUDE_DIRECTORIES}"
+                                                                     INTERFACE_LINK_LIBRARIES "${MKL_LINKER_FLAGS} ${MKL_LIBRARIES}")
+    ELSE (NOT CMAKE_CXX_COMPILER_ID MATCHES "Intel" )
+      SET_TARGET_PROPERTIES(Math::scalar_vector_functions PROPERTIES INTERFACE_COMPILE_OPTIONS "-mkl"
+                                                                     INTERFACE_LINK_LIBRARIES "-mkl")
+    ENDIF (NOT CMAKE_CXX_COMPILER_ID MATCHES "Intel" )
+  ENDIF( HAVE_MKL_VML )
+
+  IF( HAVE_MKL_FFTW3 )
+    #create FFTW3 target
+    ADD_LIBRARY(Math::FFTW3 INTERFACE IMPORTED)
+    SET_TARGET_PROPERTIES(Math::FFTW3 PROPERTIES INTERFACE_INCLUDE_DIRECTORIES "${MKL_FFTW3}"
+                                                 INTERFACE_COMPILE_DEFINITIONS "HAVE_MKL;HAVE_LIBFFTW")
+    IF (NOT CMAKE_CXX_COMPILER_ID MATCHES "Intel" )
+      SET_TARGET_PROPERTIES(Math::FFTW3 PROPERTIES INTERFACE_LINK_LIBRARIES "${MKL_LINKER_FLAGS} ${MKL_LIBRARIES}")
+    ELSE (NOT CMAKE_CXX_COMPILER_ID MATCHES "Intel" )
+      SET_TARGET_PROPERTIES(Math::FFTW3 PROPERTIES INTERFACE_COMPILE_OPTIONS "-mkl"
+                                                   INTERFACE_LINK_LIBRARIES "-mkl")
+    ENDIF (NOT CMAKE_CXX_COMPILER_ID MATCHES "Intel" )
+  ENDIF( HAVE_MKL_FFTW3 )
 ELSE( HAVE_MKL )
-  SET( MKL_FOUND 0 )
-  SET( MKL_FLAGS )
-  SET( MKL_LIBRARIES )
-  SET( MKL_LINKER_FLAGS )
-  MESSAGE("MKL not found")
+  SET( MKL_FOUND FALSE )
+  MESSAGE(STATUS "MKL not found")
 ENDIF( HAVE_MKL )

CMake/FindVectorMath.cmake

@@ -1,42 +0,0 @@
-#
-# Find vectorized math libraries and enable appropriate flags
-#
-# Initial version only for MKL VML. Works for gcc+MKL case. libm and massv detection required.
-#
-
-IF(NOT HAVE_MASS)
-SET( HAVE_VECTOR_MATH 0 )
-
-IF ( HAVE_MKL_VML )
-# We arrive here if MKL was detected earlier by FindMKL
-  SET ( HAVE_VECTOR_MATH 1 )
-  MESSAGE(STATUS "Using MKL Vector Math functions")
-ELSE()
-  #MESSAGE(STATUS "Trying MKL VML")
-  # Check for mkl_vml_functions.h
-  FILE( WRITE "${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/CMakeTmp/src_mkl_vml.cxx"
-      "#include <iostream>\n #include <mkl_vml_functions.h>\n int main() { return 0; }\n" )
-  try_compile(HAVE_MKL_VML ${CMAKE_BINARY_DIR}
-        ${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/CMakeTmp/src_mkl_vml.cxx
-        CMAKE_FLAGS "${CMAKE_CXX_FLAGS}" )
-  IF (HAVE_MKL_VML)
-    # enable VML only when MKL libraries have been picked up
-    IF (MKL_FOUND)
-      SET ( HAVE_VECTOR_MATH 1 )
-      MESSAGE(STATUS "Using MKL Vector Math functions (header file check passed)")
-    ELSE(MKL_FOUND)
-      MESSAGE(STATUS "Drop MKL Vector Math functions (header file check passed but libraries are not available)")
-      SET ( HAVE_MKL_VML 0 )
-    ENDIF(MKL_FOUND)
-  ELSE ()
-    IF (MKL_FOUND)
-      MESSAGE(WARNING "mkl_vml_functions.h check failed but MKL libraries are available. At the risk of losing performance.")
-    ENDIF()
-  ENDIF()
-ENDIF()
-ENDIF()
-IF ( NOT HAVE_VECTOR_MATH )
-  MESSAGE(STATUS "No usable vector math library detected.")
-ENDIF()
-
-

CMake/ctest_script.cmake

@@ -182,8 +182,12 @@ IF ( DEFINED QMC_MPI )
   SET( CTEST_OPTIONS "${CTEST_OPTIONS};-DQMC_MPI=${QMC_MPI}" )
 ENDIF()
 
-IF ( DEFINED MPIEXEC )
-  SET( CTEST_OPTIONS "${CTEST_OPTIONS};-DMPIEXEC=${MPIEXEC}" )
+IF ( DEFINED MPIEXEC_EXECUTABLE )
+  SET( CTEST_OPTIONS "${CTEST_OPTIONS};-DMPIEXEC_EXECUTABLE=${MPIEXEC_EXECUTABLE}" )
+ENDIF()
+
+IF ( DEFINED MPIEXEC_NUMPROC_FLAG )
+  SET( CTEST_OPTIONS "${CTEST_OPTIONS};-DMPIEXEC_NUMPROC_FLAG=${MPIEXEC_NUMPROC_FLAG}" )
 ENDIF()
 
 IF ( DEFINED MPIEXEC_PREFLAGS )

CMake/macros.cmake

@@ -70,20 +70,6 @@ FUNCTION(MAYBE_SYMLINK SRC_DIR DST_DIR)
 ENDFUNCTION()
 
 
-# Macro to add the dependencies and libraries to an executable
-MACRO( ADD_QMC_EXE_DEP EXE )
-    # Add the package dependencies
-    TARGET_LINK_LIBRARIES(${EXE} qmc qmcdriver qmcham qmcwfs qmcbase qmcutil)
-    FOREACH(l ${QMC_UTIL_LIBS})
-        TARGET_LINK_LIBRARIES(${EXE} ${l})
-    ENDFOREACH(l ${QMC_UTIL_LIBS})
-    IF(MPI_LIBRARY)
-        TARGET_LINK_LIBRARIES(${EXE} ${MPI_LIBRARY})
-    ENDIF(MPI_LIBRARY)
-ENDMACRO()
-
-
-
 # Macro to create the test name
 MACRO( CREATE_TEST_NAME TEST ${ARGN} )
     SET( TESTNAME "${TEST}" )
@@ -100,11 +86,11 @@ FUNCTION( RUN_QMC_APP_NO_COPY TESTNAME WORKDIR PROCS THREADS TEST_ADDED TEST_LAB
     MATH( EXPR TOT_PROCS "${PROCS} * ${THREADS}" )
     SET( QMC_APP "${qmcpack_BINARY_DIR}/bin/qmcpack" )
     SET( TEST_ADDED_TEMP FALSE )
-    IF ( USE_MPI )
+    IF ( HAVE_MPI )
         IF ( ${TOT_PROCS} GREATER ${TEST_MAX_PROCS} )
             MESSAGE_VERBOSE("Disabling test ${TESTNAME} (exceeds maximum number of processors ${TEST_MAX_PROCS})")
-        ELSEIF ( USE_MPI )
-            ADD_TEST( ${TESTNAME} ${MPIEXEC} ${MPIEXEC_NUMPROC_FLAG} ${PROCS} ${MPIEXEC_PREFLAGS} ${QMC_APP} ${ARGN} )
+        ELSE()
+            ADD_TEST( ${TESTNAME} ${MPIEXEC_EXECUTABLE} ${MPIEXEC_NUMPROC_FLAG} ${PROCS} ${MPIEXEC_PREFLAGS} ${QMC_APP} ${ARGN} )
             SET_TESTS_PROPERTIES( ${TESTNAME} PROPERTIES FAIL_REGULAR_EXPRESSION "${TEST_FAIL_REGULAR_EXPRESSION}"
                 PROCESSORS ${TOT_PROCS} PROCESSOR_AFFINITY TRUE WORKING_DIRECTORY ${WORKDIR}
                 ENVIRONMENT OMP_NUM_THREADS=${THREADS} )

CMake/run_qe.cmake

@@ -11,8 +11,8 @@ IF (QMC_NO_SLOW_CUSTOM_TESTING_COMMANDS)
 ELSE (QMC_NO_SLOW_CUSTOM_TESTING_COMMANDS)
 
 FUNCTION( ADD_QE_TEST TESTNAME PROCS TEST_BINARY NPOOL WORKDIR TEST_INPUT)
-    IF ( USE_MPI )
-        ADD_TEST( NAME ${TESTNAME} COMMAND ${MPIEXEC} ${MPIEXEC_NUMPROC_FLAG} ${PROCS} ${MPIEXEC_PREFLAGS} ${TEST_BINARY} -npool ${NPOOL} -inp ${TEST_INPUT} )
+    IF ( HAVE_MPI )
+        ADD_TEST( NAME ${TESTNAME} COMMAND ${MPIEXEC_EXECUTABLE} ${MPIEXEC_NUMPROC_FLAG} ${PROCS} ${MPIEXEC_PREFLAGS} ${TEST_BINARY} -npool ${NPOOL} -inp ${TEST_INPUT} )
     ELSE()
         ADD_TEST( NAME ${TESTNAME} COMMAND ${TEST_BINARY} -npool 1 ${TEST_INPUT} )
     ENDIF()

CMake/unit_test.cmake

@@ -4,8 +4,8 @@ INCLUDE("${PROJECT_SOURCE_DIR}/CMake/test_labels.cmake")
 # Runs unit tests
 FUNCTION( ADD_UNIT_TEST TESTNAME TEST_BINARY )
     MESSAGE_VERBOSE("Adding test ${TESTNAME}")
-    IF ( USE_MPI )
-        ADD_TEST(NAME ${TESTNAME} COMMAND ${MPIEXEC} ${MPIEXEC_NUMPROC_FLAG} 1 ${MPIEXEC_PREFLAGS} ${TEST_BINARY} ${ARGN})
+    IF ( HAVE_MPI )
+        ADD_TEST(NAME ${TESTNAME} COMMAND ${MPIEXEC_EXECUTABLE} ${MPIEXEC_NUMPROC_FLAG} 1 ${MPIEXEC_PREFLAGS} ${TEST_BINARY} ${ARGN})
         #SET_TESTS_PROPERTIES( ${TESTNAME} PROPERTIES ENVIRONMENT OMP_NUM_THREADS=1 )
     ELSE()
         ADD_TEST(NAME ${TESTNAME} COMMAND ${TEST_BINARY} ${ARGN})
@@ -18,8 +18,8 @@ ENDFUNCTION()
 
 FUNCTION( ADD_MPI_UNIT_TEST TESTNAME TEST_BINARY PROC_COUNT )
     MESSAGE_VERBOSE("Adding test ${TESTNAME}")
-    IF ( USE_MPI )
-      ADD_TEST(NAME ${TESTNAME} COMMAND ${MPIEXEC} ${MPIEXEC_NUMPROC_FLAG} ${PROC_COUNT} ${MPIEXEC_PREFLAGS} ${TEST_BINARY} ${ARGN})
+    IF ( HAVE_MPI )
+      ADD_TEST(NAME ${TESTNAME} COMMAND ${MPIEXEC_EXECUTABLE} ${MPIEXEC_NUMPROC_FLAG} ${PROC_COUNT} ${MPIEXEC_PREFLAGS} ${TEST_BINARY} ${ARGN})
       # Tests should be able to deal with any number of threads but mpi aware unit tests aren't
       # guaranteed yet.
     SET_TESTS_PROPERTIES( ${TESTNAME} PROPERTIES ENVIRONMENT OMP_NUM_THREADS=1 )

CMakeLists.txt

@@ -1,7 +1,7 @@
 ######################################################################
 # CMake version and policies
 ######################################################################
-CMAKE_MINIMUM_REQUIRED(VERSION 3.6.0)
+CMAKE_MINIMUM_REQUIRED(VERSION 3.10.0)
 
 cmake_policy(SET CMP0003 NEW)
 IF(NOT $ENV{CRAYPE_VERSION} MATCHES ".")
@@ -29,7 +29,7 @@ ENDIF(${CMAKE_VERSION} VERSION_GREATER_EQUAL "3.12.0")
 PROJECT(qmcpack)
 
 SET(QMCPACK_VERSION_MAJOR 3)
-SET(QMCPACK_VERSION_MINOR 8)
+SET(QMCPACK_VERSION_MINOR 9)
 SET(QMCPACK_VERSION_PATCH 9)
 SET(QMCPACK_VERSION "${QMCPACK_VERSION_MAJOR}.${QMCPACK_VERSION_MINOR}.${QMCPACK_VERSION_PATCH}")
 
@@ -40,20 +40,6 @@ SET (PROJECT_CMAKE ${qmcpack_SOURCE_DIR}/CMake)
 
 SET(QMCPACK_UNIT_TEST_DIR ${qmcpack_BINARY_DIR}/tests/bin)
 
-
-IF (COMPILING_ON_BGQ)
-  # YL: Put this after PROJECT(qmcpack) because COMPILING_ON_BGQ is defined
-  # in BGQ toolchain file which is loaded by PROJECT().
-  # The following issue probably affect all static linking machines.
-  # MD: On BG/Q, the HDF libraries variable includes some supporting shared libraries
-  # such as libm.so.  With the old behavior of CMP0060, these were converted
-  # to -lm for the linker, which then correctly selects the static version.
-  # With the new behavior of CMP0060, the form of the libraries are left alone
-  # on the link line (eg. /usr/lib64/libm.so), and this causes a linker error
-  # because shared libraries are not allowed with static linking.
-  cmake_policy(SET CMP0060 OLD)
-ENDIF()
-
 ######################################################################
 # Verify Python3 available
 ######################################################################
@@ -240,8 +226,6 @@ INCLUDE(${CMAKE_ROOT}/Modules/CheckCXXSymbolExists.cmake)
 INCLUDE(${CMAKE_ROOT}/Modules/CheckIncludeFile.cmake)
 INCLUDE(${CMAKE_ROOT}/Modules/CheckTypeSize.cmake)
 
-INCLUDE(${PROJECT_CMAKE}/CheckSincos.cmake)
-
 CHECK_FUNCTION_EXISTS(posix_memalign HAVE_POSIX_MEMALIGN)
 
 CHECK_INCLUDE_FILE(dlfcn.h HAVE_DLFCN_H)
@@ -292,10 +276,8 @@ SET(INSTALL_NEXUS 1 CACHE BOOL "Install Nexus alongside QMCPACK")
 #list of options to determine libraries. First, try to use libraries available
 #FindXYZ.cmake will set the values if successful
 ######################################################################
-SET(HAVE_LIBXML2 1)
-SET(HAVE_LIBFFTW 0)
+SET(HAVE_EINSPLINE 1) # to be removed
 SET(HAVE_LIBXMLPP 0)
-SET(HAVE_LIBSPRNG 0)
 SET(HAVE_LIBBLITZ 0)
 SET(HAVE_LIBGSL 0)
 SET(MAC_VECLIB 0)
@@ -332,10 +314,7 @@ IF(CMAKE_TOOLCHAIN_FILE)
 ELSE(CMAKE_TOOLCHAIN_FILE)
   MESSAGE(STATUS "Trying to figure out .....")
   SET(ENABLE_OPENMP 0)
-  SET(HAVE_MPI 0)
   SET(INTEL_COMPILER 0)
-  SET(HAVE_MKL 0)
-  SET(HAVE_MKL_VML 0)
   SET(HAVE_ACML 0)
 
   ######################################################
@@ -351,16 +330,18 @@ ELSE(CMAKE_TOOLCHAIN_FILE)
     MESSAGE(STATUS "Running on a Cray machine.")
     SET(CMAKE_SKIP_RPATH TRUE)
 
-    # Flags for ctest
-    IF(NOT DEFINED MPIEXEC)
-      FIND_PROGRAM(MPIEXEC aprun)
-      IF(NOT MPIEXEC)
-      # For slurm use srun
-        FIND_PROGRAM(MPIEXEC srun)
-      ENDIF(NOT MPIEXEC)
+    IF(NOT $ENV{CRAYPE_LINK_TYPE} STREQUAL "dynamic")
+      MESSAGE(WARNING "Cray Programming Environment uses static linking by default. "
+                      "We prefer dynamic linking which makes library searching easier. "
+                      "Set environment variable CRAYPE_LINK_TYPE to \"dynamic\" enables dynamic linking.\n")
     ENDIF()
-    IF(NOT DEFINED MPIEXEC_NUMPROC_FLAG)
-      SET(MPIEXEC_NUMPROC_FLAG "-n")
+
+    IF ( NOT DEFINED MPIEXEC_EXECUTABLE )
+      FIND_PROGRAM(MPIEXEC_EXECUTABLE aprun)
+      IF(NOT MPIEXEC_EXECUTABLE)
+        # For slurm use srun
+        FIND_PROGRAM(MPIEXEC_EXECUTABLE srun)
+      ENDIF(NOT MPIEXEC_EXECUTABLE)
     ENDIF()
 
     IF(QMC_CUDA)
@@ -446,53 +427,6 @@ ELSE(CMAKE_TOOLCHAIN_FILE)
   #---------------------------------------------------
   include(CMake/TestCxx14Library.cmake)
 
-  #-------------------------------------------------------------------
-  #  check MPI installation
-  #-------------------------------------------------------------------
-  IF(QMC_MPI)
-    ## mpi compilers
-    GET_FILENAME_COMPONENT(BASE_CXX_COMPILER_NAME ${CMAKE_CXX_COMPILER} NAME)
-    if(${BASE_CXX_COMPILER_NAME} MATCHES "^mp")
-      SET(HAVE_MPI 1)
-      SET(MPI_FOUND TRUE)
-    elseif($ENV{CRAYPE_VERSION} MATCHES ".")
-      SET(HAVE_MPI 1)
-      SET(MPI_FOUND TRUE)
-    else()
-      include(FindMPI)
-      IF(MPI_FOUND)
-        MESSAGE(STATUS "Found mpi.h and libraries. Check for working mpi compiler ")
-        INCLUDE_DIRECTORIES(${MPI_CXX_INCLUDE_PATH})
-        FILE(WRITE ${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/CMakeTmp/testmpi.cxx
-          "#include <mpi.h>"
-          "int main(int argc, char** argv){MPI_Init(&argc,&argv); MPI_Finalize();return 0;}\n")
-        #TRY_COMPILE(MPI_WORKS ${CMAKE_BINARY_DIR}
-        #${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/CMakeTmp/testmpi.cxx
-        #OUTPUT_VARIABLE OUTPUT)
-        set(MPI_WORKS 1)
-        SET(MPI_WARNING_LIST
-            "Building MPI version without using MPI compiler wrappers.\n"
-            "This may not build qmcpack correctly. To ensure the correct version, specify the compiler wrappers to cmake.\n"
-            "For example: cmake -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpic++\n"
-            "To build without MPI, pass '-DQMC_MPI=0' to cmake")
-        MESSAGE(WARNING ${MPI_WARNING_LIST})
-
-        IF(MPI_WORKS)
-          MESSAGE(STATUS "MPI is enabled")
-          SET(HAVE_MPI 1)
-          LINK_LIBRARIES(${MPI_CXX_LIBRARIES})
-        ELSE(MPI_WORKS)
-          MESSAGE(STATUS "MPI is disabled")
-          SET(HAVE_MPI 0)
-          SET(MPI_LIBRARY "")
-        ENDIF(MPI_WORKS)
-      ELSE(MPI_FOUND)
-        MESSAGE(STATUS "MPI not found")
-        SET(MPI_LIBRARY "")
-      ENDIF(MPI_FOUND)
-    ENDIF()
-  ENDIF(QMC_MPI)
-
   #-------------------------------------------------------------------
   # check lapack/blas
   #-------------------------------------------------------------------
@@ -508,34 +442,25 @@ ELSE(CMAKE_TOOLCHAIN_FILE)
    INCLUDE(CMake/FindMKL.cmake)
   ENDIF(ENABLE_MKL)
   IF ( HAVE_MKL )
-    # We found MKL, add the link library
-    SET(CMAKE_C_FLAGS   "${CMAKE_C_FLAGS} ${MKL_FLAGS}")
-    SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${MKL_FLAGS}")
-    SET(LAPACK_FOUND 1)
-    SET(LAPACK_LIBRARIES "${MKL_LIBRARIES}")
-    SET(LAPACK_LINKER_FLAGS "${MKL_LINKER_FLAGS}")
+    # Skip adding Math::BLAS_LAPACK target which has been imported by FindMKL.
   ELSEIF ( $ENV{CRAYPE_VERSION} MATCHES "." )
     # Cray specific environment variable indicates we are on a cray. Assume libsci will be linked
     # preferable to attempt a compile/link
-    SET(LAPACK_FOUND 1)
     MESSAGE(STATUS "Assuming Cray libsci or other BLAS/LAPACK will be linked by compiler wrappers")
+    ADD_LIBRARY(Math::BLAS_LAPACK INTERFACE IMPORTED)
   ELSE()
-    # Search for Blas/Lapack
-    find_package(LAPACK)
+    # The last search for Blas/Lapack
+    find_package(LAPACK REQUIRED)
     IF("${LAPACK_LIBRARIES}" MATCHES "mkl")
       MESSAGE(FATAL_ERROR "MKL found via LAPACK/BLAS. Please pass -DENABLE_MKL=1 to cmake for full Intel MKL utilization. -DMKL_ROOT may also be needed for the include files to be found.")
     ENDIF()
-  ENDIF()
-  IF(LAPACK_FOUND)
     MESSAGE(STATUS "LAPACK libraries: ${LAPACK_LIBRARIES}")
     MESSAGE(STATUS "LAPACK linker flags: ${LAPACK_LINKER_FLAGS}")
-    SET(LAPACK_LIBRARY ${LAPACK_LIBRARIES} ${LAPACK_LINKER_FLAGS})
-  ELSE()
-    MESSAGE(FATAL_ERROR "Could not find required libraries LAPACK &/or BLAS")
+    SET(LAPACK_LIBRARY ${LAPACK_LINKER_FLAGS} ${LAPACK_LIBRARIES})
+    # Add Math::BLAS_LAPACK target
+    ADD_LIBRARY(Math::BLAS_LAPACK INTERFACE IMPORTED)
+    SET_TARGET_PROPERTIES(Math::BLAS_LAPACK PROPERTIES INTERFACE_LINK_LIBRARIES "${LAPACK_LIBRARY}")
   ENDIF()
-
-  INCLUDE(CMake/FindVectorMath.cmake)
-
 ENDIF(CMAKE_TOOLCHAIN_FILE)
 
 IF (ENABLE_GCOV)
@@ -553,30 +478,52 @@ IF (BUILD_AFQMC AND NOT APPLE)
   LINK_LIBRARIES("rt")
 ENDIF()
 
-# setup ctest variables
-IF ( HAVE_MPI )
-  SET( USE_MPI 1 )
-  IF ( NOT DEFINED MPIEXEC )
-    SET( MPIEXEC "mpirun" )
+#-------------------------------------------------------------------
+#  check MPI installation
+#-------------------------------------------------------------------
+IF(QMC_MPI)
+  # for backward compatibility with MPIEXEC
+  IF(DEFINED MPIEXEC AND NOT DEFINED MPIEXEC_EXECUTABLE)
+    MESSAGE(WARNING "Setting MPIEXEC_EXECUTABLE based on MPIEXEC. MPIEXEC has been deprecated! Use MPIEXEC_EXECUTABLE instead!")
+    SET(MPIEXEC_EXECUTABLE ${MPIEXEC})
+  ENDIF(DEFINED MPIEXEC AND NOT DEFINED MPIEXEC_EXECUTABLE)
+  SET(MPI_CXX_SKIP_MPICXX TRUE)
+  SET(MPI_DETERMINE_LIBRARY_VERSION TRUE)
+  FIND_PACKAGE(MPI COMPONENTS CXX)
+
+  IF(NOT MPI_FOUND)
+    MESSAGE(FATAL_ERROR "MPI support not found! Provide MPI compiler wrappers or build without MPI by passing '-DQMC_MPI=0' to cmake.")
+  ENDIF(NOT MPI_FOUND)
+
+  IF(${MPI_CXX_LIBRARY_VERSION_STRING} MATCHES "MVAPICH2" AND NOT ${MPI_CXX_LIBRARY_VERSION_STRING} MATCHES "disable-registration-cache")
+    MESSAGE(FATAL_ERROR "MVAPICH2 with registration cache enabled breaks QMCPACK. "
+                        "Use a different MPI library or build MVAPICH2 with --disable-registration-cache configure option.")
   ENDIF()
 
-  IF ( NOT DEFINED MPIEXEC_NUMPROC_FLAG )
-    SET( MPIEXEC_NUMPROC_FLAG "-np" )
-  ENDIF()
-
-  IF ( NOT DEFINED MPIEXEC_PREFLAGS )
-    SET( MPIEXEC_PREFLAGS "" )
+  IF(NOT ${CMAKE_CXX_COMPILER} STREQUAL ${MPI_CXX_COMPILER})
+    SET(MPI_WARNING_LIST
+        "Building MPI version without using MPI compiler wrappers.\n"
+        "This may not build qmcpack correctly. To ensure the correct version, specify the compiler wrappers to cmake.\n"
+        "For example: cmake -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpic++\n"
+        "To build without MPI, pass '-DQMC_MPI=0' to cmake")
+    MESSAGE(WARNING ${MPI_WARNING_LIST})
   ENDIF()
 
   IF ( NOT TEST_MAX_PROCS )
     SET( TEST_MAX_PROCS 100 )
   ENDIF()
 
-  MESSAGE(STATUS "MPI runner is ${MPIEXEC}")
-  MESSAGE(STATUS "MPIEXEC_NUMPROC_FLAG is ${MPIEXEC_NUMPROC_FLAG}")
-ENDIF(HAVE_MPI)
+  MESSAGE(STATUS "MPI runner MPIEXEC_EXECUTABLE : ${MPIEXEC_EXECUTABLE}")
+  MESSAGE(STATUS "MPIEXEC_NUMPROC_FLAG : ${MPIEXEC_NUMPROC_FLAG}")
+  MESSAGE(STATUS "MPIEXEC_PREFLAGS : ${MPIEXEC_PREFLAGS}")
+  MESSAGE(STATUS "Tests run as : ${MPIEXEC_EXECUTABLE} ${MPIEXEC_NUMPROC_FLAG} NUM_PROCS ${MPIEXEC_PREFLAGS} EXECUTABLE")
 
-INCLUDE(${CMAKE_ROOT}/Modules/FindThreads.cmake)
+  SET(HAVE_MPI 1)
+  MESSAGE(STATUS "MPI is enabled")
+ELSE(QMC_MPI)
+  SET(HAVE_MPI 0)
+  MESSAGE(STATUS "MPI is disabled")
+ENDIF(QMC_MPI)
 
 ####################################################################
 #First check the required libraries. Abort if these are not found.
@@ -586,16 +533,31 @@ INCLUDE(${CMAKE_ROOT}/Modules/FindThreads.cmake)
 # installataions
 # list of libraries to be linked with the main applications using I/O
 
-SET(QMC_UTIL_LIBS ${LAPACK_LIBRARY} ${BLAS_LIBRARY})
+#-------------------------------------------------------------------
+#  set up scalar/vector math libraries
+#-------------------------------------------------------------------
+SET(ENABLE_MASS FALSE CACHE BOOL "ENABLE MASS scalar and vector math libraries for Power architecture")
 
-# This needs a better place to live. as does all the IF( HAVE_MKL ) stuff
-IF(HAVE_MASS)
-  SET(CMAKE_CXX_FLAGS ${MASS_FLAGS} ${CMAKE_CXX_FLAGS})
-  SET(CMAKE_C_FLAGS ${MASS_FLAGS} ${CMAKE_C_FLAGS})
-  SET(MASS_LIBRARY "${MASS_LINKER_FLAGS} ${MASS_LIBRARIES}")
-  SET(QMC_UTIL_LIBS ${MASS_LIBRARY} ${QMC_UTIL_LIBS})
-ENDIF(HAVE_MASS)
+# This needs to go before HAVE_SINCOS
+IF(ENABLE_MASS)
+  INCLUDE(CMake/FindIBMMASS.cmake)
+ENDIF(ENABLE_MASS)
 
+IF(NOT HAVE_MASS AND NOT HAVE_MKL_VML)
+  MESSAGE(STATUS "No usable vector math library detected.")
+  ADD_LIBRARY(Math::scalar_vector_functions INTERFACE IMPORTED)
+ENDIF(NOT HAVE_MASS AND NOT HAVE_MKL_VML)
+
+# CheckSincos relies on SINCOS_INCLUDE which may be modified based on vendor libraries
+IF(NOT SINCOS_INCLUDE)
+  SET(SINCOS_INCLUDE cmath)
+ENDIF(NOT SINCOS_INCLUDE)
+MESSAGE(STATUS "SINCOS_INCLUDE : ${SINCOS_INCLUDE}")
+INCLUDE(${PROJECT_CMAKE}/CheckSincos.cmake)
+
+#-------------------------------------------------------------------
+#  set up FFTW3 library
+#-------------------------------------------------------------------
 SET( FFTW_FOUND 0 )
 IF ( HAVE_MKL )
   IF ( HAVE_MKL_FFTW3 )
@@ -607,8 +569,6 @@ IF ( NOT FFTW_FOUND )
 ENDIF()
 IF(FFTW_FOUND)
   SET(HAVE_LIBFFTW 1)
-  INCLUDE_DIRECTORIES(${FFTW_INCLUDE_DIR})
-  SET(QMC_UTIL_LIBS ${QMC_UTIL_LIBS} ${FFTW_LIBRARIES})
   MESSAGE(STATUS "Found FFTW")
 ELSE(FFTW_FOUND)
 #  MESSAGE(STATUS "Disabling FFTW")
@@ -617,20 +577,24 @@ ENDIF(FFTW_FOUND)
 
 SET(QMC_UTIL_LIBS ${QMC_UTIL_LIBS} ${FORTRAN_LIBRARIES})
 
+#-------------------------------------------------------------------
+#  set up Libxml2 library
+#-------------------------------------------------------------------
 find_package(ZLIB)
 find_package(LibXml2)
-
 IF(LIBXML2_FOUND)
-  INCLUDE_DIRECTORIES(${LIBXML2_INCLUDE_DIR})
-  SET(HAVE_LIBXML2 1)
-  SET(QMC_UTIL_LIBS ${QMC_UTIL_LIBS} ${LIBXML2_LIBRARIES})
+  IF(${CMAKE_VERSION} VERSION_LESS "3.12.0")
+    ADD_LIBRARY(LibXml2::LibXml2 INTERFACE IMPORTED)
+    SET_TARGET_PROPERTIES(LibXml2::LibXml2 PROPERTIES INTERFACE_INCLUDE_DIRECTORIES "${LIBXML2_INCLUDE_DIR}"
+                                                      INTERFACE_LINK_LIBRARIES "${LIBXML2_LIBRARIES}")
+  ENDIF()
 ELSE()
   MESSAGE(FATAL_ERROR "Could not find required library Libxml2")
 ENDIF(LIBXML2_FOUND)
 
-####################################################################
-# set up HDF5 library
-####################################################################
+#-------------------------------------------------------------------
+#  set up HDF5 library
+#-------------------------------------------------------------------
 IF(HAVE_MPI)
   SET(HDF5_PREFER_PARALLEL 1 CACHE BOOL "Request parallel/serial HDF5 library")
 ELSE(HAVE_MPI)
@@ -680,7 +644,6 @@ IF(HDF5_FOUND)
     MESSAGE(WARNING "MPI builds may have potential performance loss not using parallel HDF5!")
   ENDIF()
 
-  INCLUDE_DIRECTORIES(${HDF5_INCLUDE_DIR})
   IF ( CMAKE_BUILD_TYPE AND HDF5_LIBRARIES_DEBUG )
     IF ( CMAKE_BUILD_TYPE MATCHES DEBUG )
       SET( HDF5_LIBRARIES ${HDF5_LIBRARIES_DEBUG} )
@@ -688,30 +651,43 @@ IF(HDF5_FOUND)
       SET( HDF5_LIBRARIES ${HDF5_LIBRARIES_RELEASE} )
     ENDIF()
   ENDIF()
-  SET(QMC_UTIL_LIBS ${QMC_UTIL_LIBS} ${HDF5_LIBRARIES})
+
+  ADD_LIBRARY(IO::HDF5 INTERFACE IMPORTED)
+  SET_TARGET_PROPERTIES(IO::HDF5 PROPERTIES INTERFACE_INCLUDE_DIRECTORIES "${HDF5_INCLUDE_DIR}"
+                                            INTERFACE_COMPILE_DEFINITIONS "HAVE_LIBHDF5;H5_USE_16_API"
+                                            INTERFACE_LINK_LIBRARIES "${HDF5_LIBRARIES}")
+  IF(ENABLE_PHDF5)
+    SET_PROPERTY(TARGET IO::HDF5 APPEND PROPERTY INTERFACE_COMPILE_DEFINITIONS "ENABLE_PHDF5")
+  ENDIF(ENABLE_PHDF5)
 ELSE(HDF5_FOUND)
   MESSAGE(FATAL_ERROR "HDF5 not found. Set HDF5_ROOT")
 ENDIF(HDF5_FOUND)
 
+#-------------------------------------------------------------------
+#  set up libboost library, header only
+#-------------------------------------------------------------------
 #make sure we can find boost if it's not in /usr
 set(Boost_NO_BOOST_CMAKE on)
 find_package(Boost 1.61.0 REQUIRED)
 IF(Boost_FOUND)
-  SET(HAVE_LIBBOOST 1)
-  INCLUDE_DIRECTORIES(${Boost_INCLUDE_DIRS})
+  SET_PROPERTY(TARGET Boost::boost APPEND PROPERTY INTERFACE_COMPILE_DEFINITIONS "HAVE_LIBBOOST")
   MESSAGE(STATUS "Setting Boost_INCLUDE_DIRS=${Boost_INCLUDE_DIRS}")
 ELSE()
   MESSAGE(FATAL_ERROR "Need boost 1.61.0 or higher.  Set BOOST_ROOT")
 ENDIF()
 
+#-------------------------------------------------------------------
+#  set up CUDA compiler options and libraries
+#-------------------------------------------------------------------
 SET(HAVE_CUDA 0)
 IF(QMC_CUDA OR ENABLE_CUDA)
   IF(CMAKE_CXX_COMPILER_ID MATCHES "Clang")
-    set(CUDA_PROPAGATE_HOST_FLAGS ON)
+    SET(CUDA_PROPAGATE_HOST_FLAGS ON)
   ELSE(CMAKE_CXX_COMPILER_ID MATCHES "Clang")
-    set(CUDA_PROPAGATE_HOST_FLAGS OFF)
+    SET(CUDA_PROPAGATE_HOST_FLAGS OFF)
   ENDIF(CMAKE_CXX_COMPILER_ID MATCHES "Clang")
   FIND_PACKAGE(CUDA REQUIRED)
+  SET(CUDA_LINK_LIBRARIES_KEYWORD PRIVATE)
   #set(CUDA_NVCC_FLAGS
   #  "-arch=sm_20;-Drestrict=__restrict__;-DNO_CUDA_MAIN;-O3;-use_fast_math")
   if(CUDA_NVCC_FLAGS MATCHES "arch")
@@ -734,7 +710,7 @@ IF(QMC_CUDA OR ENABLE_CUDA)
     SET(CUDA_NVCC_FLAGS "-arch=${CUDA_ARCH};${CUDA_NVCC_FLAGS}")
   endif(CUDA_NVCC_FLAGS MATCHES "arch")
   INCLUDE_DIRECTORIES(${CUDA_INCLUDE_DIRS})
-  set(HAVE_CUDA 1)
+  SET(HAVE_CUDA 1)
   MESSAGE("   CUDA_NVCC_FLAGS=${CUDA_NVCC_FLAGS}")
 ELSE(QMC_CUDA OR ENABLE_CUDA)
   MESSAGE(STATUS "Disabling CUDA")
@@ -762,38 +738,6 @@ ENDIF(USE_NVTX_API)
 #include qmcpack/src build/src
 INCLUDE_DIRECTORIES( ${PROJECT_SOURCE_DIR}/src ${PROJECT_BINARY_DIR}/src)
 
-######################################################################
-# Manage Einspline library
-# - toolchain file can set
-# - check if libeinspline can be found in $EINSPLINE_HOME/include and
-# $EINSPLINE_HOME/lib
-# - if not found, build it from the source directory in $EINSPLINE_HOME
-# - if everything fails, do not use it
-######################################################################
-SET(HAVE_EINSPLINE 1)
-set(build_einspline 1)
-#if(HAVE_EINSPLINE_EXT)
-#  set(HAVE_EINSPLINE 0)
-#  INCLUDE(${PROJECT_CMAKE}/FindEinspline.cmake)
-#  IF(EINSPLINE_FOUND)
-#    INCLUDE_DIRECTORIES(${EINSPLINE_INCLUDE_DIR})
-#    SET(QMC_UTIL_LIBS ${EINSPLINE_LIBRARIES} ${QMC_UTIL_LIBS})
-#    set(build_einspline 0)
-#  endif()
-#endif()
-if(build_einspline)
-  SUBDIRS(src/einspline)
-  INCLUDE_DIRECTORIES(${PROJECT_BINARY_DIR}/include)
-  SET(QMC_UTIL_LIBS ${QMC_UTIL_LIBS} einspline)
-endif()
-
-IF(BUILD_LMYENGINE_INTERFACE)
-  ADD_DEFINITIONS(-DHAVE_LMY_ENGINE)
-  SET(FORMIC_BINARY_DIR ${qmcpack_BINARY_DIR})
-  SUBDIRS(src/formic/utils)
-  SET(QMC_UTIL_LIBS formic_utils ${QMC_UTIL_LIBS})
-ENDIF(BUILD_LMYENGINE_INTERFACE)
-
 IF (USE_VTUNE_TASKS)
   IF (NOT ENABLE_TIMERS)
     MESSAGE(FATAL_ERROR "USE_VTUNE_TASKS is set, but timers are not enabled.  Set ENABLE_TIMERS=1.")
@@ -903,16 +847,9 @@ MESSAGE(STATUS "Ready to build qmcpack")
 
 ####################################################################
 # add definitions
-#  -DH5_USE_16_API for HDF5
-#  -DMPICH_SKIP_MPICXX : disable C++ binding of MPICH
 ####################################################################
 
-ADD_DEFINITIONS(-DHAVE_CONFIG_H -DH5_USE_16_API -DUSE_REAL_STRUCT_FACTOR -DINLINE_ALL=inline -DADD_)
-
-if(HAVE_MPI)
-  ADD_DEFINITIONS(-DMPICH_SKIP_MPICXX)
-endif()
-#ADD_DEFINITIONS(-DDISABLE_WS_CELL)
+ADD_DEFINITIONS(-DHAVE_CONFIG_H -DUSE_REAL_STRUCT_FACTOR -DADD_)
 
 EXECUTE_PROCESS(COMMAND hostname OUTPUT_VARIABLE QMC_HOSTNAME OUTPUT_STRIP_TRAILING_WHITESPACE)
 CONFIGURE_FILE(${qmcpack_SOURCE_DIR}/src/qmcpack.settings ${qmcpack_BINARY_DIR}/bin/qmcpack.settings)

README.md

@@ -6,11 +6,11 @@
 # Prerequisites
 
  * C++ 14 and C99 capable compilers. 
- * CMake, build utility, http://www.cmake.org
+ * CMake v3.10.0 or later, build utility, http://www.cmake.org
  * BLAS/LAPACK, numerical library. Use platform-optimized libraries.
  * LibXml2, XML parser, http://xmlsoft.org/
  * HDF5, portable I/O library, http://www.hdfgroup.org/HDF5/
- * BOOST, peer-reviewed portable C++ source libraries, http://www.boost.org
+ * BOOST v1.61.0 or newer, peer-reviewed portable C++ source libraries, http://www.boost.org
  * FFTW, FFT library, http://www.fftw.org/
  * MPI, parallel library. Optional, but a near requirement for production calculations.
  * Python3. Older versions are not supported as of January 2020.
@@ -24,18 +24,18 @@ performance and easiest configuration.
 Nightly testing currently includes the following software versions on x86:
 
 * Compilers
-  * GCC 9.2.0, 7.2.0
+  * GCC 9.2.0, 7.3.0
   * Clang/LLVM 9.0.0, 5.0.1
-  * Intel 2019.5, 2018.5 configured to use C++ library from GCC 7.4.0 
+  * Intel 2019.1.0.166, 2019.0.5.281 configured to use C++ library from GCC 7.4.0 
   * PGI 19.4 configured to use C++ library from GCC 8.3.0
 * Boost 1.70.0, 1.65.1
 * HDF5 1.10.5, 1.8.19
 * FFTW 3.3.8, 3.3.4
-* CMake 3.15.4, 3.8.2
+* CMake 3.16.2, 3.10.2
 * MPI
-  * OpenMPI 4.0.1, 2.1.1
-  * Intel MPI 2019.5, 2018.5
-* CUDA 10.1.243
+  * OpenMPI 4.0.2, 3.10.2
+  * Intel MPI 2019.1.0.166, 2019.0.5.281
+* CUDA 10.2.89
 
 # Building with CMake
 

config/build_alcf_theta.sh

@@ -12,11 +12,13 @@ export CRAYPE_LINK_TYPE=dynamic
 TYPE=Release
 Compiler=Intel
 
+CURRENT_FOLDER=`pwd`
+
 for name in real_AoS_legacy real_MP_AoS_legacy cplx_AoS_legacy cplx_MP_AoS_legacy \
             real real_MP cplx cplx_MP
 do
 
-CMAKE_FLAGS="-D CMAKE_BUILD_TYPE=$TYPE"
+CMAKE_FLAGS="-D CMAKE_BUILD_TYPE=$TYPE -D MPIEXEC_EXECUTABLE=/bin/sh -D MPIEXEC_NUMPROC_FLAG=$CURRENT_FOLDER/tests/scripts/aprunhelper.sh"
 
 if [[ $name == *"cplx"* ]]; then
   CMAKE_FLAGS="$CMAKE_FLAGS -D QMC_COMPLEX=1"

config/build_olcf_summit_XL.sh

@@ -26,10 +26,12 @@ export BOOST_ROOT=/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914
 TYPE=Release
 Compiler=XL
 
+CURRENT_FOLDER=`pwd`
+
 for name in offload_real_MP offload_real # offload_cplx offload_cplx_MP
 do
 
-CMAKE_FLAGS="-D CMAKE_BUILD_TYPE=$TYPE -D ENABLE_CUDA=1 -D CUDA_ARCH=sm_70 -DENABLE_MASS=1 -DMASS_ROOT=/sw/summit/xl/16.1.1-5/xlmass/9.1.1"
+CMAKE_FLAGS="-D CMAKE_BUILD_TYPE=$TYPE -D ENABLE_CUDA=1 -D CUDA_ARCH=sm_70 -D ENABLE_MASS=1 -D MASS_ROOT=/sw/summit/xl/16.1.1-5/xlmass/9.1.1 -D MPIEXEC_EXECUTABLE=/bin/sh -D MPIEXEC_NUMPROC_FLAG=$CURRENT_FOLDER/tests/scripts/jsrunhelper.sh"
 if [[ $name == *"cplx"* ]]; then
   CMAKE_FLAGS="$CMAKE_FLAGS -D QMC_COMPLEX=1"
 fi

manual/installation.tex

@@ -321,10 +321,10 @@ FFTW_LIBRARY_DIRS   Specify library directories for FFTW
 \item CTest related
 %
 \begin{shade}
-MPIEXEC                Specify the mpi wrapper, e.g. srun, aprun, mpirun, etc.
+MPIEXEC_EXECUTABLE     Specify the mpi wrapper, e.g. srun, aprun, mpirun, etc.
 MPIEXEC_NUMPROC_FLAG   Specify the number of mpi processes flag,
                        e.g. "-n", "-np", etc.
-MPIEXEC_PREFLAGS     Flags to pass to MPIEXEC directly before the executable to run.   
+MPIEXEC_PREFLAGS       Flags to pass to MPIEXEC_EXECUTABLE directly before the executable to run.
 \end{shade}
 
 \item LLVM/Clang Developer Options\\

nexus/lib/machines.py

@@ -3177,9 +3177,10 @@ class CadesSlurm(Supercomputer):
         c += '#SBATCH -N {}\n'.format(job.nodes)
         c += '#SBATCH --ntasks-per-node={0}\n'.format(job.processes_per_node)
         c += '#SBATCH --cpus-per-task={0}\n'.format(job.threads)
-        c += '#SBATCH --mem=1G\n' # required on Cades
+        c += '#SBATCH --mem=0\n' # required on Cades
         c += '#SBATCH -o '+job.outfile+'\n'
         c += '#SBATCH -e '+job.errfile+'\n'
+        c += '#SBATCH --exclusive\n'
         if job.user_env:
             c += '#SBATCH --export=ALL\n'   # equiv to PBS -V
         else:

nexus/tests/unit/test_machines.py

@@ -1394,9 +1394,10 @@ aprun -n 32 test.x''',
 #SBATCH -N 2
 #SBATCH --ntasks-per-node=36
 #SBATCH --cpus-per-task=1
-#SBATCH --mem=1G
+#SBATCH --mem=0
 #SBATCH -o test.out
 #SBATCH -e test.err
+#SBATCH --exclusive
 #SBATCH --export=ALL
 
 export ENV_VAR=1

src/AFQMC/CMakeLists.txt

@@ -1,4 +1,3 @@
-
 SET(AFQMC_DEBUG 3)
 
 #-------------------------------------------------------------------
@@ -63,13 +62,18 @@ ENDIF(ENABLE_CUDA)
 IF(ENABLE_CUDA)
   CUDA_ADD_LIBRARY(afqmc ${AFQMC_SRCS})
   CUDA_ADD_CUBLAS_TO_TARGET(afqmc)
-  target_link_libraries( afqmc ${CUDA_cusparse_LIBRARY} )
-  target_link_libraries( afqmc ${CUDA_cusolver_LIBRARY} )
-  target_link_libraries( afqmc ${CUDA_curand_LIBRARY} )
+  TARGET_LINK_LIBRARIES( afqmc PRIVATE ${CUDA_cusparse_LIBRARY} )
+  TARGET_LINK_LIBRARIES( afqmc PRIVATE ${CUDA_cusolver_LIBRARY} )
+  TARGET_LINK_LIBRARIES( afqmc PRIVATE ${CUDA_curand_LIBRARY} )
 ELSE(ENABLE_CUDA)
   ADD_LIBRARY(afqmc ${AFQMC_SRCS})
 ENDIF(ENABLE_CUDA)
 
+TARGET_LINK_LIBRARIES(afqmc PUBLIC qmcutil)
+TARGET_LINK_LIBRARIES(afqmc PRIVATE Math::BLAS_LAPACK Boost::boost)
+IF(HAVE_MPI)
+  TARGET_COMPILE_DEFINITIONS(afqmc PRIVATE "USE_MPI")
+ENDIF(HAVE_MPI)
 
 IF (BUILD_UNIT_TESTS)
   INCLUDE_DIRECTORIES(${PROJECT_SOURCE_DIR}/external_codes/catch)

src/AFQMC/Estimators/tests/CMakeLists.txt

@@ -18,7 +18,7 @@ SET(UTEST_DIR ${qmcpack_BINARY_DIR}/tests/afqmc/Estimators)
 EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} -E make_directory "${UTEST_DIR}")
 
 ADD_EXECUTABLE(${UTEST_EXE} test_estimators.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc qmcutil ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc)
 
 LIST(LENGTH AFQMC_UNIT_TEST_INPUTS NUM_PAIRS)
 MATH(EXPR ENDP "${NUM_PAIRS}-1")

src/AFQMC/HamiltonianOperations/tests/CMakeLists.txt

@@ -17,8 +17,7 @@ SET(UTEST_DIR ${qmcpack_BINARY_DIR}/tests/afqmc/HamiltonianOperations)
 EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} -E make_directory "${UTEST_DIR}")
 
 ADD_EXECUTABLE(${UTEST_EXE} test_hamiltonian_operations.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc qmcutil
-                      ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc)
 IF (QMC_COMPLEX)
   SET(UTEST_NAME deterministic-unit_test_${SRC_DIR}_ham_chol_uc_wfn_rhf)
   ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}"

src/AFQMC/Hamiltonians/tests/CMakeLists.txt

@@ -20,8 +20,7 @@ SET(UTEST_DIR ${qmcpack_BINARY_DIR}/tests/afqmc/Hamiltonians)
 EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} -E make_directory "${UTEST_DIR}")
 
 ADD_EXECUTABLE(${UTEST_EXE} test_hamiltonian_factory.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc qmcutil
-                      ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc)
 
 SET(UTEST_NAME deterministic-unit_test_${SRC_DIR}_ham_chol_uc)
 

src/AFQMC/Matrix/tests/CMakeLists.txt

@@ -19,7 +19,7 @@ SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
 
 
 ADD_EXECUTABLE(${UTEST_EXE} test_csr_matrix.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc qmcutil ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc)
 
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
 SET_TESTS_PROPERTIES(${UTEST_NAME} PROPERTIES WORKING_DIRECTORY ${QMCPACK_UNIT_TEST_DIR})

src/AFQMC/Numerics/shm_tests/CMakeLists.txt

@@ -16,7 +16,7 @@ SET(UTEST_EXE test_${SRC_DIR})
 SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
 
 ADD_EXECUTABLE(${UTEST_EXE} test_shm_gemm.cpp) 
-TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcutil ${QMC_UTIL_LIBS} ${MPI_LIBRARY}) 
+TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcutil)
 
 #ADD_TEST(NAME ${UTEST_NAME} COMMAND "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")

src/AFQMC/Numerics/tests/CMakeLists.txt

@@ -25,7 +25,7 @@ SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
   ADD_EXECUTABLE(${UTEST_EXE} test_dense_numerics.cpp test_ma_blas.cpp
 test_sparse_numerics.cpp test_sparse_numerics_native.cpp)
 #ENDIF(ENABLE_CUDA)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc qmcutil ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc Math::BLAS_LAPACK)
 
 #ADD_TEST(NAME ${UTEST_NAME} COMMAND "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")

src/AFQMC/Propagators/tests/CMakeLists.txt

@@ -18,8 +18,7 @@ SET(UTEST_DIR ${qmcpack_BINARY_DIR}/tests/afqmc/Propagators)
 EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} -E make_directory "${UTEST_DIR}")
 
 ADD_EXECUTABLE(${UTEST_EXE} test_propagator_factory.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc qmcutil
-                      ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc)
 
 LIST(LENGTH AFQMC_UNIT_TEST_INPUTS NUM_PAIRS)
 MATH(EXPR ENDP "${NUM_PAIRS}-1")

src/AFQMC/SlaterDeterminantOperations/tests/CMakeLists.txt

@@ -19,8 +19,7 @@ SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
 
 
 ADD_EXECUTABLE(${UTEST_EXE} test_sdet_ops.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc qmcutil
-${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc)
 
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
 SET_TESTS_PROPERTIES(${UTEST_NAME} PROPERTIES WORKING_DIRECTORY ${QMCPACK_UNIT_TEST_DIR})

src/AFQMC/Walkers/tests/CMakeLists.txt

@@ -19,8 +19,7 @@ SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
 
 
 ADD_EXECUTABLE(${UTEST_EXE} test_sharedwset.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc qmcutil
-${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc)
 
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
 SET_TESTS_PROPERTIES(${UTEST_NAME} PROPERTIES WORKING_DIRECTORY ${QMCPACK_UNIT_TEST_DIR})

src/AFQMC/Wavefunctions/tests/CMakeLists.txt

@@ -19,8 +19,7 @@ SET(UTEST_DIR ${qmcpack_BINARY_DIR}/tests/afqmc/Wavefunctions)
 EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} -E make_directory "${UTEST_DIR}")
 
 ADD_EXECUTABLE(${UTEST_EXE} test_wfn_factory.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc qmcutil
-                      ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc)
 LIST(LENGTH AFQMC_UNIT_TEST_INPUTS NUM_PAIRS)
 MATH(EXPR ENDP "${NUM_PAIRS}-1")
 FOREACH(I RANGE 0 ${ENDP} 2)
@@ -38,7 +37,7 @@ ENDFOREACH()
 IF(NOT QMC_COMPLEX)
   SET(UTEST_EXE test_afqmc_phmsd)
   ADD_EXECUTABLE(${UTEST_EXE} test_phmsd.cpp)
-  TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc qmcutil ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+  TARGET_LINK_LIBRARIES(${UTEST_EXE} afqmc)
   SET(UTEST_NAME deterministic-unit_test_afqmc_phmsd)
   ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}"
                 "--hamil ${qmcpack_SOURCE_DIR}/tests/afqmc/Be_sto-3g/qmcpack.h5 "

src/CMakeLists.txt

@@ -120,9 +120,7 @@ if (HAVE_MPI)
   INCLUDE_DIRECTORIES("${qmcpack_SOURCE_DIR}/external_codes/mpi_wrapper")
 endif()
 
-if (HAVE_LIBBOOST)
-    INCLUDE_DIRECTORIES("${qmcpack_SOURCE_DIR}/external_codes/boost_multi")
-endif()
+INCLUDE_DIRECTORIES("${qmcpack_SOURCE_DIR}/external_codes/boost_multi")
 
 
   ####################################
@@ -174,9 +172,14 @@ ENDIF()
   ENDIF(QMC_CUDA)
 
   IF(HAVE_CUDA)
-    TARGET_LINK_LIBRARIES(qmcutil ${CUDA_LIBRARIES})
+    TARGET_LINK_LIBRARIES(qmcutil PRIVATE ${CUDA_LIBRARIES})
   ENDIF(HAVE_CUDA)
 
+  TARGET_LINK_LIBRARIES(qmcutil PUBLIC LibXml2::LibXml2 IO::HDF5 Boost::boost ${QMC_UTIL_LIBS})
+  IF(HAVE_MPI)
+    TARGET_LINK_LIBRARIES(qmcutil PUBLIC MPI::MPI_CXX)
+  ENDIF()
+
   # For unit tests, enable use for the fake RNG
   FUNCTION( USE_FAKE_RNG TARGET )
     TARGET_COMPILE_DEFINITIONS(${TARGET} PRIVATE "USE_FAKE_RNG")
@@ -204,6 +207,7 @@ ENDIF()
     Particle/ParticleSet.cpp
     Particle/VirtualParticleSet.cpp
     Particle/ParticleSet.BC.cpp
+    Particle/DynamicCoordinatesBuilder.cpp
     Particle/MCWalkerConfiguration.cpp
     Particle/createDistanceTableAA.cpp
     Particle/createDistanceTableAB.cpp
@@ -227,36 +231,27 @@ ENDIF()
     ParticleIO/ParticleLayoutIO.cpp
     ParticleIO/XMLParticleIO.cpp
     ParticleIO/ParticleIOUtility.cpp
+    Particle/HDFWalkerOutput.cpp
+    Particle/HDFWalkerInput_0_0.cpp
+    Particle/HDFWalkerInput_0_4.cpp
     )
 
-  IF(HAVE_LIBHDF5)
-    SET(PARTICLEIO ${PARTICLEIO}
-      Particle/HDFWalkerOutput.cpp
-      Particle/HDFWalkerInput_0_0.cpp
-      Particle/HDFWalkerInput_0_4.cpp
-      )
-
-    IF(OHMMS_DIM  MATCHES 3)
-      SET(PARTICLEIO ${PARTICLEIO} ParticleIO/ESHDFParticleParser.cpp)
-    ENDIF(OHMMS_DIM  MATCHES 3)
-
-  ENDIF(HAVE_LIBHDF5)
+  IF(OHMMS_DIM  MATCHES 3)
+    SET(PARTICLEIO ${PARTICLEIO} ParticleIO/ESHDFParticleParser.cpp)
+  ENDIF(OHMMS_DIM  MATCHES 3)
 
   #Need to add this if nothing else but miniapps is compiled
   if(QMC_BUILD_SANDBOX_ONLY)
     set(PARTICLE  ${PARTICLE} QMCWaveFunctions/WaveFunctionComponent.cpp)
   endif()
 
-  ADD_LIBRARY(qmcbase ${PARTICLE}  ${PARTICLEIO})
+  ADD_LIBRARY(qmcbase ${PARTICLE} ${PARTICLEIO})
 
-  TARGET_LINK_LIBRARIES(qmcbase qmcminimalcontainers)
-  #IF(QMC_BUILD_STATIC)
-  #  ADD_LIBRARY(qmcbase STATIC ${PARTICLE}  ${PARTICLEIO})
-  #ELSE(QMC_BUILD_STATIC)
-  #  ADD_LIBRARY(qmcbase SHARED ${PARTICLE}  ${PARTICLEIO})
-  #ENDIF(QMC_BUILD_STATIC)
-  #SUBDIRS(einspline)
+  TARGET_LINK_LIBRARIES(qmcbase PRIVATE qmcminimalcontainers Math::BLAS_LAPACK Math::scalar_vector_functions)
 
+  TARGET_LINK_LIBRARIES(qmcbase PUBLIC qmcutil)
+
+  SUBDIRS(einspline)
   SUBDIRS(MinimalContainers)
 
   if(QMC_BUILD_SANDBOX_ONLY)
@@ -270,6 +265,11 @@ ENDIF()
   SUBDIRS(QMCHamiltonians)
   SUBDIRS(QMCDrivers)
 
+  IF(BUILD_LMYENGINE_INTERFACE)
+    SET(FORMIC_BINARY_DIR ${qmcpack_BINARY_DIR})
+    SUBDIRS(formic/utils)
+  ENDIF(BUILD_LMYENGINE_INTERFACE)
+
   IF(BUILD_AFQMC)
    SUBDIRS(AFQMC)
   ENDIF(BUILD_AFQMC)

src/Estimators/tests/CMakeLists.txt

@@ -20,16 +20,7 @@ SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
 
 SET(SRCS test_accumulator.cpp test_local_energy_est.cpp test_manager.cpp test_trace_manager.cpp)
 
-
-#EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} -E make_directory "${UTEST_DIR}")
-#EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} -E copy "${CMAKE_CURRENT_SOURCE_DIR}/simple.txt" ${UTEST_DIR})
-#EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} -E copy "${qmcpack_SOURCE_DIR}/tests/pseudopotentials_for_tests/C.BFD.xml" ${UTEST_DIR})
-
-
 ADD_EXECUTABLE(${UTEST_EXE} ${SRCS})
-TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcham qmcwfs qmcbase qmcutil qmcdriver ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcdriver)
 
-#ADD_TEST(NAME ${UTEST_NAME} COMMAND "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
-#SET_TESTS_PROPERTIES(${UTEST_NAME} PROPERTIES WORKING_DIRECTORY ${UTEST_DIR})
-

src/Lattice/tests/CMakeLists.txt

@@ -18,7 +18,7 @@ SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
 
 
 ADD_EXECUTABLE(${UTEST_EXE} test_ParticleBConds.cpp test_CrystalLattice.cpp test_LRBreakupParameters.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcutil ${QMC_UTIL_LIBS})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcutil Math::scalar_vector_functions)
 
 #ADD_TEST(NAME ${UTEST_NAME} COMMAND "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")

src/LongRange/tests/CMakeLists.txt

@@ -19,7 +19,7 @@ SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
 
 ADD_EXECUTABLE(${UTEST_EXE} test_lrhandler.cpp test_ewald3d.cpp test_temp.cpp
   test_srcoul.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcbase qmcutil ${QMC_UTIL_LIBS})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcbase)
 
 #ADD_TEST(NAME ${UTEST_NAME} COMMAND "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")

src/Message/tests/CMakeLists.txt

@@ -17,6 +17,6 @@ SET(UTEST_EXE test_${SRC_DIR})
 SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
 
 ADD_EXECUTABLE(${UTEST_EXE} test_communciate.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcutil ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcutil)
 
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")

src/Numerics/tests/CMakeLists.txt

@@ -30,7 +30,7 @@ SET(UTEST_SRCS test_grid_functor.cpp test_stdlib.cpp test_bessel.cpp
 ADD_EXECUTABLE(${UTEST_EXE} ${UTEST_SRCS})
 
 
-TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcutil ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcutil Math::scalar_vector_functions)
 
 #ADD_TEST(NAME ${UTEST_NAME} COMMAND "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")

src/OhmmsApp/ProjectData.cpp

@@ -172,8 +172,6 @@ bool ProjectData::PreviousRoot(std::string& oldroot) const
   }
 }
 
-#if defined(HAVE_LIBXML2)
-
 bool ProjectData::put(xmlNodePtr cur)
 {
   m_cur                  = cur;
@@ -219,5 +217,4 @@ bool ProjectData::put(xmlNodePtr cur)
   return true;
 }
 
-#endif
 } // namespace qmcplusplus

src/OhmmsData/OhmmsElementBase.h

@@ -21,15 +21,7 @@
 #include "config.h"
 #endif
 
-#ifdef HAVE_LIBXML2
 #include "OhmmsData/libxmldefs.h"
-#else /*HAVE_LIBXML2 */
-#ifndef xmlDocPtr
-#define xmlDocPtr void*
-#define xmlNodePtr void*
-#define xmlNsPtr void*
-#endif
-#endif /*HAVE_LIBXML2 */
 
 /**\class OhmmsElementBase
  *\brief Abstract class to provide xml-compatible I/O interfaces for the derived classes.

src/OhmmsData/RecordProperty.h

@@ -231,7 +231,6 @@ struct RecordNamedProperty : public RecordProperty
   bool put(xmlNodePtr cur);
 };
 
-#if defined(HAVE_LIBXML2)
 template<class T>
 bool RecordNamedProperty<T>::put(xmlNodePtr cur)
 {
@@ -248,4 +247,3 @@ bool RecordNamedProperty<T>::put(xmlNodePtr cur)
   return true;
 }
 #endif
-#endif

src/OhmmsData/tests/CMakeLists.txt

@@ -19,7 +19,7 @@ SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
 SET(UTEST_DIR ${qmcpack_BINARY_DIR}/tests/xml_test)
 
 ADD_EXECUTABLE(${UTEST_EXE} test_xml.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcutil ${QMC_UTIL_LIBS} )
+TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcutil)
 
 
 EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} -E make_directory "${UTEST_DIR}")

src/OhmmsPETE/tests/CMakeLists.txt

@@ -19,7 +19,5 @@ SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
 
 
 ADD_EXECUTABLE(${UTEST_EXE} test_vector.cpp test_matrix.cpp test_tinyvector.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} ${QMC_UTIL_LIBS})
 
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
-

src/OhmmsSoA/tests/CMakeLists.txt

@@ -19,7 +19,7 @@ SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
 
 
 ADD_EXECUTABLE(${UTEST_EXE} test_vector_soa.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcutil ${QMC_UTIL_LIBS})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcutil)
 
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
 

src/OpenMP/tests/CMakeLists.txt

@@ -20,7 +20,6 @@ SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
 
 
 ADD_EXECUTABLE(${UTEST_EXE} test_vector.cpp test_math.cpp test_deep_copy.cpp test_class_member.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} ${QMC_UTIL_LIBS})
 
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
 

src/Optimize/tests/CMakeLists.txt

@@ -16,6 +16,6 @@ SET(UTEST_EXE test_${SRC_DIR})
 SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
 
 ADD_EXECUTABLE(${UTEST_EXE} test_variable_set.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcutil ${QMC_UTIL_LIBS})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcutil)
 
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")

src/Particle/DynamicCoordinates.h

@@ -0,0 +1,64 @@
+//////////////////////////////////////////////////////////////////////////////////////
+// This file is distributed under the University of Illinois/NCSA Open Source License.
+// See LICENSE file in top directory for details.
+//
+// Copyright (c) 2020 QMCPACK developers.
+//
+// File developed by: Ye Luo, yeluo@anl.gov, Argonne National Laboratory
+//
+// File created by: Ye Luo, yeluo@anl.gov, Argonne National Laboratory
+//////////////////////////////////////////////////////////////////////////////////////
+
+
+/** @file DynamicCoordinates.h
+ */
+#ifndef QMCPLUSPLUS_DYNAMICCOORDINATES_H
+#define QMCPLUSPLUS_DYNAMICCOORDINATES_H
+
+#include <memory>
+#include "Configuration.h"
+#include "OhmmsSoA/Container.h"
+
+namespace qmcplusplus
+{
+/** enumerator for DynamicCoordinates kinds
+ */
+enum class DynamicCoordinateKind
+{
+  DC_POS, // SoA positions
+  DC_POS_OFFLOAD, // SoA positions with OpenMP offload
+};
+
+/** quantum variables of all the particles
+ */
+class DynamicCoordinates
+{
+public:
+  using RealType = QMCTraits::RealType;
+  using PosType = QMCTraits::PosType;
+  using ParticlePos_t = PtclOnLatticeTraits::ParticlePos_t;
+  using PosVectorSoa = VectorSoaContainer<RealType, QMCTraits::DIM>;
+
+  DynamicCoordinates(const DynamicCoordinateKind kind_in) : variable_kind_(kind_in) {}
+
+  DynamicCoordinateKind getKindvariable() const { return variable_kind_; }
+
+  virtual ~DynamicCoordinates() = default;
+
+  virtual std::unique_ptr<DynamicCoordinates> makeClone() = 0;
+
+  virtual void resize(size_t n) = 0;
+  virtual size_t size() = 0;
+
+  virtual void setAllParticlePos(const ParticlePos_t& R) = 0;
+  virtual void setOneParticlePos(const PosType& pos, size_t iat) = 0;
+
+  virtual const PosVectorSoa& getAllParticlePos() const = 0;
+  virtual PosType getOneParticlePos(size_t iat) const = 0;
+
+  virtual void donePbyP() { }
+protected:
+  const DynamicCoordinateKind variable_kind_;
+};
+} // namespace qmcplusplus
+#endif

src/Particle/DynamicCoordinatesBuilder.cpp

@@ -0,0 +1,34 @@
+//////////////////////////////////////////////////////////////////////////////////////
+// This file is distributed under the University of Illinois/NCSA Open Source License.
+// See LICENSE file in top directory for details.
+//
+// Copyright (c) 2020 QMCPACK developers.
+//
+// File developed by: Ye Luo, yeluo@anl.gov, Argonne National Laboratory
+//
+// File created by: Ye Luo, yeluo@anl.gov, Argonne National Laboratory
+//////////////////////////////////////////////////////////////////////////////////////
+
+
+#include "DynamicCoordinatesBuilder.h"
+#include "Particle/RealSpacePositions.h"
+#include "Particle/RealSpacePositionsOffload.h"
+
+namespace qmcplusplus
+{
+/** create DynamicCoordinates based on kind
+ */
+std::unique_ptr<DynamicCoordinates> createDynamicCoordinates(const DynamicCoordinateKind kind)
+{
+  if (kind == DynamicCoordinateKind::DC_POS)
+    return std::make_unique<RealSpacePositions>();
+#if defined(ENABLE_OFFLOAD)
+  else if(kind==DynamicCoordinateKind::DC_POS_OFFLOAD)
+    return std::make_unique<RealSpacePositionsOffload>();
+#endif
+  else
+    APP_ABORT("DynamicCoordinatesBuilder::createDynamicCoordinates unknown DynamicCoordinateKind");
+  // dummy return
+  return std::unique_ptr<RealSpacePositions>();
+}
+} // namespace qmcplusplus

src/Particle/DynamicCoordinatesBuilder.h

@@ -0,0 +1,26 @@
+//////////////////////////////////////////////////////////////////////////////////////
+// This file is distributed under the University of Illinois/NCSA Open Source License.
+// See LICENSE file in top directory for details.
+//
+// Copyright (c) 2020 QMCPACK developers.
+//
+// File developed by: Ye Luo, yeluo@anl.gov, Argonne National Laboratory
+//
+// File created by: Ye Luo, yeluo@anl.gov, Argonne National Laboratory
+//////////////////////////////////////////////////////////////////////////////////////
+
+
+/** @file DynamicCoordinatesBuilder.h
+ */
+#ifndef QMCPLUSPLUS_QUANTUM_VARIABLE_BUILDER_H
+#define QMCPLUSPLUS_QUANTUM_VARIABLE_BUILDER_H
+
+#include "Particle/DynamicCoordinates.h"
+
+namespace qmcplusplus
+{
+/** create DynamicCoordinates based on kind
+ */
+std::unique_ptr<DynamicCoordinates> createDynamicCoordinates(const DynamicCoordinateKind kind = DynamicCoordinateKind::DC_POS);
+} // namespace qmcplusplus
+#endif

src/Particle/ParticleSet.BC.cpp

@@ -129,7 +129,7 @@ void ParticleSet::createSK()
   for (int iat = 0; iat < GroupID.size(); iat++)
     Mass[iat] = mySpecies(massind, GroupID[iat]);
 
-  RSoA = R;
+  coordinates_->setAllParticlePos(R);
 }
 
 void ParticleSet::turnOnPerParticleSK()

src/Particle/ParticleSet.cpp

@@ -20,6 +20,7 @@
 #include <numeric>
 #include <iomanip>
 #include "Particle/ParticleSet.h"
+#include "Particle/DynamicCoordinatesBuilder.h"
 #include "Particle/DistanceTableData.h"
 #include "Particle/createDistanceTable.h"
 #include "LongRange/StructFact.h"
@@ -55,7 +56,8 @@ ParticleSet::ParticleSet()
       Properties(0, 0, 1, WP::MAXPROPERTIES),
       myTwist(0.0),
       ParentName("0"),
-      TotalNum(0)
+      TotalNum(0),
+      coordinates_(std::move(createDynamicCoordinates()))
 {
   initPropertyList();
   setup_timers(myTimers, PSTimerNames, timer_level_fine);
@@ -70,8 +72,9 @@ ParticleSet::ParticleSet(const ParticleSet& p)
       SK(0),
       Properties(p.Properties),
       myTwist(0.0),
-      ParentName(p.parentName())
-{
+      ParentName(p.parentName()),
+      coordinates_(std::move(p.coordinates_->makeClone()))
+  {
   set_quantum_domain(p.quantum_domain);
   assign(p); //only the base is copied, assumes that other properties are not assignable
   //need explicit copy:
@@ -100,9 +103,8 @@ ParticleSet::ParticleSet(const ParticleSet& p)
   setup_timers(myTimers, PSTimerNames, timer_level_fine);
   myTwist = p.myTwist;
 
-  RSoA = p.RSoA;
-  G    = p.G;
-  L    = p.L;
+  G = p.G;
+  L = p.L;
 }
 
 ParticleSet::~ParticleSet()
@@ -379,7 +381,7 @@ void ParticleSet::update(bool skipSK)
 {
   ScopedTimer update_scope(myTimers[PS_update]);
 
-  RSoA.copyIn(R);
+  coordinates_->setAllParticlePos(R);
   for (int i = 0; i < DistTables.size(); i++)
     DistTables[i]->evaluate(*this);
   if (!skipSK && SK)
@@ -509,7 +511,7 @@ bool ParticleSet::makeMoveAllParticles(const Walker_t& awalker, const ParticlePo
     for (int iat = 0; iat < deltaR.size(); ++iat)
       R[iat] = awalker.R[iat] + dt * deltaR[iat];
   }
-  RSoA.copyIn(R);
+  coordinates_->setAllParticlePos(R);
   for (int i = 0; i < DistTables.size(); i++)
     DistTables[i]->evaluate(*this);
   if (SK)
@@ -541,7 +543,7 @@ bool ParticleSet::makeMoveAllParticles(const Walker_t& awalker,
     for (int iat = 0; iat < deltaR.size(); ++iat)
       R[iat] = awalker.R[iat] + dt[iat] * deltaR[iat];
   }
-  RSoA.copyIn(R);
+  coordinates_->setAllParticlePos(R);
   for (int i = 0; i < DistTables.size(); i++)
     DistTables[i]->evaluate(*this);
   if (SK)
@@ -581,7 +583,7 @@ bool ParticleSet::makeMoveAllParticlesWithDrift(const Walker_t& awalker,
     for (int iat = 0; iat < deltaR.size(); ++iat)
       R[iat] = awalker.R[iat] + dt * deltaR[iat] + drift[iat];
   }
-  RSoA.copyIn(R);
+  coordinates_->setAllParticlePos(R);
   for (int i = 0; i < DistTables.size(); i++)
     DistTables[i]->evaluate(*this);
   if (SK)
@@ -614,7 +616,7 @@ bool ParticleSet::makeMoveAllParticlesWithDrift(const Walker_t& awalker,
     for (int iat = 0; iat < deltaR.size(); ++iat)
       R[iat] = awalker.R[iat] + dt[iat] * deltaR[iat] + drift[iat];
   }
-  RSoA.copyIn(R);
+  coordinates_->setAllParticlePos(R);
 
   for (int i = 0; i < DistTables.size(); i++)
     DistTables[i]->evaluate(*this);
@@ -643,7 +645,7 @@ void ParticleSet::acceptMove(Index_t iat, bool partial_table_update)
       SK->acceptMove(iat, GroupID[iat], R[iat]);
 
     R[iat]     = activePos;
-    RSoA(iat)  = activePos;
+    coordinates_->setOneParticlePos(activePos, iat);
     spins[iat] = activeSpinVal;
     activePtcl = -1;
   }
@@ -664,6 +666,7 @@ void ParticleSet::flex_donePbyP(const RefVector<ParticleSet>& P_list)
 void ParticleSet::donePbyP()
 {
   ScopedTimer donePbyP_scope(myTimers[PS_donePbyP]);
+  coordinates_->donePbyP();
   if (SK && !SK->DoUpdate)
     SK->UpdateAllPart(*this);
   activePtcl = -1;
@@ -680,7 +683,7 @@ void ParticleSet::makeVirtualMoves(const SingleParticlePos_t& newpos)
 void ParticleSet::loadWalker(Walker_t& awalker, bool pbyp)
 {
   R = awalker.R;
-  RSoA.copyIn(R);
+  coordinates_->setAllParticlePos(R);
 #if !defined(SOA_MEMORY_OPTIMIZED)
   G = awalker.G;
   L = awalker.L;

src/Particle/ParticleSet.h

@@ -21,6 +21,7 @@
 
 #include <Configuration.h>
 #include <ParticleTags.h>
+#include <Particle/DynamicCoordinates.h>
 #include <Particle/Walker.h>
 #include <Utilities/SpeciesSet.h>
 #include <Utilities/PooledData.h>
@@ -108,8 +109,6 @@ public:
   ParticleIndex_t GroupID;
   ///Position
   ParticlePos_t R;
-  ///SoA copy of R
-  VectorSoaContainer<RealType, DIM> RSoA;
   ///internal spin variables for dynamical spin calculations
   ParticleScalar_t spins;
   ///gradients of the particles
@@ -294,6 +293,9 @@ public:
     }
   }
 
+  inline const DynamicCoordinates& getCoordinates() const { return *coordinates_; }
+  inline void setCoordinates(const ParticlePos_t& R) { return coordinates_->setAllParticlePos(R); }
+
   //inline RealType getTotalWeight() const { return EnsembleProperty.Weight; }
 
   void resetGroups();
@@ -539,7 +541,7 @@ public:
     Z.resize(numPtcl);
     IndirectID.resize(numPtcl);
 
-    RSoA.resize(numPtcl);
+    coordinates_->resize(numPtcl);
   }
 
   inline void clear()
@@ -559,7 +561,7 @@ public:
     Z.clear();
     IndirectID.clear();
 
-    RSoA.resize(0);
+    coordinates_->resize(0);
   }
 
   inline void assign(const ParticleSet& ptclin)
@@ -677,6 +679,8 @@ protected:
 
   ///array to handle a group of distinct particles per species
   ParticleIndex_t SubPtcl;
+  ///internal representation of R. It can be an SoA copy of R
+  std::unique_ptr<DynamicCoordinates> coordinates_;
 
   /** compute temporal DistTables and SK for a new particle position
    *

src/Particle/RealSpacePositions.h

@@ -0,0 +1,54 @@
+//////////////////////////////////////////////////////////////////////////////////////
+// This file is distributed under the University of Illinois/NCSA Open Source License.
+// See LICENSE file in top directory for details.
+//
+// Copyright (c) 2020 QMCPACK developers.
+//
+// File developed by: Ye Luo, yeluo@anl.gov, Argonne National Laboratory
+//
+// File created by: Ye Luo, yeluo@anl.gov, Argonne National Laboratory
+//////////////////////////////////////////////////////////////////////////////////////
+
+
+/** @file RealSpacePostions.h
+ */
+#ifndef QMCPLUSPLUS_REALSPACE_POSITIONS_H
+#define QMCPLUSPLUS_REALSPACE_POSITIONS_H
+
+#include "Particle/DynamicCoordinates.h"
+#include "OhmmsSoA/Container.h"
+
+namespace qmcplusplus
+{
+/** Introduced to handle virtual moves and ratio computations, e.g. for non-local PP evaluations.
+   */
+class RealSpacePositions : public DynamicCoordinates
+{
+public:
+  using ParticlePos_t = PtclOnLatticeTraits::ParticlePos_t;
+  using RealType      = QMCTraits::RealType;
+  using PosType       = QMCTraits::PosType;
+
+  RealSpacePositions() : DynamicCoordinates(DynamicCoordinateKind::DC_POS) {}
+
+  std::unique_ptr<DynamicCoordinates> makeClone() override { return std::make_unique<RealSpacePositions>(*this); }
+
+  void resize(size_t n) override { RSoA.resize(n); }
+  size_t size() override { return RSoA.size(); }
+
+  void setAllParticlePos(const ParticlePos_t& R) override
+  {
+    resize(R.size());
+    RSoA.copyIn(R);
+  }
+  void setOneParticlePos(const PosType& pos, size_t iat) override { RSoA(iat) = pos; }
+
+  const PosVectorSoa& getAllParticlePos() const override { return RSoA; }
+  PosType getOneParticlePos(size_t iat) const override { return RSoA[iat]; }
+
+private:
+  ///particle positions in SoA layout
+  PosVectorSoa RSoA;
+};
+} // namespace qmcplusplus
+#endif

src/Particle/RealSpacePositionsOffload.h

@@ -0,0 +1,93 @@
+//////////////////////////////////////////////////////////////////////////////////////
+// This file is distributed under the University of Illinois/NCSA Open Source License.
+// See LICENSE file in top directory for details.
+//
+// Copyright (c) 2020 QMCPACK developers.
+//
+// File developed by: Ye Luo, yeluo@anl.gov, Argonne National Laboratory
+//
+// File created by: Ye Luo, yeluo@anl.gov, Argonne National Laboratory
+//////////////////////////////////////////////////////////////////////////////////////
+
+
+/** @file RealSpacePostionsOffload.h
+ */
+#ifndef QMCPLUSPLUS_REALSPACE_POSITIONS_OFFLOAD_H
+#define QMCPLUSPLUS_REALSPACE_POSITIONS_OFFLOAD_H
+
+#include "Particle/DynamicCoordinates.h"
+#include "OhmmsSoA/Container.h"
+#include "OpenMP/OMPallocator.hpp"
+#include "Platforms/PinnedAllocator.h"
+
+namespace qmcplusplus
+{
+/** Introduced to handle virtual moves and ratio computations, e.g. for non-local PP evaluations.
+   */
+class RealSpacePositionsOffload : public DynamicCoordinates
+{
+public:
+  RealSpacePositionsOffload() : DynamicCoordinates(DynamicCoordinateKind::DC_POS_OFFLOAD) {}
+  RealSpacePositionsOffload(const RealSpacePositionsOffload& in)
+    : DynamicCoordinates(DynamicCoordinateKind::DC_POS_OFFLOAD), RSoA(in.RSoA)
+  {
+    RSoA_hostview.attachReference(RSoA.size(), RSoA.capacity(), RSoA.data());
+    updateH2D();
+  }
+
+  std::unique_ptr<DynamicCoordinates> makeClone() override { return std::make_unique<RealSpacePositionsOffload>(*this); }
+
+  void resize(size_t n) override
+  {
+    RSoA.resize(n);
+    RSoA_hostview.attachReference(RSoA.size(), RSoA.capacity(), RSoA.data());
+  }
+
+  size_t size() override { return RSoA_hostview.size(); }
+
+  void setAllParticlePos(const ParticlePos_t& R) override
+  {
+    resize(R.size());
+    RSoA_hostview.copyIn(R);
+    updateH2D();
+  }
+
+  void setOneParticlePos(const PosType& pos, size_t iat) override
+  {
+    RSoA_hostview(iat) = pos;
+    /* This was too slow due to overhead.
+    RealType x     = pos[0];
+    RealType y     = pos[1];
+    RealType z     = pos[2];
+    RealType* data = RSoA.data();
+    size_t offset  = RSoA.capacity();
+
+    PRAGMA_OFFLOAD("omp target map(to : x, y, z, iat)")
+    {
+      data[iat]              = x;
+      data[iat + offset]     = y;
+      data[iat + offset * 2] = z;
+    }
+    */
+  }
+
+  const PosVectorSoa& getAllParticlePos() const override { return RSoA_hostview; }
+  PosType getOneParticlePos(size_t iat) const override { return RSoA_hostview[iat]; }
+
+  void donePbyP() override { updateH2D(); }
+
+private:
+  ///particle positions in SoA layout
+  VectorSoaContainer<RealType, QMCTraits::DIM, OMPallocator<RealType, PinnedAlignedAllocator<RealType>>> RSoA;
+
+  ///host view of RSoA
+  PosVectorSoa RSoA_hostview;
+
+  void updateH2D()
+  {
+    RealType* data = RSoA.data();
+    PRAGMA_OFFLOAD("omp target update to(data[0:RSoA.capacity()*QMCTraits::DIM])")
+  }
+};
+} // namespace qmcplusplus
+#endif

src/Particle/Reptile.h

@@ -67,7 +67,6 @@ public:
         headindex(0),
         prophead(0) //, r2prop(0.0), r2accept(0.0),tau(0.0)
   {
-    w = W;
     Action.resize(3);
     Action[0] = w.addProperty("ActionBackward");
     Action[1] = w.addProperty("ActionForward");

src/Particle/SoaDistanceTableAA.h

@@ -80,10 +80,9 @@ struct SoaDistanceTableAA : public DTD_BConds<T, D, SC>, public DistanceTableDat
   inline void evaluate(ParticleSet& P)
   {
     constexpr T BigR = std::numeric_limits<T>::max();
-    //P.RSoA.copyIn(P.R);
     for (int iat = 0; iat < N_targets; ++iat)
     {
-      DTD_BConds<T, D, SC>::computeDistances(P.R[iat], P.RSoA, distances_[iat].data(), displacements_[iat], 0, N_targets,
+      DTD_BConds<T, D, SC>::computeDistances(P.R[iat], P.getCoordinates().getAllParticlePos(), distances_[iat].data(), displacements_[iat], 0, N_targets,
                                              iat);
       distances_[iat][iat] = BigR; //assign big distance
     }
@@ -92,12 +91,12 @@ struct SoaDistanceTableAA : public DTD_BConds<T, D, SC>, public DistanceTableDat
   ///evaluate the temporary pair relations
   inline void move(const ParticleSet& P, const PosType& rnew, const IndexType iat, bool prepare_old)
   {
-    DTD_BConds<T, D, SC>::computeDistances(rnew, P.RSoA, temp_r_.data(), temp_dr_, 0, N_targets, P.activePtcl);
+    DTD_BConds<T, D, SC>::computeDistances(rnew, P.getCoordinates().getAllParticlePos(), temp_r_.data(), temp_dr_, 0, N_targets, P.activePtcl);
     // set up old_r_ and old_dr_ for moves may get accepted.
     if (prepare_old)
     {
       //recompute from scratch
-      DTD_BConds<T, D, SC>::computeDistances(P.R[iat], P.RSoA, old_r_.data(), old_dr_, 0, N_targets, iat);
+      DTD_BConds<T, D, SC>::computeDistances(P.R[iat], P.getCoordinates().getAllParticlePos(), old_r_.data(), old_dr_, 0, N_targets, iat);
       old_r_[iat] = std::numeric_limits<T>::max(); //assign a big number
 
       // If the full table is not ready all the time, overwrite the current value.

src/Particle/SoaDistanceTableBA.h

@@ -66,7 +66,7 @@ struct SoaDistanceTableBA : public DTD_BConds<T, D, SC>, public DistanceTableDat
 
       //be aware of the sign of Displacement
       for (int iat = 0; iat < N_targets; ++iat)
-        DTD_BConds<T, D, SC>::computeDistances(P.R[iat], Origin->RSoA, distances_[iat].data(), displacements_[iat], first,
+        DTD_BConds<T, D, SC>::computeDistances(P.R[iat], Origin->getCoordinates().getAllParticlePos(), distances_[iat].data(), displacements_[iat], first,
                                                last);
     }
   }
@@ -74,11 +74,11 @@ struct SoaDistanceTableBA : public DTD_BConds<T, D, SC>, public DistanceTableDat
   ///evaluate the temporary pair relations
   inline void move(const ParticleSet& P, const PosType& rnew, const IndexType iat, bool prepare_old)
   {
-    DTD_BConds<T, D, SC>::computeDistances(rnew, Origin->RSoA, temp_r_.data(), temp_dr_, 0, N_sources);
+    DTD_BConds<T, D, SC>::computeDistances(rnew, Origin->getCoordinates().getAllParticlePos(), temp_r_.data(), temp_dr_, 0, N_sources);
     // If the full table is not ready all the time, overwrite the current value.
     // If this step is missing, DT values can be undefined in case a move is rejected.
     if (!need_full_table_)
-      DTD_BConds<T, D, SC>::computeDistances(P.R[iat], Origin->RSoA, distances_[iat].data(), displacements_[iat], 0,
+      DTD_BConds<T, D, SC>::computeDistances(P.R[iat], Origin->getCoordinates().getAllParticlePos(), distances_[iat].data(), displacements_[iat], 0,
                                              N_sources);
   }
 

src/Particle/VirtualParticleSet.cpp

@@ -30,7 +30,7 @@ VirtualParticleSet::VirtualParticleSet(const ParticleSet& p, int nptcl) : refPS(
   Lattice  = p.Lattice;
   TotalNum = nptcl;
   R.resize(nptcl);
-  RSoA.resize(nptcl);
+  coordinates_->resize(nptcl);
 
   //create distancetables
   for (int i = 0; i < refPS.getNumDistTables(); ++i)

src/Particle/tests/CMakeLists.txt

@@ -17,7 +17,6 @@ SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
 
 
 ADD_EXECUTABLE(${UTEST_EXE} test_particle.cpp test_distance_table.cpp test_walker.cpp test_particle_pool.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcbase qmcutil ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcbase Math::scalar_vector_functions)
 
-#ADD_TEST(NAME ${UTEST_NAME} COMMAND "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")

src/Particle/tests/test_particle.cpp

@@ -86,9 +86,9 @@ TEST_CASE("symmetric_distance_table OpenBC", "[particle]")
   source.R[1][2] = 3.2;
 
   source.update();
-  /// make sure RSoA is updated no matter SoA or AoS.
-  REQUIRE(source.RSoA[0][1] == Approx(1.0));
-  REQUIRE(source.RSoA[1][2] == Approx(3.2));
+  /// make sure getCoordinates().getAllParticlePos() is updated no matter SoA or AoS.
+  REQUIRE(source.getCoordinates().getAllParticlePos()[0][1] == Approx(1.0));
+  REQUIRE(source.getCoordinates().getAllParticlePos()[1][2] == Approx(3.2));
 
   const int TableID = source.addTable(source, DT_SOA);
   source.update();

src/ParticleBase/tests/CMakeLists.txt

@@ -19,7 +19,7 @@ SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
 
 ADD_EXECUTABLE(${UTEST_EXE} test_particle_attrib.cpp test_random_seq.cpp test_attrib_ops.cpp)
 USE_FAKE_RNG(${UTEST_EXE})
-TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcbase qmcutil qmcfakerng ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcbase qmcfakerng)
 
 #ADD_TEST(NAME ${UTEST_NAME} COMMAND "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")

src/ParticleIO/tests/CMakeLists.txt

@@ -17,7 +17,6 @@ SET(UTEST_EXE test_${SRC_DIR})
 SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
 
 ADD_EXECUTABLE(${UTEST_EXE} test_xml_io.cpp test_lattice_parser.cpp test_xml_mass.cpp)
-TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcbase qmcutil ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcbase)
 
-#ADD_TEST(NAME ${UTEST_NAME} COMMAND "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")

src/QMCApp/CMakeLists.txt

@@ -16,12 +16,6 @@
 #//////////////////////////////////////////////////////////////////////////////////////
 
 
-PROJECT(qmcapp)
-# formic dependencies
-#FIND_PATH(LMYENGINE_INCLUDE_DIR formic/fqmc/engine/var_dependencies.h)
-#MESSAGE("-- LMYENGINE_INCLUDE_DIR=${LMYENGINE_INCLUDE_DIR}")
-#INCLUDE_DIRECTORIES(${LMYENGINE_INCLUDE_DIR})
-
 ####################################
 # create libqmc
 ####################################
@@ -32,48 +26,27 @@ SET(QMCAPPDIR
     CoupledMC.cpp
 )
 ADD_LIBRARY(qmc ${QMCAPPDIR})
+TARGET_LINK_LIBRARIES(qmc PUBLIC qmcutil)
 
 ####################################
-# list of QMC applications
+# QMCPACK application
 ####################################
-SET(QMCSRCS qmcpack)
-
-FOREACH(p ${QMCSRCS})
-
-# QMCPACK historically called qmcapp, and still built from qmcapp.cpp
-# Override source name for executable in this case, otherwise
-# executablename.cpp used for main()
- IF ( ${p} STREQUAL "qmcpack" )
-  ADD_EXECUTABLE(${p} qmcapp.cpp)
-ELSE()
-  ADD_EXECUTABLE(${p} ${p}.cpp)
-ENDIF()
+ADD_EXECUTABLE(qmcpack qmcapp.cpp)
 
 # Clean up git-rev-tmp.h after the build is finished
 ADD_CUSTOM_COMMAND(TARGET qmcpack
                    POST_BUILD
                    COMMAND ${CMAKE_COMMAND} -E remove ${GITREV_TMP})
 
-  TARGET_LINK_LIBRARIES(${p} qmc qmcdriver qmcham qmcwfs qmcbase qmcutil)
+TARGET_LINK_LIBRARIES(qmcpack qmc qmcdriver)
 
-  IF(BUILD_AFQMC)
-    TARGET_LINK_LIBRARIES(${p} afqmc)
-  ENDIF(BUILD_AFQMC)
+IF(BUILD_AFQMC)
+  TARGET_LINK_LIBRARIES(qmcpack afqmc)
+ENDIF(BUILD_AFQMC)
 
-  IF(BUILD_FCIQMC)
-    TARGET_LINK_LIBRARIES(${p} fciqmc)
-  ENDIF(BUILD_FCIQMC)
-
-  FOREACH(l ${QMC_UTIL_LIBS})
-    TARGET_LINK_LIBRARIES(${p} ${l})
-  ENDFOREACH(l ${QMC_UTIL_LIBS})
-
-  IF(MPI_LIBRARY)
-    TARGET_LINK_LIBRARIES(${p} ${MPI_LIBRARY})
-  ENDIF(MPI_LIBRARY)
-
-ENDFOREACH(p ${QMCSRCS})
+IF(BUILD_FCIQMC)
+  TARGET_LINK_LIBRARIES(qmcpack fciqmc)
+ENDIF(BUILD_FCIQMC)
 
 INSTALL(TARGETS qmcpack
         RUNTIME DESTINATION bin)
-

src/QMCApp/QMCMainState.cpp

@@ -28,9 +28,9 @@
 #include "QMCDrivers/VMC/VMCFactory.h"
 #include "QMCDrivers/DMC/DMCFactory.h"
 #include "QMCDrivers/RMC/RMCFactory.h"
-#include "QMCDrivers/QMCOptimize.h"
-#include "QMCDrivers/QMCFixedSampleLinearOptimize.h"
-#include "QMCDrivers/QMCCorrelatedSamplingLinearOptimize.h"
+#include "QMCDrivers/WFOpt/QMCOptimize.h"
+#include "QMCDrivers/WFOpt/QMCFixedSampleLinearOptimize.h"
+#include "QMCDrivers/WFOpt/QMCCorrelatedSamplingLinearOptimize.h"
 #include "QMCDrivers/WaveFunctionTester.h"
 #include <queue>
 #include "OhmmsData/AttributeSet.h"

src/QMCApp/tests/CMakeLists.txt

@@ -20,6 +20,6 @@ SET(UTEST_NAME deterministic-unit_test_${SRC_DIR})
 
 ADD_EXECUTABLE(${UTEST_EXE} test_project_data.cpp test_rng_control.cpp)
 
-TARGET_LINK_LIBRARIES(${UTEST_EXE} qmc qmcutil ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} qmc)
 
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")

src/QMCDrivers/CMakeLists.txt

@@ -30,13 +30,13 @@ SET(QMCDRIVERS
   QMCDriver.cpp
   QMCDriverInput.cpp
   QMCDriverNew.cpp
-  QMCOptimize.cpp
-  QMCLinearOptimize.cpp
-  QMCCorrelatedSamplingLinearOptimize.cpp
-  QMCFixedSampleLinearOptimize.cpp
+  WFOpt/QMCOptimize.cpp
+  WFOpt/QMCLinearOptimize.cpp
+  WFOpt/QMCCorrelatedSamplingLinearOptimize.cpp
+  WFOpt/QMCFixedSampleLinearOptimize.cpp
   Optimizers/DescentEngine.cpp
   Optimizers/HybridEngine.cpp
-  QMCCostFunctionBase.cpp
+  WFOpt/QMCCostFunctionBase.cpp
   WaveFunctionTester.cpp
   WalkerControlBase.cpp
   CloneManager.cpp
@@ -81,14 +81,14 @@ IF(QMC_CUDA)
    SET(QMCDRIVERS ${QMCDRIVERS}
    VMC/VMC_CUDA.cpp
    DMC/DMC_CUDA.cpp
-   QMCCostFunctionCUDA.cpp)
+   WFOpt/QMCCostFunctionCUDA.cpp)
 ENDIF(QMC_CUDA)
 
 SET(QMCDRIVERS ${QMCDRIVERS} 
     VMC/VMC.cpp
     VMC/VMCBatched.cpp
     VMC/VMCLinearOpt.cpp
-    QMCCostFunction.cpp)
+    WFOpt/QMCCostFunction.cpp)
 
 IF(HAVE_MPI) 
   SET(QMCDRIVERS ${QMCDRIVERS} 
@@ -109,11 +109,21 @@ SET(QMCDRIVERS ${QMCDRIVERS}
 
 
 ####################################
-# create libqmc 
+# create libqmcdriver
 ####################################
 ADD_LIBRARY(qmcdriver ${QMCDRIVERS})
 ADD_LIBRARY(qmcdriver_unit ${QMCDRIVERS})
 USE_FAKE_RNG(qmcdriver_unit)
+
+TARGET_LINK_LIBRARIES(qmcdriver PUBLIC qmcham)
+TARGET_LINK_LIBRARIES(qmcdriver_unit PUBLIC qmcham_unit)
+TARGET_LINK_LIBRARIES(qmcdriver PRIVATE Math::BLAS_LAPACK)
+TARGET_LINK_LIBRARIES(qmcdriver_unit PRIVATE Math::BLAS_LAPACK)
+IF(BUILD_LMYENGINE_INTERFACE)
+  TARGET_LINK_LIBRARIES(qmcdriver PRIVATE formic_utils)
+  TARGET_LINK_LIBRARIES(qmcdriver_unit PRIVATE formic_utils)
+ENDIF()
+
 #IF(QMC_BUILD_STATIC)
 #  ADD_LIBRARY(qmcdriver STATIC ${QMCDRIVERS})
 #ELSE(QMC_BUILD_STATIC)

src/QMCDrivers/LMYEngineInterface/LMYE_QMCCostFunction.cpp

@@ -10,7 +10,7 @@
 // File created by: Luning Zhao, zhaoln@berkeley.edu, University of California, Berkeley
 //////////////////////////////////////////////////////////////////////////////////////
 
-#include "QMCDrivers/QMCCostFunction.h"
+#include "QMCDrivers/WFOpt/QMCCostFunction.h"
 #include "Particle/MCWalkerConfiguration.h"
 #include "QMCWaveFunctions/TrialWaveFunction.h"
 #include "Message/CommOperators.h"

src/QMCDrivers/MCPopulation.h

@@ -108,7 +108,6 @@ public:
   MCPopulation(MCPopulation&)  = delete;
   MCPopulation& operator=(MCPopulation&) = delete;
   MCPopulation(MCPopulation&&) = default;
-  MCPopulation& operator=(MCPopulation&&) = default;
 
   /** @ingroup PopulationControl
    *

src/QMCDrivers/QMCDriverFactory.cpp

@@ -34,9 +34,9 @@
 #include "QMCDrivers/DMC/DMCFactory.h"
 #include "QMCDrivers/DMC/DMCFactoryNew.h"
 #include "QMCDrivers/RMC/RMCFactory.h"
-#include "QMCDrivers/QMCOptimize.h"
-#include "QMCDrivers/QMCFixedSampleLinearOptimize.h"
-#include "QMCDrivers/QMCCorrelatedSamplingLinearOptimize.h"
+#include "QMCDrivers/WFOpt/QMCOptimize.h"
+#include "QMCDrivers/WFOpt/QMCFixedSampleLinearOptimize.h"
+#include "QMCDrivers/WFOpt/QMCCorrelatedSamplingLinearOptimize.h"
 #include "QMCDrivers/WaveFunctionTester.h"
 #include "OhmmsData/AttributeSet.h"
 #include "OhmmsData/ParameterSet.h"

src/QMCDrivers/WFOpt/QMCCorrelatedSamplingLinearOptimize.cpp

@@ -17,15 +17,15 @@
 #include <cassert>
 #include <iostream>
 #include <fstream>
-#include "QMCDrivers/QMCCorrelatedSamplingLinearOptimize.h"
+#include "QMCDrivers/WFOpt/QMCCorrelatedSamplingLinearOptimize.h"
 #include "Particle/HDFWalkerIO.h"
 #include "OhmmsData/AttributeSet.h"
 #include "Message/CommOperators.h"
-#include "QMCDrivers/QMCCostFunctionBase.h"
-#include "QMCDrivers/QMCCostFunction.h"
+#include "QMCDrivers/WFOpt/QMCCostFunctionBase.h"
+#include "QMCDrivers/WFOpt/QMCCostFunction.h"
 #if defined(ENABLE_OPENMP)
 #include "QMCDrivers/VMC/VMC.h"
-#include "QMCDrivers/QMCCostFunction.h"
+#include "QMCDrivers/WFOpt/QMCCostFunction.h"
 #endif
 //#include "QMCDrivers/VMC/VMCSingle.h"
 //#include "QMCDrivers/QMCCostFunctionSingle.h"
@@ -34,7 +34,7 @@
 #include "Numerics/MatrixOperators.h"
 #if defined(QMC_CUDA)
 #include "QMCDrivers/VMC/VMC_CUDA.h"
-#include "QMCDrivers/QMCCostFunctionCUDA.h"
+#include "QMCDrivers/WFOpt/QMCCostFunctionCUDA.h"
 #endif
 
 /*#include "Message/Communicate.h"*/
@@ -65,7 +65,6 @@ QMCCorrelatedSamplingLinearOptimize::QMCCorrelatedSamplingLinearOptimize(MCWalke
   //read to use vmc output (just in case)
   RootName = "pot";
   QMCType  = "QMCCorrelatedSamplingLinearOptimize";
-  m_param.add(WarmupBlocks, "warmupBlocks", "int");
   m_param.add(stabilizerScale, "stabilizerscale", "double");
   m_param.add(bigChange, "bigchange", "double");
   m_param.add(w_beta, "beta", "double");

src/QMCDrivers/WFOpt/QMCCorrelatedSamplingLinearOptimize.h

@@ -17,7 +17,7 @@
 #ifndef QMCPLUSPLUS_QMCCSLINEAROPTIMIZATION_H
 #define QMCPLUSPLUS_QMCCSLINEAROPTIMIZATION_H
 
-#include "QMCDrivers/QMCLinearOptimize.h"
+#include "QMCDrivers/WFOpt/QMCLinearOptimize.h"
 #include "QMCDrivers/VMC/VMCLinearOpt.h"
 #include "Optimize/NRCOptimization.h"
 #if defined(QMC_CUDA)

src/QMCDrivers/WFOpt/QMCCostFunction.cpp

@@ -16,7 +16,7 @@
 //////////////////////////////////////////////////////////////////////////////////////
 
 
-#include "QMCDrivers/QMCCostFunction.h"
+#include "QMCDrivers/WFOpt/QMCCostFunction.h"
 #include "Particle/MCWalkerConfiguration.h"
 #include "QMCWaveFunctions/TrialWaveFunction.h"
 #include "Message/CommOperators.h"

src/QMCDrivers/WFOpt/QMCCostFunction.h

@@ -15,7 +15,7 @@
 #ifndef QMCPLUSPLUS_COSTFUNCTION_H
 #define QMCPLUSPLUS_COSTFUNCTION_H
 
-#include "QMCDrivers/QMCCostFunctionBase.h"
+#include "QMCDrivers/WFOpt/QMCCostFunctionBase.h"
 #include "QMCDrivers/CloneManager.h"
 #include "QMCWaveFunctions/OrbitalSetTraits.h"
 

src/QMCDrivers/WFOpt/QMCCostFunctionBase.cpp

@@ -15,7 +15,7 @@
 //////////////////////////////////////////////////////////////////////////////////////
 
 
-#include "QMCDrivers/QMCCostFunctionBase.h"
+#include "QMCDrivers/WFOpt/QMCCostFunctionBase.h"
 #include "Particle/MCWalkerConfiguration.h"
 #include "OhmmsData/AttributeSet.h"
 #include "OhmmsData/ParameterSet.h"

src/QMCDrivers/WFOpt/QMCCostFunctionCUDA.cpp

@@ -15,7 +15,7 @@
 //////////////////////////////////////////////////////////////////////////////////////
 
 
-#include "QMCDrivers/QMCCostFunctionCUDA.h"
+#include "QMCDrivers/WFOpt/QMCCostFunctionCUDA.h"
 #include "Particle/MCWalkerConfiguration.h"
 #include "QMCWaveFunctions/TrialWaveFunction.h"
 #include "Message/CommOperators.h"

src/QMCDrivers/WFOpt/QMCCostFunctionCUDA.h

@@ -16,7 +16,7 @@
 #ifndef QMCPLUSPLUS_COSTFUNCTION_CUDA_H
 #define QMCPLUSPLUS_COSTFUNCTION_CUDA_H
 
-#include "QMCDrivers/QMCCostFunctionBase.h"
+#include "QMCDrivers/WFOpt/QMCCostFunctionBase.h"
 #include "QMCDrivers/CloneManager.h"
 #include "QMCWaveFunctions/OrbitalSetTraits.h"
 #include "type_traits/CUDATypes.h"

src/QMCDrivers/WFOpt/QMCFixedSampleLinearOptimize.cpp

@@ -15,21 +15,21 @@
 //////////////////////////////////////////////////////////////////////////////////////
 
 
-#include "QMCDrivers/QMCFixedSampleLinearOptimize.h"
+#include "QMCDrivers/WFOpt/QMCFixedSampleLinearOptimize.h"
 #include "Particle/HDFWalkerIO.h"
 #include "OhmmsData/AttributeSet.h"
 #include "Message/CommOperators.h"
-#include "QMCDrivers/QMCCostFunctionBase.h"
-#include "QMCDrivers/QMCCostFunction.h"
+#include "QMCDrivers/WFOpt/QMCCostFunctionBase.h"
+#include "QMCDrivers/WFOpt/QMCCostFunction.h"
 #include "QMCDrivers/VMC/VMC.h"
-#include "QMCDrivers/QMCCostFunction.h"
+#include "QMCDrivers/WFOpt/QMCCostFunction.h"
 #include "QMCHamiltonians/HamiltonianPool.h"
 #include "Numerics/Blasf.h"
 #include "Numerics/MatrixOperators.h"
 #include <cassert>
 #if defined(QMC_CUDA)
 #include "QMCDrivers/VMC/VMC_CUDA.h"
-#include "QMCDrivers/QMCCostFunctionCUDA.h"
+#include "QMCDrivers/WFOpt/QMCCostFunctionCUDA.h"
 #endif
 #ifdef HAVE_LMY_ENGINE
 #include "formic/utils/matrix.h"
@@ -100,7 +100,6 @@ QMCFixedSampleLinearOptimize::QMCFixedSampleLinearOptimize(MCWalkerConfiguration
   //read to use vmc output (just in case)
   RootName = "pot";
   QMCType  = "QMCFixedSampleLinearOptimize";
-  m_param.add(WarmupBlocks, "warmupBlocks", "int");
   m_param.add(Max_iterations, "max_its", "int");
   m_param.add(nstabilizers, "nstabilizers", "int");
   m_param.add(stabilizerScale, "stabilizerscale", "double");

src/QMCDrivers/WFOpt/QMCFixedSampleLinearOptimize.h

@@ -17,7 +17,7 @@
 #ifndef QMCPLUSPLUS_QMCFSLINEAROPTIMIZATION_VMCSINGLE_H
 #define QMCPLUSPLUS_QMCFSLINEAROPTIMIZATION_VMCSINGLE_H
 
-#include "QMCDrivers/QMCLinearOptimize.h"
+#include "QMCDrivers/WFOpt/QMCLinearOptimize.h"
 #include "Optimize/NRCOptimization.h"
 #ifdef HAVE_LMY_ENGINE
 #include "formic/utils/matrix.h"

src/QMCDrivers/WFOpt/QMCLinearOptimize.cpp

@@ -14,13 +14,13 @@
 //////////////////////////////////////////////////////////////////////////////////////
 
 
-#include "QMCDrivers/QMCLinearOptimize.h"
+#include "QMCDrivers/WFOpt/QMCLinearOptimize.h"
 #include "Particle/HDFWalkerIO.h"
 #include "OhmmsData/AttributeSet.h"
 #include "Message/CommOperators.h"
 //#if defined(ENABLE_OPENMP)
 #include "QMCDrivers/VMC/VMC.h"
-#include "QMCDrivers/QMCCostFunction.h"
+#include "QMCDrivers/WFOpt/QMCCostFunction.h"
 //#endif
 //#include "QMCDrivers/VMC/VMCSingle.h"
 //#include "QMCDrivers/QMCCostFunctionSingle.h"
@@ -30,7 +30,7 @@
 #include <cassert>
 #if defined(QMC_CUDA)
 #include "QMCDrivers/VMC/VMC_CUDA.h"
-#include "QMCDrivers/QMCCostFunctionCUDA.h"
+#include "QMCDrivers/WFOpt/QMCCostFunctionCUDA.h"
 #endif
 #include "Numerics/LinearFit.h"
 #include <iostream>
@@ -49,7 +49,6 @@ QMCLinearOptimize::QMCLinearOptimize(MCWalkerConfiguration& w,
     : QMCDriver(w, psi, h, ppool, comm),
       PartID(0),
       NumParts(1),
-      WarmupBlocks(10),
       hamPool(hpool),
       wfNode(NULL),
       optNode(NULL),
@@ -172,11 +171,6 @@ void QMCLinearOptimize::finish()
 void QMCLinearOptimize::generateSamples()
 {
   app_log() << "<optimization-report>" << std::endl;
-  //if(WarmupBlocks)
-  //{
-  //  app_log() << "<vmc stage=\"warm-up\" blocks=\"" << WarmupBlocks << "\">" << std::endl;
-  //  //turn off QMC_OPTIMIZE
-  //  vmcEngine->setValue("blocks",WarmupBlocks);
   vmcEngine->qmc_driver_mode.set(QMC_WARMUP, 1);
   //  vmcEngine->run();
   //  vmcEngine->setValue("blocks",nBlocks);

src/QMCDrivers/WFOpt/QMCLinearOptimize.h

@@ -72,9 +72,7 @@ public:
   int PartID;
   ///total number of partitions that will share a set of configuratons
   int NumParts;
-  ///total number of Warmup Blocks
-  int WarmupBlocks;
-  ///total number of Warmup Blocks
+  ///total number of VMC walkers
   int NumOfVMCWalkers;
   ///Number of iterations maximum before generating new configurations.
   int Max_iterations;

src/QMCDrivers/WFOpt/QMCOptimize.cpp

@@ -14,7 +14,7 @@
 //////////////////////////////////////////////////////////////////////////////////////
 
 
-#include "QMCDrivers/QMCOptimize.h"
+#include "QMCDrivers/WFOpt/QMCOptimize.h"
 #include "Particle/HDFWalkerIO.h"
 #include "OhmmsData/AttributeSet.h"
 #include "Message/CommOperators.h"
@@ -23,10 +23,10 @@
 #include "Optimize/DampedDynamics.h"
 //#include "QMCDrivers/QMCCostFunctionSingle.h"
 #include "QMCDrivers/VMC/VMC.h"
-#include "QMCDrivers/QMCCostFunction.h"
+#include "QMCDrivers/WFOpt/QMCCostFunction.h"
 #if defined(QMC_CUDA)
 #include "QMCDrivers/VMC/VMC_CUDA.h"
-#include "QMCDrivers/QMCCostFunctionCUDA.h"
+#include "QMCDrivers/WFOpt/QMCCostFunctionCUDA.h"
 #endif
 #include "QMCHamiltonians/HamiltonianPool.h"
 
@@ -41,8 +41,6 @@ QMCOptimize::QMCOptimize(MCWalkerConfiguration& w,
     : QMCDriver(w, psi, h, ppool, comm),
       PartID(0),
       NumParts(1),
-      WarmupBlocks(10),
-      SkipSampleGeneration("no"),
       hamPool(hpool),
       optSolver(0),
       vmcEngine(0),
@@ -57,8 +55,6 @@ QMCOptimize::QMCOptimize(MCWalkerConfiguration& w,
   QMCType  = "QMCOptimize";
   //default method is cg
   optmethod = "cg";
-  m_param.add(WarmupBlocks, "warmupBlocks", "int");
-  m_param.add(SkipSampleGeneration, "skipVMC", "string");
 }
 
 /** Clean up the vector */

src/QMCDrivers/WFOpt/QMCOptimize.h

@@ -64,12 +64,8 @@ private:
   int PartID;
   ///total number of partitions that will share a set of configuratons
   int NumParts;
-  ///total number of Warmup Blocks
-  int WarmupBlocks;
-  ///total number of Warmup Blocks
+  ///total number of VMC walkers
   int NumOfVMCWalkers;
-  ///yes/no applicable only first time
-  std::string SkipSampleGeneration;
   ///need to know HamiltonianPool to use OMP
   HamiltonianPool& hamPool;
   ///target cost function to optimize

src/QMCDrivers/tests/CMakeLists.txt

@@ -43,7 +43,7 @@ ELSE(QMC_CUDA)
 ENDIF(QMC_CUDA)
 
 USE_FAKE_RNG(${UTEST_EXE})
-TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcdriver_unit qmcham_unit qmcwfs qmcbase qmcutil qmcfakerng ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcdriver_unit qmcfakerng)
 
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
 SET_TESTS_PROPERTIES(${UTEST_NAME} PROPERTIES WORKING_DIRECTORY ${UTEST_DIR})
@@ -63,7 +63,7 @@ IF(NOT QMC_CUDA AND ENABLE_SOA)
   SET(DRIVER_TEST_SRC SetupPools.cpp test_Crowd.cpp test_MCPopulation.cpp test_ContextForSteps.cpp test_QMCDriverInput.cpp test_QMCDriverNew.cpp test_VMCDriverInput.cpp test_VMCFactoryNew.cpp test_VMCBatched.cpp test_DMCBatched.cpp test_SimpleFixedNodeBranch.cpp)
   ADD_EXECUTABLE(${UTEST_EXE} ${DRIVER_TEST_SRC})
   USE_FAKE_RNG(${UTEST_EXE})
-  TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcdriver_unit qmcham_unit qmcwfs qmcbase qmcutil qmcfakerng ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+  TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcdriver_unit qmcfakerng)
 
   ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
   SET_TESTS_PROPERTIES(${UTEST_NAME} PROPERTIES WORKING_DIRECTORY ${UTEST_DIR})
@@ -77,8 +77,7 @@ IF(NOT QMC_CUDA AND ENABLE_SOA)
     #this is dependent on the directory creation and sym linking of earlier driver tests
     SET(DRIVER_TEST_SRC SetupPools.cpp test_WalkerControlMPI.cpp)
     ADD_EXECUTABLE(${UTEST_EXE} ${DRIVER_TEST_SRC})
-    #Way too many depenedencies make for very slow test linking
-    TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcdriver_unit qmcham_unit qmcwfs qmcbase qmcutil qmcfakerng ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+    TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcdriver_unit qmcfakerng)
     # Right now the unified driver mpi tests are hard coded for 3 MPI ranks
     ADD_MPI_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}" 3)
     SET_TESTS_PROPERTIES(${UTEST_NAME} PROPERTIES WORKING_DIRECTORY ${UTEST_DIR})

src/QMCDrivers/tests/SetupDMCTest.h

@@ -27,7 +27,14 @@ namespace testing
 class SetupDMCTest : public SetupPools
 {
 public:
-  SetupDMCTest(int nranks = 4) : num_ranks(nranks), qmcdrv_input(3)
+  SetupDMCTest(int nranks = 4)
+      : population(comm->size(),
+                   particle_pool->getParticleSet("e"),
+                   wavefunction_pool->getPrimary(),
+                   hamiltonian_pool->getPrimary(),
+                   comm->rank()),
+        num_ranks(nranks),
+        qmcdrv_input(3)
   {
     if (Concurrency::maxThreads<>() < 8)
       num_crowds = Concurrency::maxThreads<>();
@@ -35,8 +42,6 @@ public:
 
   DMCBatched operator()()
   {
-    int num_ranks = comm->size();
-
     Libxml2Document doc;
     doc.parseFromString(valid_dmc_input_sections[valid_dmc_input_dmc_batch_index]);
     node = doc.getRoot();
@@ -44,9 +49,6 @@ public:
     qmcdrv_input.readXML(node);
     dmcdrv_input.readXML(node);
 
-    population = MCPopulation(num_ranks, particle_pool->getParticleSet("e"), wavefunction_pool->getPrimary(),
-                              hamiltonian_pool->getPrimary(),comm->rank());
-
     QMCDriverInput qmc_input_copy(qmcdrv_input);
     DMCDriverInput dmc_input_copy(dmcdrv_input);
     return {std::move(qmc_input_copy),
@@ -69,7 +71,6 @@ public:
 
   QMCDriverInput qmcdrv_input;
   DMCDriverInput dmcdrv_input;
-
 };
 } // namespace testing
 } // namespace qmcplusplus

src/QMCHamiltonians/CMakeLists.txt

@@ -63,6 +63,7 @@ IF(OHMMS_DIM MATCHES 3)
     NonLocalECPotential.cpp
     L2Potential.cpp
     SOECPComponent.cpp
+    SOECPotential.cpp
     ECPotentialBuilder.cpp
     ECPComponentBuilder.cpp
     ECPComponentBuilder.1.cpp
@@ -107,12 +108,19 @@ ADD_LIBRARY(qmcham ${HAMSRCS})
 ADD_LIBRARY(qmcham_unit ${HAMSRCS})
 IF(QMC_CUDA)
   CUDA_ADD_LIBRARY(qmcham_cuda ${HAMSRCS_CUDA})
-  TARGET_LINK_LIBRARIES(qmcham qmcham_cuda)
-  TARGET_LINK_LIBRARIES(qmcham_unit qmcham_cuda)
+  TARGET_LINK_LIBRARIES(qmcham PRIVATE qmcham_cuda)
+  TARGET_LINK_LIBRARIES(qmcham_unit PRIVATE qmcham_cuda)
 ENDIF(QMC_CUDA)
 
 USE_FAKE_RNG(qmcham_unit)
-TARGET_LINK_LIBRARIES(qmcham_unit qmcfakerng)
+TARGET_LINK_LIBRARIES(qmcham_unit PUBLIC qmcfakerng)
+
+TARGET_LINK_LIBRARIES(qmcham PUBLIC qmcwfs)
+TARGET_LINK_LIBRARIES(qmcham_unit PUBLIC qmcwfs)
+
+TARGET_LINK_LIBRARIES(qmcham PRIVATE einspline Math::BLAS_LAPACK Math::FFTW3)
+TARGET_LINK_LIBRARIES(qmcham_unit PRIVATE einspline Math::BLAS_LAPACK Math::FFTW3)
+
 #IF(QMC_BUILD_STATIC)
 
 #IF(QMC_BUILD_STATIC)

src/QMCHamiltonians/ECPComponentBuilder.2.cpp

@@ -56,6 +56,7 @@ void ECPComponentBuilder::buildSemiLocalAndLocal(std::vector<xmlNodePtr>& semiPt
   aAttrib.add(nup, "npots-up");
   aAttrib.add(Llocal, "l-local");
   aAttrib.add(Nrule, "nrule");
+  aAttrib.add(Srule, "srule");
   aAttrib.add(nso, "npots-so");
 
   xmlNodePtr cur_semilocal = semiPtr[0];
@@ -82,12 +83,12 @@ void ECPComponentBuilder::buildSemiLocalAndLocal(std::vector<xmlNodePtr>& semiPt
   }
   // We cannot construct the potentials as we construct them since
   // we may not know which one is local yet.
-  
+
   std::vector<int> angList;
   std::vector<int> angListSO; //For spin-orbit, if it exists
   std::vector<xmlNodePtr> vpsPtr;
   std::vector<xmlNodePtr> vpsoPtr; //For spin-orbit, if it exists.
-  Lmax = -1;
+  Lmax   = -1;
   LmaxSO = -1;
   // Now read vps sections
   xmlNodePtr cur_vps = cur_semilocal->children;
@@ -108,7 +109,7 @@ void ECPComponentBuilder::buildSemiLocalAndLocal(std::vector<xmlNodePtr>& semiPt
       vpsPtr.push_back(cur_vps);
       Lmax = std::max(Lmax, l); //count the maximum L
     }
-    else if (vname== "vps_so") //This accumulates the spin-orbit corrections, if defined.
+    else if (vname == "vps_so") //This accumulates the spin-orbit corrections, if defined.
     {
       OhmmsAttributeSet aAttrib;
       std::string lstr("s");
@@ -132,13 +133,13 @@ void ECPComponentBuilder::buildSemiLocalAndLocal(std::vector<xmlNodePtr>& semiPt
     app_log() << "    Only one vps is found. Set the local component=" << Lmax << std::endl;
   }
 
-  if (angListSO.size()!=nso)
+  if (angListSO.size() != nso)
   {
     std::stringstream ssout;
-    ssout<<"Error. npots-so="<<angListSO.size()<<" while declared number of SO channels is "<<nso<<std::endl;
+    ssout << "Error. npots-so=" << angListSO.size() << " while declared number of SO channels is " << nso << std::endl;
     std::string outstring("");
-    outstring=ssout.str();
- 
+    outstring = ssout.str();
+
     APP_ABORT(outstring.c_str());
   }
   int npts = grid_global->size();
@@ -188,12 +189,11 @@ void ECPComponentBuilder::buildSemiLocalAndLocal(std::vector<xmlNodePtr>& semiPt
     //copy the numerical data with the correct map
     //So this is weird, but I feel like l should be the proper index for vnnso,
     //with angListSO[l] being the actual angular momentum channel referred to by l.
-    //This differs from the parsing of the nonlocal pseudopotential piece, but whatever. 
+    //This differs from the parsing of the nonlocal pseudopotential piece, but whatever.
     copy(vtso.begin(), vtso.end(), vnnso[l]);
   }
 
 
-  
   ////rather stupid to do this but necessary
   //vector<RealType> temp(npts);
   //for(int i=0; i<npts; i++) temp[i]=grid_global->r(i);
@@ -203,15 +203,23 @@ void ECPComponentBuilder::buildSemiLocalAndLocal(std::vector<xmlNodePtr>& semiPt
     for (int i = 0; i < vnn.rows(); i++)
       for (int j = 0; j < npts; j++)
         vnn[i][j] *= grid_global->r(j);
+    for (int i = 0; i < vnnso.rows(); i++)
+      for (int j = 0; j < npts; j++)
+        vnnso[i][j] *= grid_global->r(j);
   }
   app_log() << "   Number of angular momentum channels " << angList.size() << std::endl;
   app_log() << "   Maximum angular momentum channel " << Lmax << std::endl;
   doBreakUp(angList, vnn, rmax, Vprefactor);
 
   //If any spinorbit terms are found...
-  if(nso>0)
-    buildSO(angListSO,vnnso,rmax,1.0);
-
+  if (nso > 0)
+    buildSO(angListSO, vnnso, rmax, 1.0);
+  else
+  {
+    //No SO channels found. Delete pp_so
+    delete pp_so;
+    pp_so = 0;
+  }
 }
 
 //Most of this is copied directly from doBreakUp, but is separated to prevent from cluttering doBreakUp.
@@ -254,7 +262,7 @@ void ECPComponentBuilder::buildSO(const std::vector<int>& angList,
   std::vector<mRealType> newPin(ngIn);
   for (int l = 0; l < angList.size(); l++)
   {
-    const mRealType* restrict vp    = vnnso[l];
+    const mRealType* restrict vp = vnnso[l];
     for (int i = 0; i < ngIn; i++)
       newPin[i] = Vprefactor * vp[i];
 
@@ -263,16 +271,16 @@ void ECPComponentBuilder::buildSO(const std::vector<int>& angList,
     for (int i = 1; i < ng - 1; i++)
     {
       mRealType r = d * i;
-      newP[i]     = infunc.splint(r);
+      newP[i]     = infunc.splint(r) / r;
     }
     newP[0]                  = newP[1];
     newP[ng - 1]             = 0.0;
     RadialPotentialType* app = new RadialPotentialType(agrid, newP);
     app->spline();
-    pp_so->add(angList[l], app); 
+    pp_so->add(angList[l], app);
   }
-
-
+  NumSO       = angList.size();
+  pp_so->Rmax = rmax;
 }
 
 bool ECPComponentBuilder::parseCasino(const std::string& fname, xmlNodePtr cur)

src/QMCHamiltonians/ECPComponentBuilder.cpp

@@ -32,6 +32,7 @@ ECPComponentBuilder::ECPComponentBuilder(const std::string& aname, Communicate*
       NumNonLocal(0),
       Lmax(0),
       Nrule(4),
+      Srule(8),
       AtomicNumber(0),
       Zeff(0),
       RcutMax(-1),
@@ -57,7 +58,8 @@ ECPComponentBuilder::ECPComponentBuilder(const std::string& aname, Communicate*
 bool ECPComponentBuilder::parse(const std::string& fname, xmlNodePtr cur)
 {
   const XMLAttrString cutoff_str(cur, "cutoff");
-  if(!cutoff_str.empty()) RcutMax = std::stod(cutoff_str);
+  if (!cutoff_str.empty())
+    RcutMax = std::stod(cutoff_str);
 
   return read_pp_file(fname);
 }
@@ -264,6 +266,12 @@ void ECPComponentBuilder::SetQuadratureRule(int rule)
   pp_nonloc->sgridweight_m = myRule.weight_m;
   // Allocate storage for wave function ratios
   pp_nonloc->resize_warrays(myRule.nk, NumNonLocal, Lmax);
+  if (pp_so)
+  { //added here bc must have nonlocal terms to have SO contributions
+    pp_so->sgridxyz_m    = myRule.xyz_m;
+    pp_so->sgridweight_m = myRule.weight_m;
+    pp_so->resize_warrays(myRule.nk, NumSO, Srule);
+  }
 }
 
 } // namespace qmcplusplus

src/QMCHamiltonians/ECPComponentBuilder.h

@@ -33,8 +33,8 @@ struct ECPComponentBuilder : public MPIObjectBase, public QMCTraits
   typedef LocalECPotential::RadialPotentialType RadialPotentialType;
 
   int NumNonLocal;
-  int Lmax, Llocal, Nrule;
-  int NumSO; //The number of spin-orbit channels.
+  int Lmax, Llocal, Nrule, Srule;
+  int NumSO;  //The number of spin-orbit channels.
   int LmaxSO; //The maximum angular momentum of spin-orbit channels.
   int AtomicNumber;
   RealType Zeff;
@@ -91,7 +91,7 @@ struct ECPComponentBuilder : public MPIObjectBase, public QMCTraits
   **
   ** return void
   **
-  **/    
+  **/
   void buildSO(const std::vector<int>& angList,
                const Matrix<mRealType>& vnnso,
                RealType rmax,

src/QMCHamiltonians/ECPotentialBuilder.cpp

@@ -42,6 +42,7 @@ ECPotentialBuilder::ECPotentialBuilder(QMCHamiltonian& h,
     : MPIObjectBase(c),
       hasLocalPot(false),
       hasNonLocalPot(false),
+      hasSOPot(false),
       hasL2Pot(false),
       targetH(h),
       IonConfig(ions),
@@ -128,8 +129,7 @@ bool ECPotentialBuilder::put(xmlNodePtr cur)
 #ifdef QMC_CUDA
     NonLocalECPotential_CUDA* apot = new NonLocalECPotential_CUDA(IonConfig, targetPtcl, targetPsi, usePBC, doForces);
 #else
-    NonLocalECPotential* apot =
-        new NonLocalECPotential(IonConfig, targetPtcl, targetPsi, doForces, use_DLA == "yes");
+    NonLocalECPotential* apot = new NonLocalECPotential(IonConfig, targetPtcl, targetPsi, doForces, use_DLA == "yes");
 #endif
     int nknot_max = 0;
     for (int i = 0; i < nonLocalPot.size(); i++)
@@ -149,6 +149,26 @@ bool ECPotentialBuilder::put(xmlNodePtr cur)
 
     targetH.addOperator(apot, "NonLocalECP");
   }
+  if (hasSOPot)
+  {
+    SOECPotential* apot = new SOECPotential(IonConfig, targetPtcl, targetPsi);
+    int nknot_max       = 0;
+    int sknot_max       = 0;
+    for (int i = 0; i < soPot.size(); i++)
+    {
+      if (soPot[i])
+      {
+        nknot_max = std::max(nknot_max, soPot[i]->getNknot());
+        sknot_max = std::max(sknot_max, soPot[i]->getSknot());
+        apot->addComponent(i, soPot[i]);
+      }
+    }
+    app_log() << "\n  Using SOECP potential \n"
+              << "    Maximum grid on a sphere for SOECPotential: " << nknot_max << std::endl;
+    app_log() << "    Maximum grid for Simpson's rule for spin integral: " << sknot_max << std::endl;
+
+    targetH.addOperator(apot, "SOECP"); //default is physical operator
+  }
   if (hasL2Pot)
   {
     L2Potential* apot = new L2Potential(IonConfig, targetPtcl, targetPsi);
@@ -225,6 +245,11 @@ void ECPotentialBuilder::useXmlFormat(xmlNodePtr cur)
             hasNonLocalPot            = true;
             nonLocalPot[speciesIndex] = ecp.pp_nonloc;
           }
+          if (ecp.pp_so)
+          {
+            hasSOPot            = true;
+            soPot[speciesIndex] = ecp.pp_so;
+          }
           if (ecp.pp_L2)
           {
             hasL2Pot            = true;

src/QMCHamiltonians/ECPotentialBuilder.h

@@ -17,6 +17,7 @@
 #include "Configuration.h"
 #include "QMCHamiltonians/LocalECPotential.h"
 #include "QMCHamiltonians/NonLocalECPotential.h"
+#include "QMCHamiltonians/SOECPotential.h"
 #include "QMCHamiltonians/L2Potential.h"
 namespace qmcplusplus
 {
@@ -30,6 +31,7 @@ struct ECPotentialBuilder : public MPIObjectBase, public QMCTraits
   typedef LocalECPotential::GridType GridType;
   bool hasLocalPot;
   bool hasNonLocalPot;
+  bool hasSOPot;
   bool hasL2Pot;
 
   QMCHamiltonian& targetH;
@@ -40,6 +42,7 @@ struct ECPotentialBuilder : public MPIObjectBase, public QMCTraits
   std::vector<RealType> localZeff;
   std::vector<RadialPotentialType*> localPot;
   std::vector<NonLocalECPComponent*> nonLocalPot;
+  std::vector<SOECPComponent*> soPot;
   std::vector<L2RadialPotential*> L2Pot;
 
   ECPotentialBuilder(QMCHamiltonian& h, ParticleSet& ions, ParticleSet& els, TrialWaveFunction& psi, Communicate* c);

src/QMCHamiltonians/SOECPComponent.cpp

@@ -9,6 +9,8 @@
 //                    Jeongnim Kim, jeongnim.kim@gmail.com, University of Illinois at Urbana-Champaign
 //                    Jaron T. Krogel, krogeljt@ornl.gov, Oak Ridge National Laboratory
 //                    Mark A. Berrill, berrillma@ornl.gov, Oak Ridge National Laboratory
+//                    Cody A. Melton, cmelton@sandia.gov, Sandia National Laboratories
+//                    Raymond C. Clay, rclay@sandia.gov, Sandia National Laboratories
 //
 // File created by: Jeongnim Kim, jeongnim.kim@gmail.com, University of Illinois at Urbana-Champaign
 //////////////////////////////////////////////////////////////////////////////////////
@@ -16,12 +18,236 @@
 
 #include "Particle/DistanceTableData.h"
 #include "QMCHamiltonians/SOECPComponent.h"
+#include "Numerics/Ylm.h"
 
 namespace qmcplusplus
 {
+SOECPComponent::SOECPComponent() : lmax(0), nchannel(0), nknot(0), sknot(0), Rmax(-1) {}
+
+SOECPComponent::~SOECPComponent()
+{
+  for (int i = 0; i < sopp_m.size(); i++)
+    delete sopp_m[i];
+}
+
+void SOECPComponent::print(std::ostream& os) {}
+
 void SOECPComponent::add(int l, RadialPotentialType* pp)
 {
   angpp_m.push_back(l);
   sopp_m.push_back(pp);
 }
+
+SOECPComponent* SOECPComponent::makeClone(const ParticleSet& qp)
+{
+  SOECPComponent* myclone = new SOECPComponent(*this);
+  for (int i = 0; i < sopp_m.size(); i++)
+    myclone->sopp_m[i] = sopp_m[i]->makeClone();
+  return myclone;
 }
+
+void SOECPComponent::resize_warrays(int n, int m, int s)
+{
+  psiratio.resize(n);
+  deltaV.resize(n);
+  vrad.resize(m);
+  rrotsgrid_m.resize(n);
+  nchannel = sopp_m.size();
+  nknot    = sgridxyz_m.size();
+  sknot    = s;
+  if (m != nchannel)
+  {
+    APP_ABORT("SOECPComponent::resize_warrays has incorrect number of radial channels\n");
+  }
+}
+
+int SOECPComponent::kroneckerDelta(int x, int y) { return (x == y) ? 1 : 0; }
+
+SOECPComponent::ComplexType SOECPComponent::sMatrixElements(RealType s1, RealType s2, int dim)
+{
+  switch (dim)
+  {
+  case 0:
+    return ComplexType(std::cos(s1 + s2), 0.0);
+    break;
+  case 1:
+    return ComplexType(std::sin(s1 + s2), 0.0);
+    break;
+  case 2:
+    return ComplexType(0.0, std::sin(s1 - s2));
+    break;
+  default:
+    APP_ABORT("SOECPComponent::sMatrixElements invalid operator dimension\n");
+    return 0;
+    break;
+  }
+}
+
+SOECPComponent::ComplexType SOECPComponent::lmMatrixElements(int l, int m1, int m2, int dim)
+{
+  ComplexType val;
+  RealType onehalf = 0.5;
+  RealType zero    = 0.0;
+  switch (dim)
+  {
+  case 0: //x
+    val = onehalf *
+        ComplexType(std::sqrt(l * (l + 1) - m2 * (m2 + 1)) * kroneckerDelta(m1, m2 + 1) +
+                        std::sqrt(l * (l + 1) - m2 * (m2 - 1)) * kroneckerDelta(m1, m2 - 1),
+                    zero);
+    return val;
+    break;
+  case 1:
+    val = onehalf *
+        ComplexType(0.0,
+                    std::sqrt(l * (l + 1) - m2 * (m2 - 1)) * kroneckerDelta(m1, m2 - 1) -
+                        std::sqrt(l * (l + 1) - m2 * (m2 + 1)) * kroneckerDelta(m1, m2 + 1));
+    return val;
+    break;
+  case 2:
+    val = ComplexType(m2 * kroneckerDelta(m1, m2), zero);
+    return val;
+    break;
+  default:
+    APP_ABORT("SOECPComponent::lMatrixElements Invalid operator dimension\n");
+    return 0;
+    break;
+  }
+}
+
+SOECPComponent::ComplexType SOECPComponent::getAngularIntegral(RealType sold,
+                                                               RealType snew,
+                                                               ParticleSet& W,
+                                                               TrialWaveFunction& Psi,
+                                                               int iel,
+                                                               RealType r,
+                                                               const PosType& dr)
+{
+  //quadrature sum for angular integral
+  for (int j = 0; j < nknot; j++)
+  {
+    deltaV[j] = r * rrotsgrid_m[j] - dr;
+    W.makeMoveWithSpin(iel, deltaV[j], snew - sold);
+    psiratio[j] = Psi.calcRatio(W, iel) * sgridweight_m[j];
+    W.rejectMove(iel);
+    Psi.resetPhaseDiff();
+  }
+
+  ComplexType angint(0.0);
+  for (int j = 0; j < nknot; j++)
+  {
+    ComplexType lsum(0.0);
+    for (int il = 0; il < nchannel; il++)
+    {
+      int l = il + 1; //nchannels starts at l=1, so 0th element is p not s
+      ComplexType msums(0.0);
+      for (int m1 = -l; m1 <= l; m1++)
+      {
+        for (int m2 = -l; m2 <= l; m2++)
+        {
+          ComplexType ldots(0.0);
+          for (int d = 0; d < 3; d++)
+            ldots += lmMatrixElements(l, m1, m2, d) * sMatrixElements(sold, snew, d);
+          //Seemingly Numerics/Ylm takes unit vector with order z,x,y...why
+          RealType rmag = std::sqrt(dr[0] * dr[0] + dr[1] * dr[1] + dr[2] * dr[2]);
+          PosType rr    = dr / rmag;
+          PosType tmp;
+          tmp[0]         = rr[2];
+          tmp[1]         = rr[0];
+          tmp[2]         = rr[1];
+          ComplexType cY = std::conj(Ylm(l, m1, tmp));
+          tmp[0]         = rrotsgrid_m[j][2];
+          tmp[1]         = rrotsgrid_m[j][0];
+          tmp[2]         = rrotsgrid_m[j][1];
+          ComplexType Y  = Ylm(l, m2, tmp);
+          msums += cY * Y * ldots;
+        }
+      }
+      lsum += vrad[il] * msums;
+    }
+    angint += psiratio[j] * lsum;
+  }
+  return angint;
+}
+
+SOECPComponent::RealType SOECPComponent::evaluateOne(ParticleSet& W,
+                                                     int iat,
+                                                     TrialWaveFunction& Psi,
+                                                     int iel,
+                                                     RealType r,
+                                                     const PosType& dr)
+{
+  if (sknot < 2)
+    APP_ABORT("Spin knots must be greater than 2\n");
+
+  if (sknot % 2 != 0)
+    APP_ABORT("Spin knots uses Simpson's rule. Must have even number of knots");
+
+  for (int ip = 0; ip < nchannel; ip++)
+  {
+    vrad[ip] = sopp_m[ip]->splint(r);
+  }
+
+  RealType smin(0.0);
+  RealType smax(TWOPI);
+  RealType dS = (smax - smin) / sknot; //step size for spin
+
+  RealType sold = W.spins[iel];
+  ComplexType sint(0.0);
+
+  for (int is = 1; is <= sknot - 1; is += 2)
+  {
+    RealType snew      = smin + is * dS;
+    ComplexType angint = getAngularIntegral(sold, snew, W, Psi, iel, r, dr);
+    sint += RealType(4. / 3.) * dS * angint;
+  }
+  for (int is = 2; is <= sknot - 2; is += 2)
+  {
+    RealType snew      = smin + is * dS;
+    ComplexType angint = getAngularIntegral(sold, snew, W, Psi, iel, r, dr);
+    sint += RealType(2. / 3.) * dS * angint;
+  }
+  sint += RealType(1. / 3.) * dS * getAngularIntegral(sold, smin, W, Psi, iel, r, dr);
+  sint += RealType(1. / 3.) * dS * getAngularIntegral(sold, smax, W, Psi, iel, r, dr);
+
+  RealType pairpot = std::real(sint);
+  return pairpot;
+}
+
+void SOECPComponent::randomize_grid(RandomGenerator_t& myRNG)
+{
+  RealType phi(TWOPI * myRNG()), psi(TWOPI * myRNG()), cth(myRNG() - 0.5);
+  RealType sph(std::sin(phi)), cph(std::cos(phi)), sth(std::sqrt(1.0 - cth * cth)), sps(std::sin(psi)),
+      cps(std::cos(psi));
+  TensorType rmat(cph * cth * cps - sph * sps, sph * cth * cps + cph * sps, -sth * cps, -cph * cth * sps - sph * cps,
+                  -sph * cth * sps + cph * cps, sth * sps, cph * sth, sph * sth, cth);
+  for (int i = 0; i < sgridxyz_m.size(); i++)
+    rrotsgrid_m[i] = dot(rmat, sgridxyz_m[i]);
+}
+
+template<typename T>
+void SOECPComponent::randomize_grid(std::vector<T>& sphere, RandomGenerator_t& myRNG)
+{
+  RealType phi(TWOPI * myRNG()), psi(TWOPI * myRNG()), cth(myRNG() - 0.5);
+  RealType sph(std::sin(phi)), cph(std::cos(phi)), sth(std::sqrt(1.0 - cth * cth)), sps(std::sin(psi)),
+      cps(std::cos(psi));
+  TensorType rmat(cph * cth * cps - sph * sps, sph * cth * cps + cph * sps, -sth * cps, -cph * cth * sps - sph * cps,
+                  -sph * cth * sps + cph * cps, sth * sps, cph * sth, sph * sth, cth);
+  SpherGridType::iterator it(sgridxyz_m.begin());
+  SpherGridType::iterator it_end(sgridxyz_m.end());
+  SpherGridType::iterator jt(rrotsgrid_m.begin());
+  while (it != it_end)
+  {
+    *jt = dot(rmat, *it);
+    ++it;
+    ++jt;
+  }
+  //copy the randomized grid to sphere
+  for (int i = 0; i < rrotsgrid_m.size(); i++)
+    for (int j = 0; j < OHMMS_DIM; j++)
+      sphere[OHMMS_DIM * i + j] = rrotsgrid_m[i][j];
+}
+
+template void SOECPComponent::randomize_grid(std::vector<float>& sphere, RandomGenerator_t& myRNG);
+template void SOECPComponent::randomize_grid(std::vector<double>& sphere, RandomGenerator_t& myRNG);
+} // namespace qmcplusplus

src/QMCHamiltonians/SOECPComponent.h

@@ -29,9 +29,11 @@ namespace qmcplusplus
  **           Currently, this class does nothing other than generate and store \f$\Delta V_SO(r)\f$
  **           for different orbital angular momenta.  Implementation coming soon!  
  **/
+
 class SOECPComponent : public QMCTraits
 {
 private:
+  typedef std::vector<PosType> SpherGridType;
   typedef OneDimGridBase<RealType> GridType;
   typedef OneDimCubicSpline<RealType> RadialPotentialType;
 
@@ -39,6 +41,8 @@ private:
   int lmax;
   ///the number of non-local channels
   int nchannel;
+  int nknot;
+  int sknot;
   ///Maximum cutoff the non-local pseudopotential
   RealType Rmax;
   ///Angular momentum map
@@ -46,16 +50,41 @@ private:
   ///Non-Local part of the pseudo-potential
   std::vector<RadialPotentialType*> sopp_m;
 
-public:
+  ComplexType sMatrixElements(RealType s1, RealType s2, int dim);
+  ComplexType lmMatrixElements(int l, int m1, int m2, int dim);
+  int kroneckerDelta(int x, int y);
 
-  SOECPComponent* makeClone(const ParticleSet& qp)
-  {
-    APP_ABORT("SOECPComponent::makeClone not yet implemented");
-    return nullptr;
-  };
+  ComplexType getAngularIntegral(RealType sold,
+                                 RealType snew,
+                                 ParticleSet& W,
+                                 TrialWaveFunction& Psi,
+                                 int iel,
+                                 RealType r,
+                                 const PosType& dr);
+
+  std::vector<PosType> deltaV;
+  SpherGridType sgridxyz_m;
+  SpherGridType rrotsgrid_m;
+  std::vector<ValueType> psiratio;
+  std::vector<ValueType> vrad;
+  std::vector<RealType> sgridweight_m;
+
+
+public:
+  SOECPComponent();
+  ~SOECPComponent();
+
+  SOECPComponent* makeClone(const ParticleSet& qp);
 
   ///add a new Spin-Orbit component
   void add(int l, RadialPotentialType* pp);
+
+  void resize_warrays(int n, int m, int s);
+
+  void randomize_grid(RandomGenerator_t& myRNG);
+  template<typename T>
+  void randomize_grid(std::vector<T>& sphere, RandomGenerator_t& myRNG);
+
   ///API for accessing the value of an SO radial potential at distance r.  For unit and other testing.
   friend RealType getSplinedSOPot(SOECPComponent* so_pp, int l, double r);
   /** @brief Evaluate the spin orbit pp contribution 
@@ -70,31 +99,22 @@ public:
    *
    * @return RealType Contribution to $\frac{V\Psi_T}{\Psi_T}$ from ion iat and electron iel.
    */
-  inline RealType evaluateOne(ParticleSet& W,
-                       int iat,
-                       TrialWaveFunction& Psi,
-                       int iel,
-                       RealType r,
-                       const PosType& dr)
-                       {
-                         APP_ABORT("evaluateOne not implemented yet\n"); 
-                         return 0.0;
-                       };
+  RealType evaluateOne(ParticleSet& W, int iat, TrialWaveFunction& Psi, int iel, RealType r, const PosType& dr);
 
   // This function needs to be updated to SoA. myTableIndex is introduced temporarily.
   inline RealType evaluateValueAndDerivatives(ParticleSet& P,
-                                       int iat,
-                                       TrialWaveFunction& psi,
-                                       const opt_variables_type& optvars,
-                                       const std::vector<RealType>& dlogpsi,
-                                       std::vector<RealType>& dhpsioverpsi,
-                                       const int myTableIndex)
-                                       {
-                                         APP_ABORT("evaluateValueAndDerivatives not implemented yet\n");
-                                         return 0.0;
-                                       };
+                                              int iat,
+                                              TrialWaveFunction& psi,
+                                              const opt_variables_type& optvars,
+                                              const std::vector<RealType>& dlogpsi,
+                                              std::vector<RealType>& dhpsioverpsi,
+                                              const int myTableIndex)
+  {
+    APP_ABORT("evaluateValueAndDerivatives not implemented yet\n");
+    return 0.0;
+  };
 
-  void print(std::ostream& os){};
+  void print(std::ostream& os);
 
   //void initVirtualParticle(const ParticleSet& qp){};
 
@@ -102,9 +122,12 @@ public:
   inline RealType getRmax() const { return Rmax; }
   inline void setLmax(int Lmax) { lmax = Lmax; }
   inline int getLmax() const { return lmax; }
+  inline int getNknot() const { return nknot; }
+  inline int getSknot() const { return sknot; }
 
   friend class ECPComponentBuilder;
-}; 
+  friend void copyGridUnrotatedForTest(SOECPComponent& nlpp);
+};
 
 } // namespace qmcplusplus
 #endif

src/QMCHamiltonians/SOECPotential.cpp

@@ -0,0 +1,59 @@
+//////////////////////////////////////////////////////////////////////////////////////
+// This file is distributed under the University of Illinois/NCSA Open Source License.
+// See LICENSE file in top directory for details.
+//
+// Copyright (c) 2016 Jeongnim Kim and QMCPACK developers.
+//
+// File developed by: Ken Esler, kpesler@gmail.com, University of Illinois at Urbana-Champaign
+//                    Jeremy McMinnis, jmcminis@gmail.com, University of Illinois at Urbana-Champaign
+//                    Jeongnim Kim, jeongnim.kim@gmail.com, University of Illinois at Urbana-Champaign
+//                    Jaron T. Krogel, krogeljt@ornl.gov, Oak Ridge National Laboratory
+//                    Mark A. Berrill, berrillma@ornl.gov, Oak Ridge National Laboratory
+//
+// File created by: Cody A. Melton, cmelton@sandia.gov, Sandia National Laboratories
+//////////////////////////////////////////////////////////////////////////////////////
+
+#include "Particle/DistanceTableData.h"
+#include "QMCHamiltonians/SOECPotential.h"
+#include "Utilities/IteratorUtility.h"
+
+namespace qmcplusplus
+{
+/** constructor
+ *\param ionic positions
+ *\param els electronic poitions
+ *\param psi Trial wave function
+*/
+SOECPotential::SOECPotential(ParticleSet& ions, ParticleSet& els, TrialWaveFunction& psi)
+    : myRNG(&Random), IonConfig(ions), Psi(psi), Peln(els), ElecNeighborIons(els), IonNeighborElecs(ions)
+{
+  //set_energy_domain(potential);
+  //two_body_quantum_domain(ions,els);
+  myTableIndex = els.addTable(ions, DT_SOA_PREFERRED);
+  NumIons      = ions.getTotalNum();
+  PP.resize(NumIons, nullptr);
+  PPset.resize(IonConfig.getSpeciesSet().getTotalNum(), 0);
+  //UpdateMode.set(NONLOCAL,1);
+}
+
+//destructor
+SOECPotential::~SOECPotential() { delete_iter(PPset.begin(), PPset.end()); }
+
+void SOECPotential::resetTargetParticleSet(ParticleSet& P) {}
+
+SOECPotential::Return_t SOECPotential::evaluate(ParticleSet& P) { return 0.0; }
+
+OperatorBase* SOECPotential::makeClone(ParticleSet& qp, TrialWaveFunction& psi) { return 0; }
+
+void SOECPotential::addComponent(int groupID, SOECPComponent* ppot)
+{
+  for (int iat = 0; iat < PP.size(); iat++)
+    if (IonConfig.GroupID[iat] == groupID)
+    {
+      PP[iat] = ppot;
+    }
+  PPset[groupID] = ppot;
+}
+
+
+} // namespace qmcplusplus

src/QMCHamiltonians/SOECPotential.h

@@ -0,0 +1,68 @@
+//////////////////////////////////////////////////////////////////////////////////////
+// This file is distributed under the University of Illinois/NCSA Open Source License.
+// See LICENSE file in top directory for details.
+//
+// Copyright (c) 2016 Jeongnim Kim and QMCPACK developers.
+//
+// File developed by: Ken Esler, kpesler@gmail.com, University of Illinois at Urbana-Champaign
+//                    Jeremy McMinnis, jmcminis@gmail.com, University of Illinois at Urbana-Champaign
+//                    Jeongnim Kim, jeongnim.kim@gmail.com, University of Illinois at Urbana-Champaign
+//                    Jaron T. Krogel, krogeljt@ornl.gov, Oak Ridge National Laboratory
+//                    Mark A. Berrill, berrillma@ornl.gov, Oak Ridge National Laboratory
+//
+// File created by: Jeongnim Kim, jeongnim.kim@gmail.com, University of Illinois at Urbana-Champaign
+//////////////////////////////////////////////////////////////////////////////////////
+
+#ifndef QMCPLUSPLUS_SO_ECPOTENTIAL_H
+#define QMCPLUSPLUS_SO_ECPOTENTIAL_H
+
+#include "QMCHamiltonians/SOECPComponent.h"
+#include "Particle/NeighborLists.h"
+
+namespace qmcplusplus
+{
+class SOECPotential : public OperatorBase
+{
+public:
+  SOECPotential(ParticleSet& ions, ParticleSet& els, TrialWaveFunction& psi);
+
+  ~SOECPotential();
+
+  void resetTargetParticleSet(ParticleSet& P);
+
+  Return_t evaluate(ParticleSet& P);
+
+  bool put(xmlNodePtr cur) { return true; }
+
+  bool get(std::ostream& os) const
+  {
+    os << "SOECPotential: " << IonConfig.getName();
+    return true;
+  }
+
+  OperatorBase* makeClone(ParticleSet& qp, TrialWaveFunction& psi);
+
+  void addComponent(int groupID, SOECPComponent* pp);
+
+protected:
+  RandomGenerator_t* myRNG;
+  std::vector<SOECPComponent*> PP;
+  std::vector<SOECPComponent*> PPset;
+  ParticleSet& IonConfig;
+  TrialWaveFunction& Psi;
+
+private:
+  ///number of ions
+  int NumIons;
+  ///index of distance table for ion-el pair
+  int myTableIndex;
+  ///reference to the electrons
+  ParticleSet& Peln;
+  ///neighborlist of electrons
+  NeighborLists ElecNeighborIons;
+  ///neighborlist of ions
+  NeighborLists IonNeighborElecs;
+};
+} // namespace qmcplusplus
+
+#endif

src/QMCHamiltonians/observable_helper.h

@@ -25,7 +25,6 @@
 
 namespace qmcplusplus
 {
-const hsize_t h5_observable_type = H5T_NATIVE_DOUBLE;
 
 /** define observable_helper
  *
@@ -104,9 +103,9 @@ struct observable_helper
       hid_t p = H5Pcreate(H5P_DATASET_CREATE);
       H5Pset_chunk(p, rank, &mydims[0]);
       space1_id      = H5Screate_simple(rank, &mydims[0], &maxdims[0]);
-      value1_id      = H5Dcreate(data_id, "value", h5_observable_type, space1_id, p);
+      value1_id      = H5Dcreate(data_id, "value", H5T_NATIVE_DOUBLE, space1_id, p);
       hid_t memspace = H5Screate_simple(rank, &mydims[0], NULL);
-      herr_t ret     = H5Dwrite(value1_id, h5_observable_type, memspace, space1_id, H5P_DEFAULT, &zeros[0]);
+      herr_t ret     = H5Dwrite(value1_id, H5T_NATIVE_DOUBLE, memspace, space1_id, H5P_DEFAULT, &zeros[0]);
       H5Sclose(memspace);
       H5Pclose(p);
     }
@@ -178,7 +177,7 @@ struct observable_helper
       H5Sselect_hyperslab(space1_id, H5S_SELECT_SET, &offsets[0], NULL, &mydims[0], NULL);
       H5Dextend(value1_id, &curdims[0]);
       hid_t memspace = H5Screate_simple(rank, &mydims[0], NULL);
-      herr_t ret     = H5Dwrite(value1_id, h5_observable_type, memspace, space1_id, H5P_DEFAULT, first_v + lower_bound);
+      herr_t ret     = H5Dwrite(value1_id, H5T_NATIVE_DOUBLE, memspace, space1_id, H5P_DEFAULT, first_v + lower_bound);
       H5Sclose(memspace);
       curdims[0]++;
       offsets[0]++;

src/QMCHamiltonians/tests/CMakeLists.txt

@@ -40,11 +40,11 @@ EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} -E make_directory "${UTEST_DIR}")
 EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} -E copy "${CMAKE_CURRENT_SOURCE_DIR}/simple.txt" ${UTEST_DIR})
 EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} -E copy "${qmcpack_SOURCE_DIR}/tests/pseudopotentials_for_tests/C.BFD.xml" ${UTEST_DIR})
 EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} -E copy "${qmcpack_SOURCE_DIR}/tests/pseudopotentials_for_tests/Na.BFD.xml" ${UTEST_DIR})
-EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} -E copy "${qmcpack_SOURCE_DIR}/tests/pseudopotentials_for_tests/Zn.ccECP-SO.xml" ${UTEST_DIR})
+EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} -E copy "${qmcpack_SOURCE_DIR}/tests/pseudopotentials_for_tests/so_ecp_test.xml" ${UTEST_DIR})
 
 
 ADD_EXECUTABLE(${UTEST_EXE} ${SRCS})
-TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcham qmcwfs qmcbase qmcutil ${QMC_UTIL_LIBS} ${MPI_LIBRARY})
+TARGET_LINK_LIBRARIES(${UTEST_EXE} qmcham)
 
 #ADD_TEST(NAME ${UTEST_NAME} COMMAND "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")
 ADD_UNIT_TEST(${UTEST_NAME} "${QMCPACK_UNIT_TEST_DIR}/${UTEST_EXE}")

src/QMCHamiltonians/tests/soecp_eval_reference.cpp

@@ -0,0 +1,285 @@
+#include <iostream>
+#include <iomanip>
+#include <cmath>
+#include <vector>
+#include <complex>
+#include <omp.h>
+
+double PI = 4. * std::atan(1);
+typedef std::complex<double> dcomp;
+typedef std::vector<double> vec;
+
+//Same quadrature grid as in QMCPACK
+std::vector<vec> quad = {{1, 0, 0},
+                         {-1, 1.224646853e-16, 0},
+                         {0.4472135901, 0.8944271803, 0},
+                         {-0.4472135901, 0.7236068249, 0.5257310867},
+                         {0.4472135901, 0.2763932049, 0.8506507874},
+                         {-0.4472135901, -0.2763932049, 0.8506507874},
+                         {0.4472135901, -0.7236068249, 0.5257310867},
+                         {-0.4472135901, -0.8944271803, 1.095357398e-16},
+                         {0.4472135901, -0.7236068249, -0.5257310867},
+                         {-0.4472135901, -0.2763932049, -0.8506507874},
+                         {0.4472135901, 0.2763932049, -0.8506507874},
+                         {-0.4472135901, 0.7236068249, -0.5257310867}};
+
+vec wt = {0.08333333582, 0.08333333582, 0.08333333582, 0.08333333582, 0.08333333582, 0.08333333582,
+          0.08333333582, 0.08333333582, 0.08333333582, 0.08333333582, 0.08333333582, 0.08333333582};
+
+dcomp I = dcomp(0, 1);
+
+
+dcomp Ylm(int l, int m, const vec& sph)
+{
+  //From wiki
+  double pref;
+  switch (l)
+  {
+  case 0:
+    return dcomp(0.5 * std::sqrt(1. / PI), 0.0);
+    break;
+  case 1:
+    switch (m)
+    {
+    case -1:
+      pref = 0.5 * std::sqrt(3. / (2 * PI));
+      return pref * std::exp(-I * sph[2]) * std::sin(sph[1]);
+      break;
+    case 0:
+      pref = 0.5 * std::sqrt(3. / (PI));
+      return pref * std::cos(sph[1]);
+      break;
+    case 1:
+      pref = -0.5 * std::sqrt(3. / (2 * PI));
+      return pref * std::exp(I * sph[2]) * std::sin(sph[1]);
+      break;
+    default:
+      exit(1);
+      break;
+    }
+    break;
+  case 2:
+    switch (m)
+    {
+    case -2:
+      pref = 0.25 * std::sqrt(15. / (2 * PI));
+      return pref * std::exp(-2.0 * I * sph[2]) * std::sin(sph[1]) * std::sin(sph[1]);
+      break;
+    case -1:
+      pref = 0.5 * std::sqrt(15. / (2 * PI));
+      return pref * std::exp(-I * sph[2]) * std::sin(sph[1]) * std::cos(sph[1]);
+      break;
+    case 0:
+      pref = 0.25 * std::sqrt(5. / (PI));
+      return pref * (3. * std::cos(sph[1]) * std::cos(sph[1]) - 1.);
+      break;
+
+    case 1:
+      pref = -0.5 * std::sqrt(15. / (2 * PI));
+      return pref * std::exp(I * sph[2]) * std::sin(sph[1]) * std::cos(sph[1]);
+      break;
+    case 2:
+      pref = 0.25 * std::sqrt(15. / (2 * PI));
+      return pref * std::exp(2.0 * I * sph[2]) * std::sin(sph[1]) * std::sin(sph[1]);
+      break;
+    }
+  case 3:
+    switch (m)
+    {
+    case -3:
+      pref = 0.125 * std::sqrt(35. / PI);
+      return pref * std::exp(-3. * I * sph[2]) * std::pow(std::sin(sph[1]), 3);
+      break;
+    case -2:
+      pref = 0.25 * std::sqrt(105. / (2 * PI));
+      return pref * std::exp(-2. * I * sph[2]) * std::pow(std::sin(sph[1]), 2) * std::cos(sph[1]);
+      break;
+    case -1:
+      pref = 0.125 * std::sqrt(21. / PI);
+      return pref * std::exp(-I * sph[2]) * std::sin(sph[1]) * (5 * std::pow(std::cos(sph[1]), 2) - 1);
+      break;
+    case 0:
+      pref = 0.25 * std::sqrt(7. / PI);
+      return pref * (5 * std::pow(std::cos(sph[1]), 3) - 3. * std::cos(sph[1]));
+      break;
+    case 1:
+      pref = -0.125 * std::sqrt(21. / PI);
+      return pref * std::exp(I * sph[2]) * std::sin(sph[1]) * (5 * std::pow(std::cos(sph[1]), 2) - 1);
+      break;
+    case 2:
+      pref = 0.25 * std::sqrt(105. / (2 * PI));
+      return pref * std::exp(2. * I * sph[2]) * std::pow(std::sin(sph[1]), 2) * std::cos(sph[1]);
+      break;
+    case 3:
+      pref = -0.125 * std::sqrt(35. / PI);
+      return pref * std::exp(3. * I * sph[2]) * std::pow(std::sin(sph[1]), 3);
+      break;
+    }
+    break;
+  default:
+    exit(1);
+    break;
+  }
+}
+
+
+vec cart2sph(const vec& cart)
+{
+  //Physics convention
+  vec sph(3);
+  sph[0] = std::sqrt(cart[0] * cart[0] + cart[1] * cart[1] + cart[2] * cart[2]);
+  sph[1] = std::acos(cart[2] / sph[0]);
+  sph[2] = std::atan2(cart[1], cart[0]);
+  return sph;
+}
+
+vec sph2cart(const vec& sph)
+{
+  //Physics convention
+  vec cart(3);
+  cart[0] = sph[0] * std::sin(sph[1]) * std::cos(sph[2]);
+  cart[1] = sph[0] * std::sin(sph[1]) * std::sin(sph[2]);
+  cart[2] = sph[0] * std::cos(sph[1]);
+  return cart;
+}
+
+//spin representation
+dcomp chiu(double s) { return std::exp(I * s); }
+dcomp chid(double s) { return std::exp(-I * s); }
+
+//simple plane-wave
+dcomp orbu(const vec& sph)
+{
+  vec cart = sph2cart(sph);
+  return std::exp(I * (cart[0] + cart[1] + cart[2]));
+}
+dcomp orbd(const vec& sph)
+{
+  vec cart = sph2cart(sph);
+  return std::exp(2. * I * (cart[0] + cart[1] + cart[2]));
+}
+
+//single particle spinor
+dcomp spinor(const vec& sph, double s) { return orbu(sph) * chiu(s) + orbd(sph) * chid(s); }
+
+// <s1 | s_d | s2>
+dcomp sMatrixElement(double s1, double s2, int d)
+{
+  switch (d)
+  {
+  case 0:
+    return dcomp(std::cos(s1 + s2), 0.0);
+    break;
+  case 1:
+    return dcomp(std::sin(s1 + s2), 0.0);
+    break;
+  case 2:
+    return dcomp(0.0, std::sin(s1 - s2));
+    break;
+  default:
+    exit(1);
+    break;
+  }
+}
+
+int kroneckerDelta(int x, int y) { return (x == y) ? 1 : 0; }
+
+//<l m1 | l_d | l m2>
+dcomp lMatrixElement(int l, int m1, int m2, int d)
+{
+  double pref1, pref2, val;
+  switch (d)
+  {
+  case 0:
+    pref1 = std::sqrt(l * (l + 1) - m2 * (m2 + 1));
+    pref2 = std::sqrt(l * (l + 1) - m2 * (m2 - 1));
+    val   = 0.5 * (pref1 * kroneckerDelta(m1, m2 + 1) + pref2 * kroneckerDelta(m1, m2 - 1));
+    return dcomp(val, 0.0);
+    break;
+  case 1:
+    pref1 = std::sqrt(l * (l + 1) - m2 * (m2 - 1));
+    pref2 = std::sqrt(l * (l + 1) - m2 * (m2 + 1));
+    val   = 0.5 * (pref1 * kroneckerDelta(m1, m2 - 1) - pref2 * kroneckerDelta(m1, m2 + 1));
+    return dcomp(0.0, val);
+    break;
+  case 2:
+    return dcomp(m2 * kroneckerDelta(m1, m2), 0.0);
+    break;
+  default:
+    exit(1);
+    break;
+  }
+}
+
+//simple radial dependence. coded in so_ecp_test.xml
+//used in so_ecp_test.xml for each spin channel
+double Wso(int l, double r) { return exp(-l * r * r); }
+
+dcomp calcAngInt(const vec& sph1, double s1, double s2)
+{
+  dcomp angint(0.0, 0.0);
+  for (int i = 0; i < quad.size(); i++)
+  {
+    vec sph2 = cart2sph(quad[i]);
+    sph2[0] *= sph1[0]; //now scaled to appropriate distance
+
+    dcomp integrand(0.0, 0.0);
+    for (int l = 1; l <= 3; l++)
+    {
+      dcomp msum(0.0, 0.0);
+      for (int m1 = -l; m1 <= l; m1++)
+      {
+        for (int m2 = -l; m2 <= l; m2++)
+        {
+          dcomp ldots(0.0, 0.0);
+          for (int d = 0; d < 3; d++)
+            ldots += lMatrixElement(l, m1, m2, d) * sMatrixElement(s1, s2, d);
+          msum += std::conj(Ylm(l, m1, sph1)) * Ylm(l, m2, sph2) * ldots;
+        }
+      }
+      integrand += Wso(l, sph1[0]) * msum;
+    }
+    integrand *= spinor(sph2, s2) / spinor(sph1, s1);
+    angint += integrand * wt[i];
+  }
+  return angint;
+}
+
+void calcVal(int npts)
+{
+  vec cart1 = {0.138, -0.24, 0.216};
+  vec sph1  = cart2sph(cart1);
+  double s1 = 0.0;
+
+  double t1 = omp_get_wtime();
+
+  dcomp sint(0.0, 0.0);
+  double smin = 0.0;
+  double smax = 2 * PI;
+  double h    = (smax - smin) / npts;
+#pragma omp parallel for reduction(+ : sint)
+  for (int k = 1; k <= npts - 1; k += 2)
+  {
+    double s2    = smin + k * h;
+    dcomp angint = calcAngInt(sph1, s1, s2);
+    sint += 4 * h / 3. * angint;
+  }
+#pragma omp parallel for reduction(+ : sint)
+  for (int k = 2; k <= npts - 2; k += 2)
+  {
+    double s2    = smin + k * h;
+    dcomp angint = calcAngInt(sph1, s1, s2);
+    sint += 2 * h / 3. * angint;
+  }
+  sint += h / 3. * calcAngInt(sph1, s1, smin);
+  sint += h / 3. * calcAngInt(sph1, s1, smax);
+  double t2 = omp_get_wtime();
+  std::cout << npts << " " << std::setprecision(10) << std::real(sint) << " " << std::imag(sint) << " time: " << t2 - t1
+            << std::endl;
+}
+
+int main()
+{
+  for (int n = 2; n <= 500; n += 2)
+    calcVal(n);
+}

src/QMCHamiltonians/tests/test_ecp.cpp

@@ -17,6 +17,7 @@
 #include "Numerics/Quadrature.h"
 #include "QMCHamiltonians/ECPComponentBuilder.h"
 #include "QMCHamiltonians/NonLocalECPComponent.h"
+#include "QMCHamiltonians/SOECPComponent.h"
 
 //for wavefunction
 #include "OhmmsData/Libxml2Doc.h"
@@ -24,14 +25,21 @@
 #include "QMCWaveFunctions/TrialWaveFunction.h"
 #include "QMCWaveFunctions/Jastrow/BsplineFunctor.h"
 #include "QMCWaveFunctions/Jastrow/RadialJastrowBuilder.h"
+#include "QMCWaveFunctions/Fermion/DiracDeterminant.h"
+#include "QMCWaveFunctions/SpinorSet.h"
 //for nonlocal moves
 #include "QMCHamiltonians/NonLocalTOperator.h"
 
+
 //for Hamiltonian manipulations.
 #include "Particle/ParticleSet.h"
 #include "Particle/ParticleSetPool.h"
 #include "LongRange/EwaldHandler3D.h"
 
+#ifdef QMC_COMPLEX //This is for the spinor test.
+#include "QMCWaveFunctions/ElectronGas/ElectronGasComplexOrbitalBuilder.h"
+#endif
+
 namespace qmcplusplus
 {
 QMCTraits::RealType getSplinedSOPot(SOECPComponent* so_comp, int l, double r) { return so_comp->sopp_m[l]->splint(r); }
@@ -105,15 +113,15 @@ TEST_CASE("ReadFileBuffer_sorep", "[hamiltonian]")
 
   ECPComponentBuilder ecp("test_read_sorep", c);
 
-  bool okay = ecp.read_pp_file("Zn.ccECP-SO.xml");
+  bool okay = ecp.read_pp_file("so_ecp_test.xml");
   REQUIRE(okay);
 
-  REQUIRE(ecp.Zeff == 20);
+  REQUIRE(ecp.Zeff == 13);
 
   double rlist[5] = {0.001, 0.500, 1.000, 2.000, 10.000};
-  double so_p[5]  = {0.0614288376917, 0.10399457248, 4.85269969439e-06, 4.6722444395e-25, 0.000};
-  double so_d[5]  = {0.0850898886265, 0.0029447669325, 6.35734161822e-08, 2.8386702794e-27, 0.000};
-  double so_f[5]  = {-0.284560515732, 0.0071131554209, 6.79818097092e-05, 1.64868282163e-15, 0.000};
+  double so_p[5]  = {0.999999000005, 0.778800783071, 0.3678794411714, 0.01831563888873418, 0.000};
+  double so_d[5]  = {9.99998000e-01, 6.06530660e-01, 1.35335283e-01, 3.35462628e-04, 0.000};
+  double so_f[5]  = {9.99997000e-01, 4.72366553e-01, 4.97870684e-02, 6.14421235e-06, 0.000};
 
   for (int i = 0; i < 5; i++)
   {
@@ -157,6 +165,7 @@ TEST_CASE("ReadFileBuffer_reopen", "[hamiltonian]")
 }
 
 void copyGridUnrotatedForTest(NonLocalECPComponent& nlpp) { nlpp.rrotsgrid_m = nlpp.sgridxyz_m; }
+void copyGridUnrotatedForTest(SOECPComponent& sopp) { sopp.rrotsgrid_m = sopp.sgridxyz_m; }
 
 TEST_CASE("Evaluate_ecp", "[hamiltonian]")
 {
@@ -404,8 +413,8 @@ TEST_CASE("Evaluate_ecp", "[hamiltonian]")
       if (nlpp != nullptr && dist[iat] < nlpp->getRmax())
       {
         Value2 += nlpp->evaluateOneWithForces(elec, iat, psi, jel, dist[iat], -displ[iat], HFTerm[iat]);
-        Value3 += nlpp->evaluateOneWithForces(elec, ions, iat, psi, jel, dist[iat], -displ[iat], HFTerm2[iat],
-                                              PulayTerm);
+        Value3 +=
+            nlpp->evaluateOneWithForces(elec, ions, iat, psi, jel, dist[iat], -displ[iat], HFTerm2[iat], PulayTerm);
       }
   }
   //These values are validated against print statements.
@@ -455,4 +464,148 @@ TEST_CASE("Evaluate_ecp", "[hamiltonian]")
   //HFTerm[1][2]+PulayTerm[1][2] =  0.0
 #endif
 }
+
+#ifdef QMC_COMPLEX
+TEST_CASE("Evaluate_soecp", "[hamiltonian]")
+{
+  app_log() << "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n";
+  app_log() << "!!!! Evaluate SOECPComponent !!!!\n";
+  app_log() << "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n";
+  typedef QMCTraits::RealType RealType;
+  typedef QMCTraits::ValueType ValueType;
+  typedef QMCTraits::PosType PosType;
+
+  OHMMS::Controller->initialize(0, NULL);
+  Communicate* c = OHMMS::Controller;
+
+  //Cell definition:
+
+  CrystalLattice<OHMMS_PRECISION, OHMMS_DIM> Lattice;
+  Lattice.BoxBConds = false; // periodic
+  Lattice.R.diagonal(20);
+  Lattice.LR_dim_cutoff = 15;
+  Lattice.reset();
+
+
+  ParticleSet ions;
+  ParticleSet elec;
+
+  ions.setName("ion0");
+  ions.create(1);
+  ions.R[0][0] = 0.0;
+  ions.R[0][1] = 0.0;
+  ions.R[0][2] = 0.0;
+
+
+  SpeciesSet& ion_species       = ions.getSpeciesSet();
+  int pIdx                      = ion_species.addSpecies("H");
+  int pChargeIdx                = ion_species.addAttribute("charge");
+  int iatnumber                 = ion_species.addAttribute("atomic_number");
+  ion_species(pChargeIdx, pIdx) = 0;
+  ion_species(iatnumber, pIdx)  = 1;
+  ions.Lattice                  = Lattice;
+  ions.createSK();
+
+
+  elec.Lattice = Lattice;
+  elec.setName("e");
+  elec.create(1);
+  elec.R[0][0]  = 0.138;
+  elec.R[0][1]  = -0.24;
+  elec.R[0][2]  = 0.216;
+  elec.spins[0] = 0.0;
+
+  SpeciesSet& tspecies       = elec.getSpeciesSet();
+  int upIdx                  = tspecies.addSpecies("u");
+  int chargeIdx              = tspecies.addAttribute("charge");
+  int massIdx                = tspecies.addAttribute("mass");
+  tspecies(chargeIdx, upIdx) = -1;
+  tspecies(massIdx, upIdx)   = 1.0;
+
+  elec.createSK();
+
+  ParticleSetPool ptcl = ParticleSetPool(c);
+  ptcl.addParticleSet(&elec);
+  ptcl.addParticleSet(&ions);
+
+  ions.resetGroups();
+  elec.resetGroups();
+
+  TrialWaveFunction psi(c);
+
+  std::vector<PosType> kup, kdn;
+  std::vector<RealType> k2up, k2dn;
+  QMCTraits::IndexType nelec = elec.getTotalNum();
+  REQUIRE(nelec == 1);
+
+  kup.resize(nelec);
+  kup[0] = PosType(1, 1, 1);
+
+  k2up.resize(nelec);
+  //For some goofy reason, EGOSet needs to be initialized with:
+  //1.) A k-vector list (fine).
+  //2.) A list of -|k|^2.  To save on expensive - sign multiplication apparently.
+  k2up[0] = -dot(kup[0], kup[0]);
+
+  kdn.resize(nelec);
+  kdn[0] = PosType(2, 2, 2);
+
+  k2dn.resize(nelec);
+  k2dn[0] = -dot(kdn[0], kdn[0]);
+
+  std::shared_ptr<EGOSet> spo_up(new EGOSet(kup, k2up));
+  std::shared_ptr<EGOSet> spo_dn(new EGOSet(kdn, k2dn));
+
+  SpinorSet* spinor_set = new SpinorSet();
+  spinor_set->set_spos(spo_up, spo_dn);
+
+  DiracDeterminant<>* dd    = new DiracDeterminant<>(spinor_set);
+  QMCTraits::IndexType norb = spo_up->size();
+  REQUIRE(norb == 1);
+  dd->resize(nelec, norb);
+
+  psi.addComponent(dd, "spinor");
+
+  //Now we set up the SO ECP component.
+  ECPComponentBuilder ecp("test_read_soecp", c);
+
+  bool okay3 = ecp.read_pp_file("so_ecp_test.xml");
+  REQUIRE(okay3);
+
+  SOECPComponent* sopp = ecp.pp_so;
+  REQUIRE(sopp != nullptr);
+  copyGridUnrotatedForTest(*sopp);
+
+  const int myTableIndex = elec.addTable(ions, DT_SOA_PREFERRED);
+
+  const auto& myTable = elec.getDistTable(myTableIndex);
+
+  // update all distance tables
+  ions.update();
+  elec.update();
+
+  //Need to set up temporary data for this configuration in trial wavefunction.  Needed for ratios.
+  double logpsi = psi.evaluateLog(elec);
+
+  RealType Value1(0.0);
+
+#ifdef ENABLE_SOA
+  for (int jel = 0; jel < elec.getTotalNum(); jel++)
+  {
+    const auto& dist  = myTable.getDistRow(jel);
+    const auto& displ = myTable.getDisplRow(jel);
+    for (int iat = 0; iat < ions.getTotalNum(); iat++)
+    {
+      if (sopp != nullptr && dist[iat] < sopp->getRmax())
+      {
+        Value1 += sopp->evaluateOne(elec, iat, psi, jel, dist[iat], RealType(-1) * displ[iat]);
+      }
+    }
+  }
+  REQUIRE(Value1 == Approx(0.1644374207));
+#endif
+}
+#endif
+
+
 } // namespace qmcplusplus

src/QMCHamiltonians/tests/test_force.cpp

@@ -75,7 +75,7 @@ TEST_CASE("Bare Force", "[hamiltonian]")
   ion_species(pMembersizeIdx, pIdx) = 1;
 
   ions.resetGroups();
-  // Must update ions first in SoA so ions.RSoA is valid
+  // Must update ions first in SoA so ions.coordinates_ is valid
   ions.update();
 
 #ifdef ENABLE_SOA

src/QMCTools/CMakeLists.txt

@@ -34,49 +34,41 @@ SET(MOSRCS
   GamesFMOParser.cpp
   QPParser.cpp
   LCAOHDFParser.cpp
-  convertpw4qmc.cpp 
-  WriteEshdf.cpp
-  XmlRep.cpp
 )
 
 # create libmocommon 
 ADD_LIBRARY(mocommon ${MOSRCS})
+TARGET_LINK_LIBRARIES(mocommon PUBLIC qmcbase)
 
 set(QTOOLS convert4qmc qmc-extract-eshdf-kvectors)
-ADD_EXECUTABLE(qmc-get-supercell getSupercell.cpp)
-INSTALL(TARGETS qmc-get-supercell
-        RUNTIME DESTINATION bin)
-ADD_EXECUTABLE(qmc-check-affinity check-affinity.cpp)
-INSTALL(TARGETS qmc-check-affinity
-        RUNTIME DESTINATION bin)
-
-ADD_EXECUTABLE(convertpw4qmc convertpw4qmc.cpp XmlRep.cpp WriteEshdf.cpp)
-FOREACH(l ${QMC_UTIL_LIBS})		       
-  TARGET_LINK_LIBRARIES(convertpw4qmc ${l})
-ENDFOREACH(l ${QMC_UTIL_LIBS})
-
-INSTALL(TARGETS convertpw4qmc
-        RUNTIME DESTINATION bin)
-
-
 FOREACH(p ${QTOOLS})
 
   ADD_EXECUTABLE(${p} ${p}.cpp)
-  TARGET_LINK_LIBRARIES(${p} mocommon qmcbase qmcutil)
+  TARGET_LINK_LIBRARIES(${p} PRIVATE mocommon)
 
   INSTALL(TARGETS ${p}
           RUNTIME DESTINATION bin)
 
-  FOREACH(l ${QMC_UTIL_LIBS})
-    TARGET_LINK_LIBRARIES(${p} ${l})
-  ENDFOREACH(l ${QMC_UTIL_LIBS})
-
-  IF(MPI_LIBRARY)
-    TARGET_LINK_LIBRARIES(${p} ${MPI_LIBRARY})
-  ENDIF(MPI_LIBRARY)
-
 ENDFOREACH(p ${QTOOLS})
 
+ADD_EXECUTABLE(qmc-get-supercell getSupercell.cpp)
+INSTALL(TARGETS qmc-get-supercell
+        RUNTIME DESTINATION bin)
+
+ADD_EXECUTABLE(qmc-check-affinity check-affinity.cpp)
+IF(HAVE_MPI)
+  TARGET_LINK_LIBRARIES(qmc-check-affinity PRIVATE MPI::MPI_CXX)
+ENDIF()
+
+INSTALL(TARGETS qmc-check-affinity
+        RUNTIME DESTINATION bin)
+
+ADD_EXECUTABLE(convertpw4qmc convertpw4qmc.cpp XmlRep.cpp WriteEshdf.cpp)
+TARGET_LINK_LIBRARIES(convertpw4qmc PRIVATE qmcutil Math::FFTW3)
+
+INSTALL(TARGETS convertpw4qmc
+        RUNTIME DESTINATION bin)
+
 #ppconvert
 if( BUILD_PPCONVERT )
   SUBDIRS(ppconvert)

src/QMCTools/ppconvert/src/CMakeLists.txt

@@ -11,10 +11,4 @@ target_sources(ppconvert
     "${CMAKE_CURRENT_SOURCE_DIR}/ParserClass.cc"
 )
 
-target_link_libraries(ppconvert common ${LAPACK_LIBRARIES})
-set_property(TARGET ppconvert PROPERTY INSTALL_RPATH "${LAPACK_LINK_DIRECTORIES}")
-set_property(TARGET ppconvert PROPERTY BUILD_WITH_INSTALL_RPATH ON)
-set_property(TARGET ppconvert PROPERTY LINK_FLAGS "${LAPACK_LINKER_FLAGS}")
-target_include_directories(ppconvert PUBLIC
-  ${LAPACK_INCLUDE_DIRECTORIES}
-  )
+target_link_libraries(ppconvert common Math::BLAS_LAPACK)

src/QMCWaveFunctions/CMakeLists.txt

@@ -230,9 +230,12 @@ ENDIF(QMC_CUDA OR ENABLE_CUDA)
 
 IF(HAVE_CUDA)
   CUDA_ADD_CUBLAS_TO_TARGET(qmcwfs)
-  TARGET_LINK_LIBRARIES( qmcwfs ${CUDA_cusolver_LIBRARY} )
+  TARGET_LINK_LIBRARIES( qmcwfs PRIVATE ${CUDA_cusolver_LIBRARY} )
 ENDIF(HAVE_CUDA)
 
+TARGET_LINK_LIBRARIES(qmcwfs PUBLIC qmcbase)
+TARGET_LINK_LIBRARIES(qmcwfs PRIVATE einspline Math::BLAS_LAPACK Math::FFTW3 Math::scalar_vector_functions)
+
 #IF(QMC_BUILD_STATIC)
 #  ADD_LIBRARY(qmcwfs STATIC ${WFBASE_SRCS} ${JASTROW_SRCS} ${FERMION_SRCS})
 #ELSE(QMC_BUILD_STATIC)