These are my raw notes when installing OpenFOAM and FOAM-extend. They are mostly for myself, but made available for anyone who would like to use them.
General installation instructions can be found at http://openfoamwiki.net/index.php/Installation
First, a separate page (and later an ugly copy-paste of notes):
Install Ubuntu 14.04 LTS, OpenFOAM-2.3.x, foam-extend-3.1 etc. on your own laptop
Table of contents:
Count number of files in a directory:
find . -type f -print | wc -l
or:
tree
Check size of directories:
du -s *|sort -n
It is not useful to try to make a common installation of ThirdParty products for different installations of OpenFOAM/FOAM-extend. The binary files donot take that make space, so it is better just to remove intermediate files and source code directories. See below for how to do that for different versions.
wmake does not necessarily figure out exactly what needs to be recompiled, so recompile everything to be on the safe side:
cd $WM_PROJECT_DIR
./Allclean
(cd src; find . -name \*.o | xargs -n 1 rm)
(cd applications; find . -name \*.o | xargs -n 1 rm)
./Allwmake
Cleanup of ThirdParty after installation
Once compiled (with Allwmake.firstInstall), the only important ThirdParty directory you need to keep as is for running foam-extend is ThirdParty/packages.
The rest can be cleaned up using two methods:
Cleanup using scripts (if those scripts are updated with the presently installed package versions):
ThirdParty/AllClean or ThirdParty/AllClean.stage[1-6]
Will remove all unnecessary files from the ThirdParty packages once your installation/compilation of the packages is completed
If those scripts are called with the command-line parameter 'alsoPackage', the installed packages will be removed too, so you will have to recompile them to get a usable installation of foam-extend.
If the ThirdParty/AllClean.stage[1-6] are not up-to-date (with the presently installed package versions), then some files will need to be cleaned up manually...
Manual cleanup:
ThirdParty/rpmBuild/BUILD : remove everything inside directory
ThirdParty/rpmBuild/BUILDROOT : remove everything inside directory
ThirdParty/rpmBuild/TGZS : remove everything inside directory
ThirdParty/rpmBuild/RPMS : remove everything inside directory. Will get regenerated if you recompile the ThirdParty packages
ThirdParty/rpmBuild/SOURCES : remove every compressed *.tar.* files, but only those. They will get downloaded once more if you recompile the ThirdParty packages
If you found out you've inadvertently removed too much files, the command "git checkout theDirectoryWhereYouRemovedTooMuch" will get back all the necessary files you are missing
Recommended:
The easy way:
rm -r rpmBuild/BUILD/*
rm -r rpmBuild/BUILDROOT/*
rm -r rpmBuild/TGZS/*
rm -r rpmBuild/RPMS/*
rm rpmBuild/SOURCES/*tar*
Note that one should not use Allwmake.firstInstall again, unless it is desired to recompile the ThirdParty packages. Instead use Allwmake.
hlwoc
The hwloc package is providing the command 'lstopo' for reliably detecting the number of processor cores on a given machine. If the command 'lstopo' is not found, a less reliable fallback solution will be used. See the script 'bin/foamGetSystemInfo' for the details.
Cleanup of ThirdParty after installation
New, in v2112:
cd $WM_THIRD_PARTY_DIR
./Allclean -build -src
In fact, it is recommended to keep only the platforms directory, as described below.
Old:
The following only cleans up intermediate files from compilation, but will keep the enormous amount of source code:
cd $WM_THIRD_PARTY_DIR
./Allclean
The following will also clean up the installed packages (binaries, libraries, include files), which is not usually something one wants to do (therefore commented):
#Commented, since you probably don't want to do this:
#./Allclean -all
Recommended:
Only the "platforms" directory needs to be kept after a successful installation. Just remove the rest, by (where the "dummy" shows how to keep more than one directory if necessary):
cd $WM_THIRD_PARTY_DIR
###############################################################################################
# MAKE SURE YOU ARE IN THE ThirdParty DIRECTORY, SINCE THIS IS A DANGEROUS RECURSIVE COMMAND!!!
# IT IS COMMENTED HERE, SO YOU DON'T MAKE ANY COPY-PASTE MISTAKE
###############################################################################################
#shopt -s extglob
#rm -rv !("platforms"|"dummy")
Find available versions in /chalmers/sw/sup64/gcc*, and then select the one needed, as:
vcs-select -p gcc-8.1.0
It will then be used for all new terminal windows.
However, I had problems using it. One thing that may be needed is to also do:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/chalmers/sw/sup64/gcc-8.1.0/lib64
A hint from support is to do:
export LDFLAGS="-Wl,-rpath=/chalmers/sw/sup64/gcc-8.1.0/lib64 $LDFLAGS"
The flag -Wl,xxx will make gcc forward xxx as additional arguments to ld when linking. The flag -rpath makes ld add to the runpath of the binaries.
For OpenFOAM, it may be better to add the required Gcc in ThirdParty. See e.g. installation of OpenFOAM-v1906 below.
Unselect by:
vcs-select -u -p gcc-8.1.0
It is presently necessary to specify:
#SBATCH -C ICELAKE
Script for submitting the dambreak case from vera2:
#!/usr/bin/env bash
#SBATCH -A C3SE2022-1-9
#SBATCH -n 4
#SBATCH -t 01:00:00
#SBATCH -C ICELAKE
#-----------------------------------------------------------
module load OpenFOAM/v2112-foss-2021b
. $FOAM_BASH
#------------ Copy and run the damBreak tutorial -----------
cp -r $FOAM_TUTORIALS/multiphase/interFoam/RAS/damBreak/damBreak .
cd damBreak
sed -i s/"writeFormat\s*ascii"/"writeFormat binary"/g system/controlDict
cp -r 0.orig 0
blockMesh #Generates the grid
setFields #Sets the initial fields
decomposePar #Decomposes for parallel simulation
mpirun interFoam -parallel #Runs interFoam in parallel
reconstructPar #Collects the decomposed results
#rm -rf processor* #Deletes the decomposed files
cd /cephyr/NOBACKUP/groups/openfoam/foam
git clone -b nextRelease https://git.code.sf.net/p/foam-extend/foam-extend-4.1 foam-extend-4.1_NR_2d99850ee_workingMixingPlane
cd foam-extend-4.1_NR_2d99850ee_workingMixingPlane
git checkout 2d99850ee
sed -i.orig s/"5\.0$"/"4\.1_NR_2d99850ee_workingMixingPlane"/g etc/bashrc
echo "export PARAVIEW_SYSTEM=1" >> etc/prefs.sh #Avoid having to type "Y" at compilation
Fix problem with gold linker (see comments for previous installations):
sed -i s/fPIC/"fPIC -fuse-ld=bfd"/g wmake/rules/linux64Gcc/c++
Use system OpenMPI:
echo "export WM_MPLIB=SYSTEMOPENMPI" >> etc/prefs.sh
echo "export OPENMPI_DIR=$EBROOTOPENMPI" >> etc/prefs.sh
echo "export OPENMPI_BIN_DIR=$EBROOTOPENMPI/bin" >> etc/prefs.sh
Revert some default setting, to make METIS etc work (don't use system installation):
sed -i s/"export METIS_SYSTEM=1"/"#export METIS_SYSTEM=1"/g etc/bashrc
sed -i s/"export WM_THIRD_PARTY_USE_PARMGRIDGEN_10=1"/"#export WM_THIRD_PARTY_USE_PARMGRIDGEN_10=1"/g etc/bashrc
sed -i s/"export SCOTCH_SYSTEM=1"/"#export SCOTCH_SYSTEM=1"/g etc/bashrc
sed -i s/"export PARMETIS_SYSTEM=1"/"#export PARMETIS_SYSTEM=1"/g etc/bashrc
sed -i s/"export WM_THIRD_PARTY_USE_PYFOAM_069=1"/"#export WM_THIRD_PARTY_USE_PYFOAM_069=1"/g etc/bashrc
sed -i s/"export WM_THIRD_PARTY_USE_HWLOC_201=1"/"#export WM_THIRD_PARTY_USE_HWLOC_201=1"/g etc/bashrc
echo "export WM_THIRD_PARTY_USE_METIS_510=1" >> etc/prefs.sh
echo "export WM_THIRD_PARTY_USE_PARMGRIDGEN_10=1" >> etc/prefs.sh
echo "export WM_THIRD_PARTY_USE_PARMETIS_403=1" >> etc/prefs.sh
echo "export WM_THIRD_PARTY_USE_SCOTCH_604=1" >> etc/prefs.sh
The first URL for mesquite in AllMake.stage3 gives a tar.gz file that is not of correct format. The second (commented) URL is ok. However, I anyhow get some compilation error for meshquite later. I don't care since we are not using it.
rpmbuild was not available at Vera at the time of compilation. Martin Beaudoin supplied me with a script that installed it and all its dependencies. Before running it, change the INSTALL_DIR at the start of the script to a preferred location and add that path to PATH before compiling.
#!/bin/bash
INSTALL_DIR=/cephyr/NOBACKUP/groups/openfoam/rpmbuild
# Cleanup
rm -rf popt-1.15 beecrypt-4.1.2 pcre2-10.42 rpm-5.2.1
#Mandatory package: popt
wget -N https://launchpad.net/popt/head/1.15/+download/popt-1.15.tar.gz
tar -xvf popt-1.15.tar.gz
cd popt-1.15
./configure --prefix=$INSTALL_DIR
make install
cd ..
#Mandatory package: beecrypt
wget -N https://downloads.sourceforge.net/project/beecrypt/beecrypt/4.1.2/beecrypt-4.1.2.tar.gz
tar -xvf beecrypt-4.1.2.tar.gz
cd beecrypt-4.1.2
./configure --without-java --without-cplusplus --prefix=$INSTALL_DIR
make install
cd ..
#Mandatory package: pcre
wget -N https://downloads.sourceforge.net/project/pcre/pcre/7.9/pcre-7.9.tar.gz
tar -xvf pcre-7.9.tar.gz
cd pcre-7.9
./configure --prefix=$INSTALL_DIR
make install
cd ..
# Compile rpmbuild version 5.2.1
wget -N https://launchpad.net/rpm/5.2/5.2.1/+download/rpm-5.2.1.tar.gz
tar -xvf rpm-5.2.1.tar.gz
cd rpm-5.2.1
#Need to patch this file...
sed -ibak 's/!defined(__LCLINT__)/defined(DONT_SKIP_THIS) \&\& !defined(__LCLINT__)/' rpmio/rpmio.h
LDFLAGS="-L$INSTALL_DIR/lib64 -L$INSTALL_DIR/lib" CPPFLAGS=-I$INSTALL_DIR/include ./configure --with-beecrypt=$INSTALL_DIR/lib64 --with-pcre=$INSTALL_DIR/lib --without-neon --prefix=$INSTALL_DIR
make
make install
cd ..
# Test
export PATH=$INSTALL_DIR/bin:$PATH
which rpmbuild
rpmbuild --version
Compile at Vera using following submit-script (including path to rpmbuild):
#!/usr/bin/env bash
#SBATCH -A C3SE408-18-1 -p vera
#SBATCH -n 32
#SBATCH -t 4:00:00
#------------------- Environment --------------------
ml foss/2023a
ml CMake/3.26.3-GCCcore-12.3.0
ml flex/2.6.4
export PATH=/cephyr/NOBACKUP/groups/openfoam/rpmbuild/bin:$PATH
export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/openfoam/foam
. $FOAM_INST_DIR/foam-extend-4.1_NR_2d99850ee_workingMixingPlane/etc/bashrc
cd $FOAM_INST_DIR/foam-extend-4.1_NR_2d99850ee_workingMixingPlane
#--------------------- Compile ----------------------
./Allwmake.firstInstall -j >& log_compile
Submit the script again to make sure that everything is compiled. meshquite and things that depend on it failed, as well as vectorSnapshots.C, but I don't bother since we are not using them.
Change rights:
chgrp -R pg_openfoam /cephyr/NOBACKUP/groups/openfoam/foam/foam-extend-4.1_NR_2d99850ee_workingMixingPlane
chmod -R go+Xr /cephyr/NOBACKUP/groups/openfoam/foam/foam-extend-4.1_NR_2d99850ee_workingMixingPlane
Submit script:
#!/usr/bin/env bash
#SBATCH -A C3SE408-18-1 -p vera
#SBATCH -n 4
#SBATCH -t 1:00:00
#------------------- Environment --------------------
export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/openfoam/foam
. $FOAM_INST_DIR/foam-extend-4.1_NR_2d99850ee_workingMixingPlane/etc/bashrc
#------------ Copy and run the damBreak tutorial -----------
cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak .
cd damBreak
blockMesh #Generates the grid
setFields #Sets the initial fields
decomposePar #Decomposes for parallel simulation
mpirun -np 4 interFoam -parallel #Runs interFoam in parallel
reconstructPar #Collects the decomposed results
#rm -rf processor* #Deletes the decomposed files
For Paraview, in a separate terminal without OpenFOAM (only in Thinlinc):
paraview_ml
vglrun paraview
Clean up (see top of page):
cd ThirdParty
./AllClean
rm -r rpmBuild/BUILD/*
rm -r rpmBuild/BUILDROOT/*
rm -r rpmBuild/TGZS/*
rm -r rpmBuild/RPMS/*
rm rpmBuild/SOURCES/*tar*
foam
wcleanAllButLibBinLnInclude
cd /proj/hani/OF_inst/Tetralith
mkdir OpenFOAM
cd OpenFOAM/
wget https://sourceforge.net/projects/openfoam/files/v2112/OpenFOAM-v2112.tgz
wget https://sourceforge.net/projects/openfoam/files/v2112/ThirdParty-v2112.tgz
tar xzf OpenFOAM-v2112.tgz
tar xzf ThirdParty-v2112.tgz
rm OpenFOAM-v2112.tgz ThirdParty-v2112.tgz
cd OpenFOAM-v2112
sed -i s/fPIC/"fPIC -fuse-ld=bfd"/g wmake/rules/linux64Gcc/c++
Compilation in parallel through submit-script:
#!/usr/bin/env bash
#SBATCH -A snic2021-3-10
#SBATCH -n 32
#SBATCH -t 4:00:00
#-----------------------------------------------------------
ml buildtool-easybuild/4.5.3-nsce8837e7 foss/2020b
export MPI_ROOT=$EBROOTOPENMPI
export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/OpenFOAM
. $FOAM_INST_DIR/OpenFOAM-v2112/etc/bashrc WM_MPLIB=SYSTEMOPENMPI
cd $FOAM_INST_DIR/OpenFOAM-v2112
#--------------------- Run simulation ----------------------
./Allwmake -j >& log_compile_OFv2112
NOTE: ADIOS, CGAL and kahip failed, in the ThirdParty packages.
Test script:
#!/usr/bin/env bash
#SBATCH -A snic2021-3-10
#SBATCH -n 4
#SBATCH -t 01:00:00
#-----------------------------------------------------------
ml buildtool-easybuild/4.5.3-nsce8837e7 foss/2020b
export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/OpenFOAM
. $FOAM_INST_DIR/OpenFOAM-v2112/etc/bashrc WM_MPLIB=SYSTEMOPENMPI
export FOAM_FILEHANDLER=collated #See https://www.openfoam.com/releases/openfoam-v1712/parallel.php
#------------ Copy and run the damBreak tutorial -----------
cp -r $FOAM_TUTORIALS/multiphase/interFoam/RAS/damBreak/damBreak .
cd damBreak
cp -r 0.orig 0
sed -i s/"writeFormat\s*ascii"/"writeFormat binary"/g system/controlDict
blockMesh #Generates the grid
setFields #Sets the initial fields
decomposePar #Decomposes for parallel simulation
mpirun interFoam -parallel #Runs interFoam in parallel
reconstructPar #Collects the decomposed results
Post-process in another terminal window:
paraview_ml
touch case.foam
vglrun paraview #Read in case.foam
Create aliases (in addToBashrc):
alias OFv2112hani="ml buildtool-easybuild/4.5.3-nsce8837e7 foss/2020b; . /proj/hani/OF_inst/Tetralith/OpenFOAM/OpenFOAM-v2112/etc/bashrc WM_MPLIB=SYSTEMOPENMPI; export FOAM_FILEHANDLER=collated"
Make available:
chmod -R go+Xr OpenFOAM-v2112
chmod -R go+Xr ThirdParty-v2112
Clean ThirdParty (see notes at top of page):
Only the "platforms" directory needs to be kept after a successful installation. Just remove the rest, by (where the "dummy" shows how to keep more than one directory if necessary):
cd $WM_THIRD_PARTY_DIR
###############################################################################################
# MAKE SURE YOU ARE IN THE ThirdParty DIRECTORY, SINCE THIS IS A DANGEROUS RECURSIVE COMMAND!!!
# IT IS COMMENTED HERE, SO YOU DON'T MAKE ANY COPY-PASTE MISTAKE
###############################################################################################
#shopt -s extglob
#rm -rv !("platforms"|"dummy")
cd /cephyr/NOBACKUP/groups/openfoam/hebbe
mkdir OpenFOAM
cd OpenFOAM/
wget https://sourceforge.net/projects/openfoam/files/v2112/OpenFOAM-v2112.tgz
wget https://sourceforge.net/projects/openfoam/files/v2112/ThirdParty-v2112.tgz
tar xzf OpenFOAM-v2112.tgz
tar xzf ThirdParty-v2112.tgz
rm OpenFOAM-v2112.tgz ThirdParty-v2112.tgz
cd OpenFOAM-v2112
sed -i s/fPIC/"fPIC -fuse-ld=bfd"/g wmake/rules/linux64Gcc/c++
Compilation in parallel through submit-script:
#!/usr/bin/env bash
#SBATCH -A C3SE408-18-1 -p vera
#SBATCH -n 32
#SBATCH -t 4:00:00
#-----------------------------------------------------------
ml foss/2021b
export MPI_ROOT=$EBROOTOPENMPI
export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/openfoam/hebbe/OpenFOAM
. $FOAM_INST_DIR/OpenFOAM-v2112/etc/bashrc WM_MPLIB=SYSTEMOPENMPI
cd $FOAM_INST_DIR/OpenFOAM-v2112
#--------------------- Run simulation ----------------------
./Allwmake -j >& log_compile_OFv2112
Compilation in sequential through submit-script:
#!/usr/bin/env bash
#SBATCH -A C3SE408-18-1 -p vera
#SBATCH -n 1
#SBATCH -t 168:00:00
#-----------------------------------------------------------
ml foss/2021b
export MPI_ROOT=$EBROOTOPENMPI
export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/openfoam/hebbe/OpenFOAM
. $FOAM_INST_DIR/OpenFOAM-v2112/etc/bashrc WM_MPLIB=SYSTEMOPENMPI
cd $FOAM_INST_DIR/OpenFOAM-v2112
#--------------------- Run simulation ----------------------
./Allwmake >& log_compile_OFv2112
NOTE: ADIOS and kahip failed, in the ThirdParty packages.
Test script:
#!/usr/bin/env bash
#SBATCH -A C3SE408-18-1 -p vera
#SBATCH -n 4
#SBATCH -t 01:00:00
#-----------------------------------------------------------
export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/openfoam/hebbe/OpenFOAM
. $FOAM_INST_DIR/OpenFOAM-v2112/etc/bashrc WM_MPLIB=SYSTEMOPENMPI
export FOAM_FILEHANDLER=collated #See https://www.openfoam.com/releases/openfoam-v1712/parallel.php
#------------ Copy and run the damBreak tutorial -----------
cp -r $FOAM_TUTORIALS/multiphase/interFoam/RAS/damBreak/damBreak .
cd damBreak
cp -r 0.orig 0
sed -i s/"writeFormat\s*ascii"/"writeFormat binary"/g system/controlDict
blockMesh #Generates the grid
setFields #Sets the initial fields
decomposePar #Decomposes for parallel simulation
mpirun interFoam -parallel #Runs interFoam in parallel
reconstructPar #Collects the decomposed results
Post-process in another terminal window:
paraview_ml
touch case.foam
vglrun paraview #Read in case.foam
Create aliases (in addToBashrc):
alias OFv2112Cephyr="ml foss/2021b; export MPI_ROOT=\$EBROOTOPENMPI; export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/openfoam/hebbe/OpenFOAM; . \$FOAM_INST_DIR/OpenFOAM-v2112/etc/bashrc WM_MPLIB=SYSTEMOPENMPI; export FOAM_FILEHANDLER=collated"
Make available:
chmod -R go+Xr OpenFOAM-v2112
chmod -R go+Xr ThirdParty-v2112
Clean ThirdParty (see notes at top of page):
Only the "platforms" directory needs to be kept after a successful installation. Just remove the rest, by (where the "dummy" shows how to keep more than one directory if necessary):
cd $WM_THIRD_PARTY_DIR
###############################################################################################
# MAKE SURE YOU ARE IN THE ThirdParty DIRECTORY, SINCE THIS IS A DANGEROUS RECURSIVE COMMAND!!!
# IT IS COMMENTED HERE, SO YOU DON'T MAKE ANY COPY-PASTE MISTAKE
###############################################################################################
#shopt -s extglob
#rm -rv !("platforms"|"dummy")
Notes about system installations (for the original EasyBuild installation, after removing the -ftree-vectorize flag to remove the problem):
EasyBuild compilation at Vera gives incorrect results. It uses and gives the following logs:
/apps/easybuild-easyconfigs/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-v2112-foss-2021b.eb
/apps/Vera/software/OpenFOAM/v2112-foss-2021b/easybuild/easybuild-OpenFOAM-v2112-20220216.125146.log
/apps/Vera/software/OpenFOAM/v2112-foss-2021b/easybuild/easybuild-OpenFOAM-v2112-20220216.134121.log
The only meaningful differences between a regular installation using foss-2021b (which gives good results) are:
diff -r OpenFOAM-v2112/wmake/rules/General/Gcc/c /apps/Vera/software/OpenFOAM/v2112-foss-2021b/OpenFOAM-v2112/wmake/rules/General/Gcc/c
3c3
< cc = gcc$(COMPILER_VERSION)
---
> cc = OMPI_CC="gcc" mpicc
7c7
< cOPT = -O2
---
> cOPT = -O2 -ftree-vectorize -march=native -fno-math-errno -std=c++11 -fuse-ld=bfd
diff -r OpenFOAM-v2112/wmake/rules/General/Gcc/c++ /apps/Vera/software/OpenFOAM/v2112-foss-2021b/OpenFOAM-v2112/wmake/rules/General/Gcc/c++
9c9
< CC = g++$(COMPILER_VERSION) -std=c++11
---
> CC = OMPI_CXX="g++" mpicxx
13c13
< c++OPT = -O2
---
> c++OPT = -O2 -ftree-vectorize -march=native -fno-math-errno -std=c++11 -fuse-ld=bfd
diff -r OpenFOAM-v2112/wmake/rules/linux64Gcc/cOpt /apps/Vera/software/OpenFOAM/v2112-foss-2021b/OpenFOAM-v2112/wmake/rules/linux64Gcc/cOpt
2c2
< cOPT = -O3
---
> cOPT = -O2 -ftree-vectorize -march=native -fno-math-errno -std=c++11 -fuse-ld=bfd
diff -r OpenFOAM-v2112/wmake/rules/linux64Gcc/c++Opt /apps/Vera/software/OpenFOAM/v2112-foss-2021b/OpenFOAM-v2112/wmake/rules/linux64Gcc/c++Opt
2c2
< c++OPT = -O3
---
> c++OPT = -O2 -ftree-vectorize -march=native -fno-math-errno -std=c++11 -fuse-ld=bfd
Flags for gcc (irrespectively of OpenFOAM) seem to be changed in (same for 4.5.1 and 4.5.3):
/apps/Common/software/EasyBuild/4.5.3/lib/python3.6/site-packages/easybuild/toolchains/compiler/gcc.py
EasyBuild compilation at Tetralith gives correct results. It uses intel-2018b.
Recipe to compile with EasyBuild flags:
In a directory that is NOT where you have already installed OpenFOAM-v2112:
wget https://sourceforge.net/projects/openfoam/files/v2112/OpenFOAM-v2112.tgz
tar xzf OpenFOAM-v2112.tgz
mv OpenFOAM-v2112 OpenFOAM-v2112-eb
Move that directory to the directory where you usually install OpenFOAM and go into it
sed -i s/v2112/v2112-eb/g etc/bashrc
sed -i s/'gcc$(COMPILER_VERSION)'/'OMPI_CC="gcc" mpicc'/g wmake/rules/General/Gcc/c
sed -i s/'-O2'/'-O2 -ftree-vectorize -march=native -fno-math-errno -std=c++11 -fuse-ld=bfd'/g wmake/rules/General/Gcc/c
sed -i s/'gcc$(COMPILER_VERSION) -std=c++11'/'OMPI_CC="gcc" mpicc'/g wmake/rules/General/Gcc/c++
sed -i s/'-O2'/'-O2 -ftree-vectorize -march=native -fno-math-errno -std=c++11 -fuse-ld=bfd'/g wmake/rules/General/Gcc/c++
sed -i s/'-O3'/'-O2 -ftree-vectorize -march=native -fno-math-errno -std=c++11 -fuse-ld=bfd'/g wmake/rules/linux64Gcc/cOpt
sed -i s/'-O3'/'-O2 -ftree-vectorize -march=native -fno-math-errno -std=c++11 -fuse-ld=bfd'/g wmake/rules/linux64Gcc/c++Opt
Avoid recompiling ThirdParty by commenting out that part in the Allwmake script. Also go to the directory where ThirdParty-v2112 is located and do:
ln -s ThirdParty-v2112 ThirdParty-v2112-eb
Then just compile and run the tests.
cd /cephyr/NOBACKUP/groups/openfoam/hebbe/foam
git clone -b nextRelease https://git.code.sf.net/p/foam-extend/foam-extend-4.1 foam-extend-4.1_NR_updated
cd foam-extend-4.1_NR_updated
sed -i.orig s/"5\.0$"/"4\.1_NR_updated"/g etc/bashrc
echo "export PARAVIEW_SYSTEM=1" >> etc/prefs.sh #Avoid having to type "Y" at compilation
Fix problem with gold linker (see comments for previous installations):
sed -i s/fPIC/"fPIC -fuse-ld=bfd"/g wmake/rules/linux64Gcc/c++
Continue, loading modules:
module load foss/2019a
module load CMake/3.13.3 #Uses same GCCcore as foss/2019a: GCCcore/8.2.0 (checked with 'module spider')
Use system OpenMPI:
echo "export WM_MPLIB=SYSTEMOPENMPI" >> etc/prefs.sh
echo "export OPENMPI_DIR=$EBROOTOPENMPI" >> etc/prefs.sh
echo "export OPENMPI_BIN_DIR=$EBROOTOPENMPI/bin" >> etc/prefs.sh
Revert some default setting, to make METIS etc work (don't use system installation):
sed -i s/"export METIS_SYSTEM=1"/"#export METIS_SYSTEM=1"/g etc/bashrc
sed -i s/"export WM_THIRD_PARTY_USE_PARMGRIDGEN_10=1"/"#export WM_THIRD_PARTY_USE_PARMGRIDGEN_10=1"/g etc/bashrc
sed -i s/"export SCOTCH_SYSTEM=1"/"#export SCOTCH_SYSTEM=1"/g etc/bashrc
sed -i s/"export PARMETIS_SYSTEM=1"/"#export PARMETIS_SYSTEM=1"/g etc/bashrc
sed -i s/"export WM_THIRD_PARTY_USE_PYFOAM_069=1"/"#export WM_THIRD_PARTY_USE_PYFOAM_069=1"/g etc/bashrc
sed -i s/"export WM_THIRD_PARTY_USE_HWLOC_201=1"/"#export WM_THIRD_PARTY_USE_HWLOC_201=1"/g etc/bashrc
echo "export WM_THIRD_PARTY_USE_METIS_510=1" >> etc/prefs.sh
echo "export WM_THIRD_PARTY_USE_PARMGRIDGEN_10=1" >> etc/prefs.sh
echo "export WM_THIRD_PARTY_USE_PARMETIS_403=1" >> etc/prefs.sh
echo "export WM_THIRD_PARTY_USE_SCOTCH_604=1" >> etc/prefs.sh
Compile:
export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/openfoam/hebbe/foam; . $FOAM_INST_DIR/foam-extend-4.1_NR_updated/etc/bashrc
foam
./Allwmake.firstInstall >& log&
#Do again after completion, since parallel compilation may make things fail:
./Allwmake.firstInstall >& log&
Problem with hwloc (seems to have disappeared after above corrections):
tar: packages/hwloc-2.0.1: file changed as we read it
error: Bad exit status from /cephyr/NOBACKUP/groups/openfoam/hebbe/foam/foam-extend-4.1_NR_updated/ThirdParty/rpmBuild/tmp/rpm-tmp.vYWEEP (%install)
RPM build errors:
Bad exit status from /cephyr/NOBACKUP/groups/openfoam/hebbe/foam/foam-extend-4.1_NR_updated/ThirdParty/rpmBuild/tmp/rpm-tmp.vYWEEP (%install)
Installing package: hwloc-2.0.1
Uninstalling hwloc-2.0.1 using RPM: hwloc-2.0.1-linux64GccDPInt32Opt.x86_64
Installing hwloc-2.0.1 using RPM file: /cephyr/NOBACKUP/groups/openfoam/hebbe/foam/foam-extend-4.1_NR_updated/ThirdParty/rpmBuild/RPMS/x86_64/hwloc-2.0.1-linux64GccDPInt32Opt.x86_64.rpm
error: open of /cephyr/NOBACKUP/groups/openfoam/hebbe/foam/foam-extend-4.1_NR_updated/ThirdParty/rpmBuild/RPMS/x86_64/hwloc-2.0.1-linux64GccDPInt32Opt.x86_64.rpm failed: No such file or directory
Problem with metisDecom, parMetisDecomp and scotchDecomp, if environment variables are not set to those installations (fixed above by installing them):
metisDecomp.C:34:13: fatal error: metis.h: No such file or directory
parMetisDecomp.C:39:13: fatal error: parmetis.h: No such file or directory
scotchDecomp.C:115:10: fatal error: scotch.h: No such file or directory
decompositionMethod.H not found (since environment variable in Make/options is not set)
Continue:
chgrp -R pg_openfoam /cephyr/NOBACKUP/groups/openfoam/hebbe/foam/foam-extend-4.1_NR_updated
chmod -R go+Xr /cephyr/NOBACKUP/groups/openfoam/hebbe/foam/foam-extend-4.1_NR_updated
Submit script:
#!/usr/bin/env bash
#SBATCH -A C3SE408-18-1 -p vera
#SBATCH -n 4
#SBATCH -t 01:00:00
#-----------------------------------------------------------
ml foss/2019a
export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/openfoam/hebbe/foam; . $FOAM_INST_DIR/foam-extend-4.1_NR_updated/etc/bashrc
#------------ Copy and run the damBreak tutorial -----------
#mkdir -p $FOAM_RUN/testDamBreak
#cd $FOAM_RUN/testDamBreak
cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak .
cd damBreak
blockMesh #Generates the grid
setFields #Sets the initial fields
decomposePar #Decomposes for parallel simulation
mpirun interFoam -parallel #Runs interFoam in parallel
reconstructPar #Collects the decomposed results
#rm -rf processor* #Deletes the decomposed files
For Paraview, in a separate terminal without OpenFOAM (only in Thinlinc):
paraview_ml
vglrun paraview
HERE 2022-01-26
Clean up (see top of page):
cd ThirdParty
./AllClean
rm -r rpmBuild/BUILD/*
rm -r rpmBuild/BUILDROOT/*
rm -r rpmBuild/TGZS/*
rm -r rpmBuild/RPMS/*
rm rpmBuild/SOURCES/*tar*
foam
wcleanAllButLibBinLnInclude
Compile Debug version without recompiling ThirdParty
New terminal window
ml foss/2019a; export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/openfoam/hebbe/foam; export WM_COMPILE_OPTION=Debug; . $FOAM_INST_DIR/foam-extend-4.1_NR_updated/etc/bashrc
cd $WM_THIRD_PARTY_DIR/packages/libccmio-2.6.1/platforms
ln -s linux64GccDPInt32Opt linux64GccDPInt32Debug
cd $WM_THIRD_PARTY_DIR/packages/mesquite-2.3.0/platforms
ln -s linux64GccDPInt32Opt linux64GccDPInt32Debug
cd $WM_THIRD_PARTY_DIR/packages/metis-5.1.0/platforms
ln -s linux64GccDPInt32Opt linux64GccDPInt32Debug
cd $WM_THIRD_PARTY_DIR/packages/parmetis-4.0.3/platforms
ln -s linux64GccDPInt32Opt linux64GccDPInt32Debug
cd $WM_THIRD_PARTY_DIR/packages/ParMGridGen-1.0/platforms
ln -s linux64GccDPInt32Opt linux64GccDPInt32Debug
cd $WM_THIRD_PARTY_DIR/packages/scotch-6.0.4/platforms
ln -s linux64GccDPInt32Opt linux64GccDPInt32Debug
foam
./Allwmake >& log
cd /cephyr/NOBACKUP/groups/openfoam/hebbe/OpenFOAM
wget https://sourceforge.net/projects/openfoam/files/v1812/OpenFOAM-v1812.tgz
wget https://sourceforge.net/projects/openfoam/files/v1812/ThirdParty-v1812.tgz
tar xzf OpenFOAM-v1812.tgz
tar xzf ThirdParty-v1812.tgz
rm OpenFOAM-v1812.tgz ThirdParty-v1812.tgz
module load foss/2018b
export MPI_ROOT=$EBROOTOPENMPI
export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/openfoam/hebbe/OpenFOAM
. $FOAM_INST_DIR/OpenFOAM-v1812/etc/bashrc WM_MPLIB=SYSTEMOPENMPI
foam
sed -i s/fPIC/"fPIC -fuse-ld=bfd"/g wmake/rules/linux64Gcc/c++
./Allwmake >& log.make&
Test script:
#!/usr/bin/env bash
#SBATCH -A C3SE2021-1-10 -p vera
#SBATCH -n 4
#SBATCH -t 01:00:00
#-----------------------------------------------------------
module load foss/2018b
export MPI_ROOT=$EBROOTOPENMPI
export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/openfoam/hebbe/OpenFOAM
. $FOAM_INST_DIR/OpenFOAM-v1812/etc/bashrc WM_MPLIB=SYSTEMOPENMPI
export FOAM_FILEHANDLER=collated #See https://www.openfoam.com/releases/openfoam-v1712/parallel.php
#------------ Copy and run the damBreak tutorial -----------
#mkdir -p $FOAM_RUN/testDamBreak
#cd $FOAM_RUN/testDamBreak
cp -r $FOAM_TUTORIALS/multiphase/interFoam/RAS/damBreak/damBreak .
cd damBreak
sed -i s/"writeFormat\s*ascii"/"writeFormat binary"/g system/controlDict
blockMesh #Generates the grid
setFields #Sets the initial fields
decomposePar #Decomposes for parallel simulation
mpirun interFoam -parallel #Runs interFoam in parallel
reconstructPar #Collects the decomposed results
#rm -rf processor* #Deletes the decomposed files
Post-process in another terminal window:
paraview_ml
touch case.foam
#vglrun seems not to be needed 211220
vglrun paraview #Read in case.foam
Create alias (in addToBashrc):
alias OFv1812Cephyr="module load foss/2018b; export MPI_ROOT=\$EBROOTOPENMPI; export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/openfoam/hebbe/OpenFOAM; . \$FOAM_INST_DIR/OpenFOAM-v1812/etc/bashrc WM_MPLIB=SYSTEMOPENMPI; export FOAM_FILEHANDLER=collated"
Clean ThirdParty (see notes at top of page):
Only the "platforms" directory needs to be kept after a successful installation. Just remove the rest, by (where the "dummy" shows how to keep more than one directory if necessary):
cd $WM_THIRD_PARTY_DIR
###############################################################################################
# MAKE SURE YOU ARE IN THE ThirdParty DIRECTORY, SINCE THIS IS A DANGEROUS RECURSIVE COMMAND!!!
# IT IS COMMENTED HERE, SO YOU DON'T MAKE ANY COPY-PASTE MISTAKE
###############################################################################################
#shopt -s extglob
#rm -rv !("platforms"|"dummy")
cd /cephyr/NOBACKUP/groups/openfoam/hebbe/foam
git clone -b TurboWG_experimental https://git.code.sf.net/p/turbowg/foam-extend-5.0 foam-extend-5.0_TurboWG_experimental
cd foam-extend-5.0_TurboWG_experimental
sed -i.orig s/"5\.0$"/"5\.0_TurboWG_experimental"/g etc/bashrc
echo "export PARAVIEW_SYSTEM=1" >> etc/prefs.sh #Avoid having to type "Y" at compilation
echo "export WM_THIRD_PARTY_BUILD_TGZ_ARCHIVES=0" >> etc/prefs.sh #Avoid creating tar archives, which has made compilation fail
Fix problem with:
" /apps/Hebbe7/software/Compiler/GCCcore/8.2.0/binutils/2.31.1/bin/ld.gold: error: --add-needed is not supported but is required for libopen-rte.so.40 in /apps/Hebbe7/software/Compiler/GCC/8.2.0-2.31.1/OpenMPI/3.1.3/lib/libmpi.so "
Comment from Mikael Öhman, C3SE:
" Jag vet att det varit problem med gold-länkaren för vissa version av openfoam. Du får nog använda dig av gamla vanliga BFD-länkaren istället. Du kan specifiera ett länkningskommando på något sätt, jag vet inte hur WMake gör, men, det är "ld.bfd" dom du vill använda under länkningssteget, men, om du har något ställa där du kan lägga till en länkningsflagga; LDFLAGS=" -fuse-ld=bfd " "
sed -i s/fPIC/"fPIC -fuse-ld=bfd"/g wmake/rules/linux64Gcc/c++
Continue:
module load foss/2019a
module load CMake/3.13.3 #Uses same GCCcore as foss/2019a: GCCcore/8.2.0 (checked with 'module spider')
echo "export WM_MPLIB=SYSTEMOPENMPI" >> etc/prefs.sh
echo "export OPENMPI_DIR=$EBROOTOPENMPI" >> etc/prefs.sh
echo "export OPENMPI_BIN_DIR=$EBROOTOPENMPI/bin" >> etc/prefs.sh
export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/openfoam/hebbe/foam; . $FOAM_INST_DIR/foam-extend-5.0_TurboWG_experimental/etc/bashrc
foam
./Allwmake.firstInstall >& log
#Type CTRL-z, bg, and tailf log. Wait until finished
#Do again, since parallel compilation may make things fail:
./Allwmake.firstInstall >& log
#Type CTRL-z, bg, and tailf log. Wait until finished
chmod -R go+Xr /cephyr/NOBACKUP/groups/openfoam/hebbe/foam/foam-extend-5.0_TurboWG_experimental
alias fe50_TurboWG="ml foss/2019a; export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/openfoam/hebbe/foam; . \$FOAM_INST_DIR/foam-extend-5.0_TurboWG_experimental/etc/bashrc"
Submit script:
#!/usr/bin/env bash
#SBATCH -A C3SE2021-1-10 -p hebbe
#SBATCH -n 4
#SBATCH -t 01:00:00
#-----------------------------------------------------------
ml foss/2019a
export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/openfoam/hebbe/foam; . $FOAM_INST_DIR/foam-extend-5.0_TurboWG_experimental/etc/bashrc
#------------ Copy and run the damBreak tutorial -----------
#mkdir -p $FOAM_RUN/testDamBreak
#cd $FOAM_RUN/testDamBreak
cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak .
cd damBreak
blockMesh #Generates the grid
setFields #Sets the initial fields
decomposePar #Decomposes for parallel simulation
mpirun interFoam -parallel #Runs interFoam in parallel
reconstructPar #Collects the decomposed results
#rm -rf processor* #Deletes the decomposed files
Experimental: For Paraview, in a separate terminal without OpenFOAM (only in Thinlinc):
alias paraview_ml='ml GCC/9.3.0 OpenMPI/4.0.3 ParaView/5.8.0-Python-3.8.2-mpi'
paraview_ml
vglrun paraview
Clean up (see top of page):
cd ThirdParty
./AllClean
rm -r rpmBuild/BUILD/*
rm -r rpmBuild/BUILDROOT/*
rm -r rpmBuild/TGZS/*
rm -r rpmBuild/RPMS/*
rm rpmBuild/SOURCES/*tar*
foam
wcleanAllButLibBinLnInclude
cd /proj/hani/OF_inst/Tetralith/foam
git clone -b nextRelease https://git.code.sf.net/p/foam-extend/foam-extend-4.1 foam-extend-4.1_NR
cd foam-extend-4.1_NR
sed -i.orig s/"5\.0$"/"4\.1_NR"/g etc/bashrc
module load buildenv-gcc/2018a-eb
#Avoid having to type Y at compilation:
echo "export PARAVIEW_SYSTEM=1" > etc/prefs.sh
echo "export WM_MPLIB=SYSTEMOPENMPI" >> etc/prefs.sh
#The following inspired by etc/prefs.csh:
echo "export OPENMPI_DIR=/software/sse/easybuild/prefix/software/OpenMPI/2.1.2-GCC-6.4.0-2.28" >> etc/prefs.sh
echo "export OPENMPI_BIN_DIR=\$OPENMPI_DIR/bin" >> etc/prefs.sh
#The following line gave two paths, so I had to give the correct one explicitly (in the next line)
#echo 'export OPENMPI_LIB_DIR="`$OPENMPI_BIN_DIR/mpicc --showme:libdirs`"' >> etc/prefs.sh
echo "export OPENMPI_LIB_DIR=/software/sse/easybuild/prefix/software/OpenMPI/2.1.2-GCC-6.4.0-2.28/lib" >> etc/prefs.sh
echo 'export OPENMPI_INCLUDE_DIR="`$OPENMPI_BIN_DIR/mpicc --showme:incdirs`"' >> etc/prefs.sh
echo 'export OPENMPI_COMPILE_FLAGS="`$OPENMPI_BIN_DIR/mpicc --showme:compile`"' >> etc/prefs.sh
echo 'export OPENMPI_LINK_FLAGS="`$OPENMPI_BIN_DIR/mpicc --showme:link`"' >> etc/prefs.sh
export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/foam; . $FOAM_INST_DIR/foam-extend-4.1_NR/etc/bashrc
foam
#Fix some compilation problems:
rm -r src/mesh/blockMesh/blockEdges #Not used, and had at least one file with the same name as another class (lnInclude problem). https://sourceforge.net/p/foam-extend/tickets/56/
#In src/conjugateHeatTransfer/fvPatchFields/chtRcThermalDiffusivityResistance/chtRcThermalDiffusivityResistanceFvPatchScalarField.C, comment lines 411 and 412. https://sourceforge.net/p/foam-extend/tickets/57/
#Remaining error in applications/utilities/postProcessing/POD/vectorSnapshots not fixed. https://sourceforge.net/p/foam-extend/tickets/58/
./Allwmake.firstInstall &>log_Allwmake& #("firstInstall" includes ThirdParty, which should not be done if it has been cleaned up)
./Allwmake.firstInstall &>log_Allwmake& #("firstInstall" includes ThirdParty, which should not be done if it has been cleaned up)
echo "alias fe41NR='export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/foam; . \$FOAM_INST_DIR/foam-extend-4.1_NR/etc/bashrc'" >> ../../../addToBashrc
echo "alias fe41NR_compile='module load buildenv-gcc/2018a-eb; export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/foam; . \$FOAM_INST_DIR/foam-extend-4.1_NR/etc/bashrc'" >> ../../../addToBashrc
#If not done before:
#alias paraview_ml='module load ParaView/5.4.1-nsc1-gcc-2018a-eb'
Test script:
#!/bin/bash
#
#SBATCH -n 4
#SBATCH -t 00:20:00
#SBATCH -J damBreak
#SBATCH -A snic2020-3-23
export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/foam; . $FOAM_INST_DIR/foam-extend-4.1_NR/etc/bashrc
cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak .
blockMesh -case damBreak
setFields -case damBreak
decomposePar -case damBreak
mpirun interFoam -parallel -case damBreak &> log.damBreak
Clean up (see top of page):
cd ThirdParty
./AllClean
rm -r rpmBuild/BUILD/*
rm -r rpmBuild/BUILDROOT/*
rm -r rpmBuild/TGZS/*
rm -r rpmBuild/RPMS/*
rm rpmBuild/SOURCES/*tar*
foam
wcleanAllButLibBinLnInclude
Install solids4Foam:
cd extend-bazaar
git clone -b development https://dr_hakan_nilsson@bitbucket.org/philip_cardiff/solids4foam-release.git
find solids4foam-release -name files | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done
find solids4foam-release -name files.foamextend | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done
find solids4foam-release -name files | while read item; do sed -i -e 's/FOAM_USER_APPBIN/FOAM_SITE_APPBIN/g' $item; done
find solids4foam-release -name files.foamextend | while read item; do sed -i -e 's/FOAM_USER_APPBIN/FOAM_SITE_APPBIN/g' $item; done
find solids4foam-release -name options | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done
sed -i s/FOAM_USER_APPBIN/FOAM_SITE_APPBIN/g solids4foam-release/applications/scripts/Allwmake
cd solids4foam-release
#The following commented should not be done. See solids4foam-release/filesToReplaceInOF/README.txt
##Next line to check which files to copy in the following lines:
#(cd filesToReplaceInOF && ./Allcheck)
#cp filesToReplaceInOF/solution.C /proj/hani/OF_inst/Tetralith/foam/foam-extend-4.1_NR/src/foam/matrices/solution/
#cp filesToReplaceInOF/WedgePointPatchField.C /proj/hani/OF_inst/Tetralith/foam/foam-extend-4.1_NR/src/foam/fields/PointPatchFieldTemplates/constraint/wedge/
#cp filesToReplaceInOF/tensor.C /proj/hani/OF_inst/Tetralith/foam/foam-extend-4.1_NR/src/foam/primitives/Tensor/tensor/
#cp filesToReplaceInOF/findRefCell.C /proj/hani/OF_inst/Tetralith/foam/foam-extend-4.1_NR/src/finiteVolume/cfdTools/general/findRefCell/
##Next line to check that the files were copied correctly:
#(cd filesToReplaceInOF && ./Allcheck)
#INSTEAD, comment "# Replace files in the main foam libraries" in Allwmake
./Allwmake >& log_Allwmake
Test in new terminal window:
fe41NR
mkdir solids4FoamTest
cd solids4FoamTest
scp -r $WM_PROJECT_DIR/extend-bazaar/solids4foam-release/tutorials/fluidSolidInteraction/beamInCrossFlow/elasticBeam .
cd elasticBeam
./Allrun >& log&
Post-process in another terminal window:
paraview_ml
touch case.foam
vglrun paraview
Read in case.foam
Make available:
cd /proj/hani/OF_inst/Tetralith/foam
chmod -R go+Xr foam-extend-4.1_NR
chmod -R go+Xr site/4.1_NR
cd /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe
mkdir OpenFOAM
cd OpenFOAM/
wget https://sourceforge.net/projects/openfoam/files/v1912/OpenFOAM-v1912.tgz
wget https://sourceforge.net/projects/openfoam/files/v1912/ThirdParty-v1912.tgz
tar xzf OpenFOAM-v1912.tgz
tar xzf ThirdParty-v1912.tgz
rm OpenFOAM-v1912.tgz ThirdParty-v1912.tgz
module load foss/2019a
export MPI_ROOT=$EBROOTOPENMPI
export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/OpenFOAM
. $FOAM_INST_DIR/OpenFOAM-v1912/etc/bashrc WM_MPLIB=SYSTEMOPENMPI
foam
sed -i s/fPIC/"fPIC -fuse-ld=bfd"/g wmake/rules/linux64Gcc/c++
cd modules
git clone -b development https://dr_hakan_nilsson@bitbucket.org/philip_cardiff/solids4foam-release.git
find solids4foam-release -name files | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done
find solids4foam-release -name files.openfoam | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done
find solids4foam-release -name files | while read item; do sed -i -e 's/FOAM_USER_APPBIN/FOAM_SITE_APPBIN/g' $item; done
find solids4foam-release -name files.openfoam | while read item; do sed -i -e 's/FOAM_USER_APPBIN/FOAM_SITE_APPBIN/g' $item; done
find solids4foam-release -name options | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done
sed -i s/FOAM_USER_APPBIN/FOAM_SITE_APPBIN/g solids4foam-release/applications/scripts/Allwmake
cp modules/solids4foam-release/filesToReplaceInOF/dataTemplates.C /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/OpenFOAM/OpenFOAM-v1912/src/OpenFOAM/meshes/data/
cp modules/solids4foam-release/filesToReplaceInOF/AMIInterpolation.C /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/OpenFOAM/OpenFOAM-v1912/src/meshTools/AMIInterpolation/AMIInterpolation/
cp modules/solids4foam-release/filesToReplaceInOF/AMIInterpolation.H /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/OpenFOAM/OpenFOAM-v1912/src/meshTools/AMIInterpolation/AMIInterpolation/
foam
./Allwmake >& log.make&
Test script:
#!/usr/bin/env bash
#SBATCH -A SNIC2020-5-103 -p hebbe
#SBATCH -n 4
#SBATCH -t 01:00:00
#-----------------------------------------------------------
module load foss/2019a
export MPI_ROOT=$EBROOTOPENMPI
export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/OpenFOAM
. $FOAM_INST_DIR/OpenFOAM-v1912/etc/bashrc WM_MPLIB=SYSTEMOPENMPI
export FOAM_FILEHANDLER=collated #See https://www.openfoam.com/releases/openfoam-v1712/parallel.php
#------------ Copy and run the damBreak tutorial -----------
#mkdir -p $FOAM_RUN/testDamBreak
#cd $FOAM_RUN/testDamBreak
cp -r $FOAM_TUTORIALS/multiphase/interFoam/RAS/damBreak/damBreak .
cd damBreak
sed -i s/"writeFormat\s*ascii"/"writeFormat binary"/g system/controlDict
blockMesh #Generates the grid
setFields #Sets the initial fields
decomposePar #Decomposes for parallel simulation
mpirun interFoam -parallel #Runs interFoam in parallel
reconstructPar #Collects the decomposed results
#rm -rf processor* #Deletes the decomposed files
Test solids4Foam (does not work "Duplicate entry skewCorrected in runtime selection table snGradScheme"):
mkdir solids4FoamTest
cd solids4FoamTest
cp -r /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/OpenFOAM/OpenFOAM-v1912/modules/solids4foam-release/tutorials/fluidSolidInteraction/beamInCrossFlow/elasticBeam .
cd elasticBeam
./Allrun >& log&
Post-process in another terminal window:
paraview_ml
touch case.foam
vglrun paraview #Read in case.foam
Create alias (in addToBashrc):
alias OFv1912Cephyr="module load foss/2019a; export MPI_ROOT=\$EBROOTOPENMPI; export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/OpenFOAM; . \$FOAM_INST_DIR/OpenFOAM-v1912/etc/bashrc WM_MPLIB=SYSTEMOPENMPI; export FOAM_FILEHANDLER=collated"
Clean ThirdParty (see notes at top of page):
Only the "platforms" directory needs to be kept after a successful installation. Just remove the rest, by (where the "dummy" shows how to keep more than one directory if necessary):
cd $WM_THIRD_PARTY_DIR
###############################################################################################
# MAKE SURE YOU ARE IN THE ThirdParty DIRECTORY, SINCE THIS IS A DANGEROUS RECURSIVE COMMAND!!!
# IT IS COMMENTED HERE, SO YOU DON'T MAKE ANY COPY-PASTE MISTAKE
###############################################################################################
#shopt -s extglob
#rm -rv !("platforms"|"dummy")
NOTE: Avoid having to type "Y" at compilation:
echo "export PARAVIEW_SYSTEM=1" >> etc/prefs.sh
Seems to be unstable, reporting floating point exception after writing files...
[hebbe23-5:00522] *** Process received signal *** [hebbe23-5:00522] Signal: Floating point exception (8)
cd /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/foam
git clone -b nextRelease https://git.code.sf.net/p/foam-extend/foam-extend-4.1 foam-extend-4.1_NR
cd foam-extend-4.1_NR
sed -i.orig s/"5\.0$"/"4\.1_NR"/g etc/bashrc
Fix problem with:
" /apps/Hebbe7/software/Compiler/GCCcore/8.2.0/binutils/2.31.1/bin/ld.gold: error: --add-needed is not supported but is required for libopen-rte.so.40 in /apps/Hebbe7/software/Compiler/GCC/8.2.0-2.31.1/OpenMPI/3.1.3/lib/libmpi.so "
Comment from Mikael Öhman, C3SE:
" Jag vet att det varit problem med gold-länkaren för vissa version av openfoam. Du får nog använda dig av gamla vanliga BFD-länkaren istället. Du kan specifiera ett länkningskommando på något sätt, jag vet inte hur WMake gör, men, det är "ld.bfd" dom du vill använda under länkningssteget, men, om du har något ställa där du kan lägga till en länkningsflagga; LDFLAGS=" -fuse-ld=bfd " "
sed -i s/fPIC/"fPIC -fuse-ld=bfd"/g wmake/rules/linux64Gcc/c++
Continue:
module load foss/2019a
module load CMake/3.13.3 #Uses same GCCcore as foss/2019a: GCCcore/8.2.0 (checked with 'module spider')
echo "export WM_MPLIB=SYSTEMOPENMPI" >> etc/prefs.sh
echo "export OPENMPI_DIR=$EBROOTOPENMPI" >> etc/prefs.sh
echo "export OPENMPI_BIN_DIR=$EBROOTOPENMPI/bin" >> etc/prefs.sh
export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/foam; . $FOAM_INST_DIR/foam-extend-4.1_NR/etc/bashrc
foam
./Allwmake.firstInstall >& log
Y
#Type CTRL-z, bg, and tailf log. Wait until finished
#Do again, since parallel compilation may make things fail:
./Allwmake.firstInstall >& log
Y
#Type CTRL-z, bg, and tailf log. Wait until finished
chgrp -R c3-fluiddyn /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/foam/foam-extend-4.1_NR
chmod -R go+Xr /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/foam/foam-extend-4.1_NR
Submit script:
#!/usr/bin/env bash
#SBATCH -A SNIC2020-5-103 -p hebbe
#SBATCH -n 4
#SBATCH -t 01:00:00
#-----------------------------------------------------------
ml foss/2019a
export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/foam; . $FOAM_INST_DIR/foam-extend-4.1_NR/etc/bashrc
#------------ Copy and run the damBreak tutorial -----------
#mkdir -p $FOAM_RUN/testDamBreak
#cd $FOAM_RUN/testDamBreak
cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak .
cd damBreak
blockMesh #Generates the grid
setFields #Sets the initial fields
decomposePar #Decomposes for parallel simulation
mpirun interFoam -parallel #Runs interFoam in parallel
reconstructPar #Collects the decomposed results
#rm -rf processor* #Deletes the decomposed files
Experimental: For Paraview, in a separate terminal without OpenFOAM (only in Thinlinc):
ml GCC/8.3.0 OpenMPI/3.1.4 ParaView/5.6.2-Python-3.7.4-mpi
vglrun paraview
Clean up (see top of page):
cd ThirdParty
./AllClean
rm -r rpmBuild/BUILD/*
rm -r rpmBuild/BUILDROOT/*
rm -r rpmBuild/TGZS/*
rm -r rpmBuild/RPMS/*
rm rpmBuild/SOURCES/*tar*
foam
wcleanAllButLibBinLnInclude
NOTE: Avoid having to type "Y" at compilation:
echo "export PARAVIEW_SYSTEM=1" >> etc/prefs.sh
cd $SNIC_NOBACKUP/hebbe/foam
git clone git://git.code.sf.net/p/openfoam-extend/foam-extend-3.1 foam-extend-3.1
cd foam-extend-3.1
sed -i s/"http\:\/\/portal.nersc.gov\/svn\/visit\/tags\/2.4.2\/third_party\/libccmio-2.6.1.tar.gz"/"http\:\/\/downloads.sourceforge.net\/project\/foam-extend\/ThirdParty\/libccmio-2.6.1.tar.gz"/g ThirdParty/AllMake.stage3
sed -i s/fPIC/"fPIC -fuse-ld=bfd"/g wmake/rules/linux64Gcc/c++
Avoided compilation error(s):
fields/fvPatchFields/derived/advective/advectiveFvPatchField.H:224:24: error: cannot bind non-const lvalue reference of type ‘bool&’ to an rvalue of type ‘bool’
return inletOutlet_;
by deactivating things we don't need (that cause problems):
sed -i.bak1 s/"\$(derivedFvPatchFields)\/advective\/advectiveFvPatchFields.C"/""/g src/finiteVolume/Make/files
sed -i.bak2 s/"\$(derivedFvPatchFields)\/waveTransmissive\/waveTransmissiveFvPatchFields.C"/""/g src/finiteVolume/Make/files
sed -i.bak3 s/"\$(derivedFvPatchFields)\/waveTransmissiveInlet\/waveTransmissiveInletFvPatchFields.C"/""/g src/finiteVolume/Make/files
sed -i.bak4 s/"derivedFvPatchFields\/engineTimeVaryingUniformFixedValue\/engineTimeVaryingUniformFixedValueFvPatchFields.C"/""/g src/engine/Make/files
Continued:
echo "export WM_MPLIB=SYSTEMOPENMPI" >> etc/prefs.sh
echo "export OPENMPI_DIR=$EBROOTOPENMPI" >> etc/prefs.sh
echo "export OPENMPI_BIN_DIR=$EBROOTOPENMPI/bin" >> etc/prefs.sh
module load foss/2019a
module load CMake/3.13.3 #Uses same GCCcore as foss/2019a: GCCcore/8.2.0 (checked with 'module spider')
export FOAM_INST_DIR=$SNIC_NOBACKUP/hebbe/foam; . $FOAM_INST_DIR/foam-extend-3.1/etc/bashrc
foam
./Allwmake.firstInstall >& log
Y
#Type CTRL-z, bg, and tailf log. Wait until finished
#Do again, since parallel compilation may make things fail:
./Allwmake.firstInstall >& log
Y
#Type CTRL-z, bg, and tailf log. Wait until finished
FSI:
cd extend-bazaar/
wget http://openfoamwiki.net/images/5/52/Fsi_31.tar.gz
tar zxvf Fsi_31.tar.gz
rm Fsi_31.tar.gz
cd FluidStructureInteraction/
find src -name options | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done
find src -name files | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done
find run -name options | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done
#Change to -I$(WM_PROJECT_DIR)/extend-bazaar/FluidStructureInteraction/src/fluidStructureInteraction/lnInclude
#in:
#run/fsiFoam/HronTurekFsi3/hronTurekReport/Make/options
#run/fsiFoam/beamInCrossFlow/setInletVelocity/Make/options
#run/stressFoam/plateHole/setPlateHoleBC/Make/options
#
pushd src/fluidStructureInteraction/stressModels/constitutiveModel
wget https://raw.githubusercontent.com/wyldckat/FluidStructureInteraction/fe31/src/fluidStructureInteraction/stressModels/constitutiveModel/constitutiveModel.C -O constitutiveModel.C
wget https://raw.githubusercontent.com/wyldckat/FluidStructureInteraction/fe31/src/fluidStructureInteraction/stressModels/constitutiveModel/constitutiveModel.H -O constitutiveModel.H
popd
cd src/
./Allwmake >& log&
./Allwmake >& log&
Set rights:
chgrp -R c3-fluiddyn $SNIC_NOBACKUP/hebbe/foam/foam-extend-3.1
chmod -R go+Xr $SNIC_NOBACKUP/hebbe/foam/foam-extend-3.1
Submit script:
#!/usr/bin/env bash
#SBATCH -A SNIC2020-5-103 -p hebbe
#SBATCH -n 4
#SBATCH -t 01:00:00
#-----------------------------------------------------------
ml foss/2019a
export FOAM_INST_DIR=/c3se/NOBACKUP/users/hani/hebbe/foam; . $FOAM_INST_DIR/foam-extend-3.1/etc/bashrc
#------------ Copy and run the damBreak tutorial -----------
#mkdir -p $FOAM_RUN/testDamBreak
#cd $FOAM_RUN/testDamBreak
cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak .
cd damBreak
blockMesh #Generates the grid
setFields #Sets the initial fields
decomposePar #Decomposes for parallel simulation
mpirun interFoam -parallel #Runs interFoam in parallel
reconstructPar #Collects the decomposed results
#rm -rf processor* #Deletes the decomposed files
Experimental: For Paraview, in a separate terminal without OpenFOAM (only in Thinlinc):
ml GCC/8.3.0 OpenMPI/3.1.4 ParaView/5.6.2-Python-3.7.4-mpi vglrun paraview
Clean up (see top of page):
cd ThirdParty
./AllClean
rm -r rpmBuild/BUILD/*
rm -r rpmBuild/BUILDROOT/*
rm -r rpmBuild/TGZS/*
rm -r rpmBuild/RPMS/*
rm rpmBuild/SOURCES/*tar*
foam
wcleanAllButLibBinLnInclude
NOTE: Avoid having to type "Y" at compilation:
echo "export PARAVIEW_SYSTEM=1" >> etc/prefs.sh
cd $SNIC_NOBACKUP/hebbe/foam
git clone https://git.code.sf.net/p/foam-extend/foam-extend-4.1 foam-extend-4.1
cd foam-extend-4.1
Fix problem with:
" /apps/Hebbe7/software/Compiler/GCCcore/8.2.0/binutils/2.31.1/bin/ld.gold: error: --add-needed is not supported but is required for libopen-rte.so.40 in /apps/Hebbe7/software/Compiler/GCC/8.2.0-2.31.1/OpenMPI/3.1.3/lib/libmpi.so "
Comment from Mikael Öhman, C3SE:
" Jag vet att det varit problem med gold-länkaren för vissa version av openfoam. Du får nog använda dig av gamla vanliga BFD-länkaren istället. Du kan specifiera ett länkningskommando på något sätt, jag vet inte hur WMake gör, men, det är "ld.bfd" dom du vill använda under länkningssteget, men, om du har något ställa där du kan lägga till en länkningsflagga; LDFLAGS=" -fuse-ld=bfd " "
sed -i s/fPIC/"fPIC -fuse-ld=bfd"/g wmake/rules/linux64Gcc/c++
Continue:
module load foss/2019a
module load CMake/3.13.3 #Uses same GCCcore as foss/2019a: GCCcore/8.2.0 (checked with 'module spider')
echo "export WM_MPLIB=SYSTEMOPENMPI" >> etc/prefs.sh
echo "export OPENMPI_DIR=$EBROOTOPENMPI" >> etc/prefs.sh
echo "export OPENMPI_BIN_DIR=$EBROOTOPENMPI/bin" >> etc/prefs.sh
export FOAM_INST_DIR=$SNIC_NOBACKUP/hebbe/foam; . $FOAM_INST_DIR/foam-extend-4.1/etc/bashrc
foam
./Allwmake.firstInstall >& log
Y
#Type CTRL-z, bg, and tailf log. Wait until finished
#Do again, since parallel compilation may make things fail:
./Allwmake.firstInstall >& log
Y
#Type CTRL-z, bg, and tailf log. Wait until finished
chgrp -R c3-fluiddyn $SNIC_NOBACKUP/hebbe/foam
chmod -R go+Xr $SNIC_NOBACKUP/hebbe/foam
Submit script:
#!/usr/bin/env bash
#SBATCH -A SNIC2019-3-79 -p hebbe
#SBATCH -n 4
#SBATCH -t 01:00:00
#-----------------------------------------------------------
ml foss/2019a
export FOAM_INST_DIR=/c3se/NOBACKUP/users/hani/hebbe/foam; . $FOAM_INST_DIR/foam-extend-4.1/etc/bashrc
#------------ Copy and run the damBreak tutorial -----------
#mkdir -p $FOAM_RUN/testDamBreak
#cd $FOAM_RUN/testDamBreak
cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak .
cd damBreak
blockMesh #Generates the grid
setFields #Sets the initial fields
decomposePar #Decomposes for parallel simulation
mpirun interFoam -parallel #Runs interFoam in parallel
reconstructPar #Collects the decomposed results
#rm -rf processor* #Deletes the decomposed files
Experimental: For Paraview, in a separate terminal without OpenFOAM (only in Thinlinc):
ml GCC/8.3.0 OpenMPI/3.1.4 ParaView/5.6.2-Python-3.7.4-mpi
vglrun paraview
cd /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/foam/foam-extend-4.1/extend-bazaar
Login at bitbucket.org (login details in usual place), and go to Repositories and solids4foam-release, AND/OR:
git clone https://dr_hakan_nilsson@bitbucket.org/philip_cardiff/solids4foam-release.git
Do some modifications and compile:
find . -name files | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done
find . -name files.foamextend | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done
find . -name files | while read item; do sed -i -e 's/FOAM_USER_APPBIN/FOAM_SITE_APPBIN/g' $item; done
find . -name files.foamextend | while read item; do sed -i -e 's/FOAM_USER_APPBIN/FOAM_SITE_APPBIN/g' $item; done
find . -name options | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done
fe41
./Allwmake &>log_Allwmake
./Allwmake &>log_Allwmake
./Allwmake &>log_Allwmake
cd ..
chmod -R go+Xr solids4foam-release
Test in new terminal window:
fe41
mkdir solids4FoamTest
cd solids4FoamTest
scp -r /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/foam/foam-extend-4.1/extend-bazaar/solids4foam-release/tutorials/fluidSolidInteraction/beamInCrossFlow/elasticBeam .
cd elasticBeam
./Allrun >& log&
Post-process in another terminal window:
paraview_ml
touch case.foam
vglrun paraview
Read in case.foam
Clean up (see top of page):
cd ThirdParty
./AllClean
rm -r rpmBuild/BUILD/*
rm -r rpmBuild/BUILDROOT/*
rm -r rpmBuild/TGZS/*
rm -r rpmBuild/RPMS/*
rm rpmBuild/SOURCES/*tar*
foam
wcleanAllButLibBinLnInclude
NOTE: Avoid having to type "Y" at compilation:
echo "export PARAVIEW_SYSTEM=1" >> etc/prefs.sh
DID NOT WORK, BUT IS KEPT AS TEMPLATE!
ERROR MESSAGES:
/bin/sh: cmake: command not found
#See solution for Gcc (but that solution may mess up compilers)
lnInclude/meshObjectBase.H(70): error: no suitable user-defined conversion from "Foam::HashTable<Foam::meshObjectBase *, Foam::word,
Foam::string::hash>" to "Foam::HashTable<const Foam::meshObjectBase *, Foam::word, Foam::string::hash>" exists
mesh.objectRegistry::template lookupClass<meshObjectBase>();
^
compilation aborted for meshes/polyMesh/polyMeshUpdate.C (code 2)
metisDecomp.H(38): catastrophic error: cannot open source file "decompositionMethod.H"
#include "decompositionMethod.H"
Procedure:
cd $SNIC_NOBACKUP/hebbe/foam
git clone https://git.code.sf.net/p/foam-extend/foam-extend-4.1 foam-extend-4.1
cd foam-extend-4.1
echo "export WM_COMPILER=Icc" >> etc/prefs.sh
echo "export WM_MPLIB=INTELMPI" >> etc/prefs.sh
echo "export MPI_ROOT=\$I_MPI_ROOT" >> etc/prefs.sh
echo "export OPENMPI_DIR=\$I_MPI_ROOT/intel64" >> etc/prefs.sh
module load intel/2019a
export FOAM_INST_DIR=$SNIC_NOBACKUP/hebbe/foam; . $FOAM_INST_DIR/foam-extend-4.1/etc/bashrc
foam
./Allwmake.firstInstall >& log
Y
#Type CTRL-z, bg, and tailf log. Wait until finished
#Do again, since parallel compilation may make things fail:
./Allwmake.firstInstall >& log
Y
#Type CTRL-z, bg, and tailf log. Wait until finished
chgrp -R c3-fluiddyn $SNIC_NOBACKUP/hebbe
chmod -R go+Xr $SNIC_NOBACKUP/hebbe
Clean up (see top of page):
cd ThirdParty
./AllClean
rm -r rpmBuild/BUILD/*
rm -r rpmBuild/BUILDROOT/*
rm -r rpmBuild/TGZS/*
rm -r rpmBuild/RPMS/*
rm rpmBuild/SOURCES/*tar*
foam
wcleanAllButLibBinLnInclude
Read https://www.nsc.liu.se/software/buildenv/. There it says that one should not load the build module when using the compiled code.
cd /proj/hani/OF_inst/Tetralith/OpenFOAM
wget https://sourceforge.net/projects/openfoam/files/v1912/OpenFOAM-v1912.tgz
wget https://sourceforge.net/projects/openfoam/files/v1912/ThirdParty-v1912.tgz
tar xzf OpenFOAM-v1912.tgz
tar xzf ThirdParty-v1912.tgz
rm OpenFOAM-v1912.tgz ThirdParty-v1912.tgz
module load buildenv-intel/2018a-eb; export MPI_ROOT=$I_MPI_ROOT; export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/OpenFOAM; . $FOAM_INST_DIR/OpenFOAM-v1912/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI
cp /proj/hani/OF_inst/addToBashrc /proj/hani/OF_inst/addToBashrc_old
echo "alias OFv1912='export MPI_ROOT=/software/sse/easybuild/prefix/software/impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28; module load ParaView/5.4.1-nsc1-gcc-2018a-eb; export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/OpenFOAM; . $FOAM_INST_DIR/OpenFOAM-v1912/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI'" >> /proj/hani/OF_inst/addToBashrc
foam
./Allwmake >& log.make&
Look for 'rror' and 'skip' in log file:
Error building: boost
---------------------------------------------------
Optional component (CGAL) had build issues
OpenFOAM will nonetheless remain largely functional
---------------------------------------------------
CMake Error at CMakeLists.txt:6 (cmake_minimum_required):
CMake 3.6 or higher is required. You are running version 2.8.12.2
-- Configuring incomplete, errors occurred!
Error building: ADIOS2-2.4.0
---------------------------------------------------
Optional component (ADIOS2) had build issues
OpenFOAM will nonetheless remain largely functional
---------------------------------------------------
==> skip metis (no header)
==> skip SloanRenumber (no boost)
==> skip zoltanRenumber (no library)
==> skip ccmio (no header)
==> skip optional libccm adapter
WARNING: skip runTimePostProcessing (no VTK or ParaView)
==> skip optional ccm conversion components (no libccm)
==> skip paraview-plugin (could not determine target or major.minor version)
WARNING: skip ParaView Catalyst (missing or incorrrect ParaView)
skip wmake target for modules doxygen directory
Note that metis is not available, but scotch!
Make it available:
cd ..
chmod -R go+Xr OpenFOAM-v1912
chmod -R go+Xr ThirdParty-v1912
Test by submitting the damBreak case. First create a submit script named submitDambreak_OFv1912_Tetralith (note that mpprun did not work):
#!/bin/bash
#
#SBATCH -J damBreak #Name of job
#SBATCH -t 00:30:00 #HH:MM:SS
#SBATCH -n 4
#SBATCH --exclusive
#
#Same as alias OFv1912 (without ParaView):
export MPI_ROOT=/software/sse/easybuild/prefix/software/impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28; export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/OpenFOAM; . $FOAM_INST_DIR/OpenFOAM-v1912/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI
#Prepare
cp -r $FOAM_TUTORIALS/multiphase/interFoam/RAS/damBreak/damBreak .
blockMesh -case damBreak
setFields -case damBreak
decomposePar -case damBreak
#--nranks is used when less than the number of available cores is used
mpirun interFoam -parallel -case damBreak >& log #mpprun did not work!!!
# Script ends here
Submit by (add -A <projnumber> just after sbatch if you have access to more than one project):
sbatch submitDambreak_OFv1912_Tetralith
Check if it is running (change x_hakni to your user name):
squeue -u x_hakni
Clean ThirdParty (see notes at top of page):
Only the "platforms" directory needs to be kept after a successful installation. Just remove the rest, by (where the "dummy" shows how to keep more than one directory if necessary):
cd $WM_THIRD_PARTY_DIR
###############################################################################################
# MAKE SURE YOU ARE IN THE ThirdParty DIRECTORY, SINCE THIS IS A DANGEROUS RECURSIVE COMMAND!!!
# IT IS COMMENTED HERE, SO YOU DON'T MAKE ANY COPY-PASTE MISTAKE
###############################################################################################
#shopt -s extglob
#rm -rv !("platforms"|"dummy")
cd /chalmers/sw/unsup64/OpenFOAM
wget https://sourceforge.net/projects/openfoam/files/v1912/OpenFOAM-v1912.tgz
wget https://sourceforge.net/projects/openfoam/files/v1912/ThirdParty-v1912.tgz
tar xzf OpenFOAM-v1912.tgz
tar xzf ThirdParty-v1912.tgz
rm OpenFOAM-v1912.tgz ThirdParty-v1912.tgz
cp /chalmers/sw/unsup64/OpenFOAM/addToBashrc /chalmers/sw/unsup64/OpenFOAM/addToBashrc_backup
echo "alias OFv1912='source /chalmers/sw/unsup64/OpenFOAM/OpenFOAM-v1912/etc/bashrc;alias paraview=\"echo Using system paraview;paraview-mesa paraview\";alias paraFoam=\"echo Using system paraview \\(File/Open case.foam\\);touch case.foam;paraview-mesa paraview\"'" >> /chalmers/sw/unsup64/OpenFOAM/addToBashrc
. /chalmers/sw/unsup64/OpenFOAM/addToBashrc
OFv1912
foam
./Allwmake >& log.make&
Make the installation available
foam
cd ..
chmod -R go+Xr OpenFOAM-v1912
chmod -R go+Xr ThirdParty-v1912
cd /chalmers/sw/unsup64/
mkdir foam; cd foam
git clone git://git.code.sf.net/p/openfoam-extend/foam-extend-3.1 foam-extend-3.1
cd foam-extend-3.1
sed -i s/"http\:\/\/portal.nersc.gov\/svn\/visit\/tags\/2.4.2\/third_party\/libccmio-2.6.1.tar.gz"/"http\:\/\/downloads.sourceforge.net\/project\/foam-extend\/ThirdParty\/libccmio-2.6.1.tar.gz"/g ThirdParty/AllMake.stage3
cp /chalmers/sw/unsup64/OpenFOAM/addToBashrc /chalmers/sw/unsup64/OpenFOAM/addToBashrc_backup
echo "alias fe31='export FOAM_INST_DIR=/chalmers/sw/unsup64/foam; . \$FOAM_INST_DIR/foam-extend-3.1/etc/bashrc;alias paraview=\"echo Using system paraview;paraview-mesa paraview\";alias paraFoam=\"echo Using system paraview \\(File/Open case.foam\\);touch case.foam;paraview-mesa paraview\"'" >> /chalmers/sw/unsup64/OpenFOAM/addToBashrc
. /chalmers/sw/unsup64/OpenFOAM/addToBashrc
fe31
./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y"
./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y"
chmod -R go+Xr ../foam-extend-3.1
FSI:
cd extend-bazaar/
wget http://openfoamwiki.net/images/5/52/Fsi_31.tar.gz
tar zxvf Fsi_31.tar.gz
rm Fsi_31.tar.gz
cd FluidStructureInteraction/
find src -name options | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done
find src -name files | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done
find run -name options | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done
#Change to -I$(WM_PROJECT_DIR)/extend-bazaar/FluidStructureInteraction/src/fluidStructureInteraction/lnInclude
#in:
#run/fsiFoam/HronTurekFsi3/hronTurekReport/Make/options
#run/fsiFoam/beamInCrossFlow/setInletVelocity/Make/options
#run/stressFoam/plateHole/setPlateHoleBC/Make/options
#
pushd src/fluidStructureInteraction/stressModels/constitutiveModel
wget https://raw.githubusercontent.com/wyldckat/FluidStructureInteraction/fe31/src/fluidStructureInteraction/stressModels/constitutiveModel/constitutiveModel.C -O constitutiveModel.C
wget https://raw.githubusercontent.com/wyldckat/FluidStructureInteraction/fe31/src/fluidStructureInteraction/stressModels/constitutiveModel/constitutiveModel.H -O constitutiveModel.H
popd
cd src/
./Allwmake >& log&
./Allwmake >& log&
chmod -R go+Xr ../../FluidStructureInteraction/
Clean up (see top of page):
cd ThirdParty
./AllClean
rm -r rpmBuild/BUILD/*
rm -r rpmBuild/BUILDROOT/*
rm -r rpmBuild/TGZS/*
rm -r rpmBuild/RPMS/*
rm rpmBuild/SOURCES/*tar*
foam
wcleanAllButLibBinLnInclude
cd /chalmers/sw/unsup64/
mkdir foam; cd foam
git clone https://git.code.sf.net/p/foam-extend/foam-extend-3.2 foam-extend-3.2
cd foam-extend-3.2
sed -i s/"http\:\/\/portal.nersc.gov\/project\/visit\/third_party\/libccmio-2.6.1.tar.gz"/"http\:\/\/downloads.sourceforge.net\/project\/foam-extend\/ThirdParty\/libccmio-2.6.1.tar.gz"/g ThirdParty/AllMake.stage3
cp /chalmers/sw/unsup64/OpenFOAM/addToBashrc /chalmers/sw/unsup64/OpenFOAM/addToBashrc_backup
echo "alias fe32='export FOAM_INST_DIR=/chalmers/sw/unsup64/foam; . \$FOAM_INST_DIR/foam-extend-3.2/etc/bashrc;alias paraview=\"echo Using system paraview;paraview-mesa paraview\";alias paraFoam=\"echo Using system paraview \\(File/Open case.foam\\);touch case.foam;paraview-mesa paraview\"'" >> /chalmers/sw/unsup64/OpenFOAM/addToBashrc
. /chalmers/sw/unsup64/OpenFOAM/addToBashrc
fe32
./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y"
./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y"
chmod -R go+Xr ../foam-extend-3.2
cd /chalmers/sw/unsup64/
mkdir foam; cd foam
git clone https://git.code.sf.net/p/foam-extend/foam-extend-4.1 foam-extend-4.1
cd foam-extend-4.1/
cp /chalmers/sw/unsup64/OpenFOAM/addToBashrc /chalmers/sw/unsup64/OpenFOAM/addToBashrc_backup
echo "alias fe41='export FOAM_INST_DIR=/chalmers/sw/unsup64/foam; . \$FOAM_INST_DIR/foam-extend-4.1/etc/bashrc;alias paraview=\"echo Using system paraview;paraview-mesa paraview\";alias paraFoam=\"echo Using system paraview \\(File/Open case.foam\\);touch case.foam;paraview-mesa paraview\"'" >> /chalmers/sw/unsup64/OpenFOAM/addToBashrc
. /chalmers/sw/unsup64/OpenFOAM/addToBashrc
fe41
./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y"
./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y"
chmod -R go+Xr ../foam-extend-4.1
Regarding FSI package, which I did not manage to compile at this point: See details for FOAM-extend-4.0 below. In addition: Remove -Werror from compiler flags and fix some problems, and compile again:
sed -i s/"-Werror "/""/g FluidSolidInteraction/src/fluidSolidInteraction/Make/options
sed -i s/"procPatch.owner()"/"procPatch.master()"/g FluidSolidInteraction/src/fluidSolidInteraction/solidSolvers/solidModels/constitutiveModel/plasticityStressReturnMethods/kirchhoffMises/kirchhoffMises.C
sed -i s/"procPatch.neighbour()"/"\!procPatch.master()"/g FluidSolidInteraction/src/fluidSolidInteraction/solidSolvers/solidModels/materialInterfaces/ULLSMaterialInterface/ULLSMaterialInterface.C
cd /chalmers/sw/unsup64/
mkdir foam; cd foam
git clone https://git.code.sf.net/p/foam-extend/foam-extend-4.0 foam-extend-4.0
cd foam-extend-4.0/
#Fix libccmio link:
sed -i s/"http:\/\/portal.nersc.gov\/project\/visit\/third_party\/"/"http:\/\/downloads.sourceforge.net\/project\/foam-extend\/ThirdParty\/"/g ThirdParty/AllMake.stage3
#Fix problem with setFluxRequired for FSI, see https://www.cfd-online.com/Forums/openfoam-cc-toolkits-fluid-structure-interaction/185685-fsifoam-4-0-can-not-run-tutorials-2.html
sed -i '1044,1051s/.*/\/\/&/' $WM_PROJECT_DIR/src/finiteVolume/fvMatrices/fvMatrix/fvMatrix.C
cp /chalmers/sw/unsup64/OpenFOAM/addToBashrc /chalmers/sw/unsup64/OpenFOAM/addToBashrc_backup
echo "alias fe40='export FOAM_INST_DIR=/chalmers/sw/unsup64/foam; . \$FOAM_INST_DIR/foam-extend-4.0/etc/bashrc;alias paraview=\"echo Using system paraview;paraview-mesa paraview\";alias paraFoam=\"echo Using system paraview \\(File/Open case.foam\\);touch case.foam;paraview-mesa paraview\"'" >> /chalmers/sw/unsup64/OpenFOAM/addToBashrc
. /chalmers/sw/unsup64/OpenFOAM/addToBashrc
fe40
./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y"
./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y"
chmod -R go+Xr ../foam-extend-4.0
FSI package:
cd extend-bazaar
export EXTEND_BAZAAR_TO_SITE=1 #To make available to everyone
./Allwmake >& log&
#Fix some problems due to installation paths, and compile again:
cd FluidSolidInteraction
find src -name options | while read item
do
sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item
done
find run -name options | while read item
do
sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item
sed -i -e 's=$(WM_PROJECT_DIR)/applications/solvers/FSI=$(WM_PROJECT_DIR)/extend-bazaar/FluidSolidInteraction/src=' $item
sed -i -e 's=$(WM_THIRD_PARTY_DIR)/packages/eigen3=$(WM_PROJECT_DIR)/extend-bazaar/FluidSolidInteraction/src/ThirdParty/eigen3=' $item
done
find run -name files | while read item
do
sed -i -e 's/FOAM_SITE_LIBBIN/FOAM_USER_LIBBIN/g' $item
done
#Fix bug due to version update
sed -i s/'"Time.H"'/'"foamTime.H"'/g MakeAxialMesh/makeAxialMesh/makeAxialMesh.C
cd FluidSolidInteraction
./Allwmake >& log&
#Strangely, fix linking to MPI, by:
fe40
./Allwmake >& log&
chmod -R go+Xr ../../foam-extend-4.0
Test FSI package:
cp -r $WM_PROJECT_DIR/extend-bazaar/FluidSolidInteraction/run $FOAM_RUN/FSI_run
cd $FOAM_RUN/FSI_run/fsiFoam/HronTurekFsi3/fluid
#Change endTime to 4!!! At 2.4s there are only small movements.
./Allrun >& log&
We here compile OpenFOAM-v1906 with ThirdParty gcc-8.2.0 and openmpi-1.10.7, since they are needed for PreCICE and Calculix. It was fine to compile only OpenFOAM with system gcc-4.8.5 and openmpi. Note that it seems to be important that both OpenFOAM, ThirdParty, and all packages for PreCICE are compiled using the same Gcc. It particularly seems like it was important that yaml was compiled with the same compiler as OFv1906 (8.2.0). The LD_LIBRARY_PATHs related to preCICE, yaml, boost_1_65_1, Petsc and Calculix in alias OFv1906 were added after entire procedure, since they are needed to use those libraries. It can be seen in the instructions when they were needed for the compilations.
It is mentioned in the instructions for PreCICE that the Boost version should be 1.65.1 or higher. For OpenFOAM-v1906, the ThirdParty Boost version is 1.64.0. Below I have used 1.64.0 for OpenFOAM and 1.65.1 for PreCICE. It would be nicer to use only one of them.
cd /chalmers/sw/unsup64/OpenFOAM
wget https://sourceforge.net/projects/openfoam/files/v1906/OpenFOAM-v1906.tgz
wget https://sourceforge.net/projects/openfoam/files/v1906/ThirdParty-v1906.tgz
tar xzf OpenFOAM-v1906.tgz
tar xzf ThirdParty-v1906.tgz
rm OpenFOAM-v1906.tgz ThirdParty-v1906.tgz
cp /chalmers/sw/unsup64/OpenFOAM/addToBashrc /chalmers/sw/unsup64/OpenFOAM/addToBashrc_backup
echo "alias OFv1906='source /chalmers/sw/unsup64/OpenFOAM/OpenFOAM-v1906/etc/bashrc WM_COMPILER_TYPE=ThirdParty WM_COMPILER=Gcc82 WM_MPLIB=OPENMPI; export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:/chalmers/sw/unsup64/OpenFOAM/ThirdParty-v1906/platforms/linux64/mpfr-4.0.1/lib64; export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:/chalmers/sw/unsup64/OpenFOAM/ThirdParty-v1906/preCICE/precice-1.6.1/lib; export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:/chalmers/sw/unsup64/CalculiX/yaml-cpp-yaml-cpp-0.6.2/build; export LD_LIBRARY_PATH=/chalmers/sw/unsup64/OpenFOAM/ThirdParty-v1906/preCICE/boost_1_65_1/lib:\$LD_LIBRARY_PATH; export LD_LIBRARY_PATH=/chalmers/sw/unsup64/OpenFOAM/ThirdParty-v1906/preCICE/petsc/arch-linux2-c-opt/lib:\$LD_LIBRARY_PATH; export PATH=\$PATH:/chalmers/sw/unsup64/CalculiX/CalculiX/calculix-adapter-master/bin; alias paraview=\"echo Using system paraview;paraview-mesa paraview\";alias paraFoam=\"echo Using system paraview \\(File/Open case.foam\\);touch case.foam;paraview-mesa paraview\"'" >> /chalmers/sw/unsup64/OpenFOAM/addToBashrc
. /chalmers/sw/unsup64/OpenFOAM/addToBashrc
OFv1906
foam
Install Gcc-8.2.0, INCLUDING FORTRAN - FOR CALCULIX THAT NEEDS gfortran and it was important to be from the same gcc, (this takes hours! See dependencies at e.g. http://www.linuxfromscratch.org/lfs/view/8.3-systemd-rc1/chapter05/gcc-pass1.html):
cd ../ThirdParty-v1906
#Include fortran (gfortran, for Calculix):
sed -i s/"--enable-languages=c,c++"/"--enable-languages=c,c++,fortran"/g makeGcc
wget https://ftp.gnu.org/gnu/gcc/gcc-8.2.0/gcc-8.2.0.tar.gz
tar xzf gcc-8.2.0.tar.gz
rm gcc-8.2.0.tar.gz
wget https://ftp.gnu.org/gnu/mpfr/mpfr-4.0.1.tar.gz
tar xzf mpfr-4.0.1.tar.gz
rm mpfr-4.0.1.tar.gz
wget https://ftp.gnu.org/gnu/gmp/gmp-6.1.2.tar.xz
tar xf gmp-6.1.2.tar.xz
rm gmp-6.1.2.tar.xz
wget https://ftp.gnu.org/gnu/mpc/mpc-1.1.0.tar.gz
tar xzf mpc-1.1.0.tar.gz
rm mpc-1.1.0.tar.gz
export WM_ARCH_OPTION=64
./makeGcc gcc-8.2.0 gmp-6.1.2 mpfr-4.0.1 mpc-1.1.0 >& log_makeGcc&
#Check:
which gfortran #Should give /chalmers/sw/unsup64/OpenFOAM/ThirdParty-v1906/platforms/linux64/gcc-8.2.0/bin/gfortran
Install OpenMPI, INCLUDING FORTRAN - FOR PETSC THAT NEEDS mpif90, and it was important to be from the same gcc. Petsc was needed for the OpenFOAM FSI adaptor by PreCICE (RBF mapping).
sed -i s/"--enable-mpi-fortran=none"/"--enable-mpi-fortran=yes"/g makeOPENMPI
./makeOPENMPI openmpi-1.10.7 >& log_makeOPENMPI&
#Check:
which mpif90 #Should give /chalmers/sw/unsup64/OpenFOAM/ThirdParty-v1906/platforms/linux64Gcc82/openmpi-1.10.7/bin/mpif90
Compile OpenFOAM
foam
./Allwmake >& log.make&
Make the installation available
foam
cd ..
chmod -R go+Xr OpenFOAM-v1906
chmod -R go+Xr ThirdParty-v1906
Tried also to install ParaView (but problems with OpenGL, so commented - don't do this):
#cd /chalmers/sw/unsup64/OpenFOAM/ThirdParty-v1906
#./makeParaView >& log.makeParaView&
#cd $WM_PROJECT_DIR/applications/utilities/postProcessing/graphics/PVReaders
#./Allwclean
#./Allwmake
#chmod -R go+Xr ../../graphics
Now complete the instructions at http://www.tfd.chalmers.se/~hani/wikis/haniWiki/index.php?title=Tips_and_tricks_/_preCICE_/_Installation#preCICE-1.6.1_for_OFv1906_in_StuDAT_.28non-root.29.2C_Redhat_Enterprise_7_.2C_2019-12-26
Note that the following lines were typed after the installation, so be careful copy-pasting! The instructions are guidelines.
cd /proj/hani/OF_inst/Tetralith/foam
git clone git://git.code.sf.net/p/foam-extend/foam-extend-4.1 foam-extend-4.1_hani
cd foam-extend-4.1_hani
cp etc/bashrc etc/bashrc_hani
sed -i s/"export WM_PROJECT_VERSION=4.1"/"export WM_PROJECT_VERSION=4.1_hani"/g etc/bashrc_hani
module load buildenv-gcc/2018a-eb
export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/foam; . $FOAM_INST_DIR/foam-extend-4.1_hani/etc/bashrc_hani WM_MPLIB=SYSTEMOPENMPI OPENMPI_DIR=/software/sse/easybuild/prefix/software/OpenMPI/2.1.2-GCC-6.4.0-2.28 OPENMPI_BIN_DIR=/software/sse/easybuild/prefix/software/OpenMPI/2.1.2-GCC-6.4.0-2.28/bin
foam
./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y"
./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y"
chgrp -R pg_hani ../foam-extend-4.1_hani
chmod -R go+Xr ../foam-extend-4.1_hani
Submit-script to test:
#!/bin/bash
#
#SBATCH -n 4
#SBATCH -t 00:20:00
#SBATCH -J damBreak
#SBATCH -A snic2018-3-678
#OpenFOAM versions:
#module load buildenv-gcc/2018a-eb
export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/foam; . $FOAM_INST_DIR/foam-extend-4.1_hani/etc/bashrc_hani WM_MPLIB=SYSTEMOPENMPI OPENMPI_DIR=/software/sse/easybuild/prefix/software/OpenMPI/2.1.2-GCC-6.4.0-2.28 OPENMPI_BIN_DIR=/software/sse/easybuild/prefix/software/OpenMPI/2.1.2-GCC-6.4.0-2.28/bin
cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak .
blockMesh -case damBreak
setFields -case damBreak
decomposePar -case damBreak
mpirun interFoam -parallel -case damBreak &> result.out
cd /proj/hani/OF_inst/Tetralith/foam
git clone -b nextRelease git://git.code.sf.net/p/foam-extend/foam-extend-4.0 foam-extend-4.0
cd foam-extend-4.0
git tag -l
git checkout tags/v4.0
echo "export WM_COMPILER=Icc" >> etc/prefs.sh
echo "export WM_MPLIB=INTELMPI" >> etc/prefs.sh
echo "export MPI_ROOT=\$I_MPI_ROOT" >> etc/prefs.sh
echo "export OPENMPI_DIR=\$I_MPI_ROOT/intel64" >> etc/prefs.sh
module load buildenv-intel/2018a-eb
export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/foam; . $FOAM_INST_DIR/foam-extend-4.0/etc/bashrc
foam
./Allwmake.firstInstall >& log&
chgrp -R pg_hani Tetralith
chmod -R go+Xr Tetralith
Test by submitting the damBreak case. First create a submit script named submitDambreak:
#!/bin/bash
#
#SBATCH -n 4
#SBATCH -t 00:20:00
#SBATCH -J damBreak
#SBATCH -A snic2017-1-651
module load buildenv-intel/2018a-eb
export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/foam; source $FOAM_INST_DIR/foam-extend-4.0/etc/bashrc
cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak .
blockMesh -case damBreak
setFields -case damBreak
decomposePar -case damBreak
mpirun interFoam -parallel -case damBreak &> result.out #mpprun did not work!!!
# Script ends here
Followed some of the instructions at http://openfoamwiki.net/index.php/Installation/Linux/OpenFOAM%2B-v1612%2B/CentOS_SL_RHEL
Skipping Qt and Paraview at the moment
cd /chalmers/sw/unsup64/OpenFOAM
git clone https://develop.openfoam.com/Development/OpenFOAM-plus.git #This requires a user name and password from your registration
mv OpenFOAM-plus OpenFOAM-v1706
cd OpenFOAM-v1706
git checkout OpenFOAM-v1706
cd ..
The following on one line (starting with wget, ending with vorboss, and no spaces after https):
wget -O ThirdParty-v1706.tgz https://downloads.sourceforge.net/project/openfoamplus/v1706/ThirdParty-v1706.tgz?r=http%3A%2F%2Fopenfoam.com%2Fcode%2Frepositories.php&ts=1502098894&use_mirror=vorboss
Press enter to get prompt back.
md5sum ThirdParty-v1706.tgz #Should give 003d38fe3f97ccaa291bafbf94c768eb
tar xzf ThirdParty-v1706.tgz
rm ThirdParty-v1706.tgz
gcc needs a higher version that 4.4.7, which is in the system. It was recommended at the above link to also install binutils, so I do that (it is fast). I also installed openmpi since it is not in the system. cmake is already version 3.7.2, which is above required for paraview and CGAL
sed -i -e 's=gmp-system=gmp-5.1.2=' -e 's=mpfr-system=mpfr-3.1.2=' -e 's=mpc-system=mpc-1.0.1=' OpenFOAM-v1706/etc/config.sh/compiler
source /chalmers/sw/unsup64/OpenFOAM/OpenFOAM-v1706/etc/bashrc WM_COMPILER_TYPE=ThirdParty WM_MPLIB=OPENMPI
cd $WM_THIRD_PARTY_DIR
wget "https://raw.github.com/wyldckat/scripts4OpenFOAM3rdParty/master/getGcc"
wget "https://raw.github.com/wyldckat/ThirdParty-2.0.x/binutils/makeBinutils"
wget "https://raw.github.com/wyldckat/ThirdParty-2.0.x/binutils/getBinutils"
chmod +x get* make*
./getGcc gcc-4.8.5 gmp-5.1.2 mpfr-3.1.2 mpc-1.0.1
./makeGcc > log.mkgcc 2>&1
wmREFRESH #I actually didn't do this, but the next lines worked anyway.
./getBinutils
./makeBinutils gcc-4.8.5 >& log.mkbinutils #Not until gcc-4.8.5 is built
./makeOPENMPI >& log_makeOPENMPI
The following echo command once destroyed the original file. Make a copy of it first and make sure that the echo command worked.
cp /chalmers/sw/unsup64/OpenFOAM/addToBashrc /chalmers/sw/unsup64/OpenFOAM/addToBashrc_backup
echo "alias OF1706+='source /chalmers/sw/unsup64/OpenFOAM/OpenFOAM-v1706/etc/bashrc WM_COMPILER_TYPE=ThirdParty WM_MPLIB=OPENMPI'" >> /chalmers/sw/unsup64/OpenFOAM/addToBashrc
Opened a new terminal window and typed:
OF1706+
foam
./Allwmake >& log.make
Make the installation available
foam
cd ..
chmod -R go+Xr OpenFOAM-v1706
chmod -R go+Xr ThirdParty-v1706
HP SL230s compute nodes, equipped with two Intel E5-2660 (2.2 GHz Sandybridge) processors with 8 cores each, i.e 16 cores per node. Hyper-threading is not enabled. Infiniband from Mellanox (FDR IB, 56 Gb/s)
I do not bother about paraview at the moment
mkdir -p /proj/hani/OF_inst/Triolith/OpenFOAM
cd /proj/hani/OF_inst/Triolith/OpenFOAM
wget http://downloads.sourceforge.net/foam/OpenFOAM-3.0.1.tgz?use_mirror=mesh --output-document=OpenFOAM-3.0.1.tgz
wget http://downloads.sourceforge.net/foam/ThirdParty-3.0.1.tgz?use_mirror=mesh --output-document=ThirdParty-3.0.1.tgz
tar -xzf OpenFOAM-3.0.1.tgz
tar -xzf ThirdParty-3.0.1.tgz
rm OpenFOAM-3.0.1.tgz
rm ThirdParty-3.0.1.tgz
cp -r /proj/hani/OF_inst/Triolith/OpenFOAM/OpenFOAM-3.0.1/wmake/rules/linux64Icc /proj/hani/OF_inst/Triolith/OpenFOAM/OpenFOAM-3.0.1/wmake/rules/linux64IccHani
sed -i s/fPIC/"fPIC -Nmpi -Nmkl -xavx"/g /proj/hani/OF_inst/Triolith/OpenFOAM/OpenFOAM-3.0.1/wmake/rules/linux64IccHani/c++
Used the currently recommended version at the cluster for impi, intel compiler, and mkl. (see instructions for v1606+):
alias OF301='export FOAM_INST_DIR=/proj/hani/OF_inst/Triolith/OpenFOAM; module load impi/5.0.2.044 intel/15.0.1.133 mkl/11.2.1; export MPI_ROOT=$I_MPI_ROOT; . $FOAM_INST_DIR/OpenFOAM-3.0.1/etc/bashrc WM_COMPILER=IccHani WM_MPLIB=INTELMPI'
OF301
foam
./Allwmake >& log&
Problem with SloanRenumber: cannot find -lboost_thread
Commented the compilation of SloanRenumber in src/renumber/Allwmake:
- Code from here (not to paste)
#if [ -n "$BOOST_ARCH_PATH" ]
#then
# wmake $targetType SloanRenumber
#else
echo
echo "Skipping SloanRenumber"
echo
#fi
- Code to here (not to paste)
Problem with foamyMesh: cannot open source file "CGAL/Delaunay_triangulation_3.h", "CGAL/Delaunay_triangulation_2.h"
Commented compilation in applications/utilities/mesh/generation/Allwmake:
- Code from here (not to paste)
#if [ -n "$CGAL_ARCH_PATH" ]
#then
# foamyMesh/Allwmake
#fi
- Code to here (not to paste)
The above are related, since CGAL requires boost. I do not bother at the moment since we don't use the above functionalities.
Note that as the above errors occurred it seems like some additional parts were not compiled, so run the next line a couple of times until all are already "up to date".
./Allwmake >& log&
Make it available:
cd ..
chmod -R go+Xr OpenFOAM-3.0.1 ThirdParty-3.0.1
chgrp -R pg_hani OpenFOAM-3.0.1 ThirdParty-3.0.1
cd /proj/hani/OF_inst/Triolith/foam
git clone git://git.code.sf.net/p/foam-extend/foam-extend-3.2 foam-extend-4.0
cd foam-extend-4.0
git checkout nextRelease
Do required modifications to the installation procedure, in this case use Qt-4.8.6 from module (check version in ThirdParty/AllMake.stage4 and etc/prefs.sh-EXAMPLES):
module load qt/4.8.6 paraview/4.4.0
echo "export QT_DIR=/software/apps/qt/4.8.6" >> etc/prefs.sh
echo "export QT_BIN_DIR=\$QT_DIR/bin" >> etc/prefs.sh
echo "export PARAVIEW_SYSTEM=1" >> etc/prefs.sh
echo "export PARAVIEW_DIR=/software/apps/paraview/4.4.0" >> etc/prefs.sh
echo "export PARAVIEW_BIN_DIR=\$PARAVIEW_DIR/bin" >> etc/prefs.sh
echo "export WM_COMPILER=Icc" >> etc/prefs.sh
echo "export WM_MPLIB=INTELMPI" >> etc/prefs.sh
echo "export MPI_ROOT=\$I_MPI_ROOT" >> etc/prefs.sh
echo "export OPENMPI_DIR=\$I_MPI_ROOT/intel64" >> etc/prefs.sh
There is a problem with compilation of cfMesh using the Intel compiler giving messages like:
polyMeshGenModifierReorderBoundaryFaces.C(224): internal error: null pointer
Found a solution in:
Applied by:
sed -i.orig 's/EXE_INC/OMP_FLAGS =\nEXE_INC/g' $FOAM_SRC/mesh/cfMesh/Make/options
Used the currently recommended version at the cluster for impi, intel compiler, and mkl. Get the recommended version numbers by:
module load impi
module load intel
module load mkl
It was realized that mkl was needed by a compilation error message (when compiling v1606+). I don't know what it is used for.
It is necessary to add a couple of compilation flags for mpi at Triolith:
sed -i s/fPIC/"fPIC -Nmpi -Nmkl -xavx"/g $FOAM_INST_DIR/foam-extend-4.0/wmake/rules/linux64Icc/c++
(error message otherwise: Could not retrieve MPI tag from /proj/hani/OF_inst/Triolith/foam/foam-extend-4.0/applications/bin/linux64IccDPOpt/interFoam)
Source and prepare:
alias OFextend40='export FOAM_INST_DIR=/proj/hani/OF_inst/Triolith/foam; module load impi/5.0.2.044 intel/15.0.1.133 mkl/11.2.1 qt/4.8.6 paraview/4.4.0; . $FOAM_INST_DIR/foam-extend-4.0/etc/bashrc'
OFextend40
foam
Compile all (perhaps do this a couple of times and check that all is compiled in the log file):
./Allwmake.firstInstall >& log&
Make it available:
cd ..
chmod -R go+Xr foam-extend-4.0
chgrp -R pg_hani foam-extend-4.0
Test by submitting the damBreak case. First create a submit script named submitDambreak:
#!/bin/bash
#
#SBATCH -J damBreak #Name of job
#SBATCH -t 00:30:00 #HH:MM:SS
#SBATCH -N 1
#SBATCH --exclusive
#
#Same as alias OFextend40:
export FOAM_INST_DIR=/proj/hani/OF_inst/Triolith/foam
module load impi/5.0.2.044 intel/15.0.1.133 mkl/11.2.1 qt/4.8.6 paraview/4.4.0
. $FOAM_INST_DIR/foam-extend-4.0/etc/bashrc
#Prepare
cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak .
blockMesh -case damBreak
setFields -case damBreak
decomposePar -case damBreak
#--nranks is used when less than the number of available cores is used
mpprun --nranks=4 `which interFoam` -parallel -case damBreak >& log
# Script ends here
Submit by:
sbatch submitDambreak
Check if it is running (change x_hakni to your user name):
squeue -u x_hakni
mkdir -p /c3se/NOBACKUP/users/hani/hebbe/OpenFOAM
cd /c3se/NOBACKUP/users/hani/hebbe/OpenFOAM
git clone git://github.com/OpenFOAM/OpenFOAM-4.x.git
git clone git://github.com/OpenFOAM/ThirdParty-4.x.git
. /apps/new_modules.sh #To use the new modules
ml load iimpi/2016b #For Intel compiler and MPI
ml load Boost/1.61.0 #For SloanRenumber)
ml load CGAL/4.8.1 #For foamyMesh)
Compiler: either GCC version 4.5 or above; or LLVM Clang version 3.6 or above; or Intel ICC version 15.0.3 or above. GCC is most commonly available and the version can be checked by typing
icc --version
Gives:
icc (ICC) 16.0.3 20160415
Source (note that the installation path is automatically detected):
. OpenFOAM-4.x/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI MPI_ROOT=/apps/Hebbe/software/Compiler/intel/2016.3.210-GCC-5.4.0-2.26/impi/5.1.3.181
Compile ThirdParty:
cd ThirdParty-4.x
./Allwmake >& log&
./Allwmake >& log& #Check log file for errors
Skipped compiling Paraview
Compile OpenFOAM:
cd ../OpenFOAM-4.x
./Allwmake -j >& log&
./Allwmake -j >& log& #Got some HANG-UP the first time, so continued with this line
./Allwmake -j >& log& #Check for errors in the log file
rm log
Meanwhile, check revision:
git rev-parse --short HEAD
In this case:
56a4152
Example of a submit script for parallel simulation (submit by "sbatch -A PROJ_NUMBER scriptName"):
#!/usr/bin/env bash
#SBATCH -N 1 -n 4 #Will use only 4 cores on 1 node (there are 20 cores per node on Hebbe)
#SBATCH -p hebbe
#SBATCH -t 00:01:00
#---------------- Uncomment ONE module line -------------------------
. /apps/new_modules.sh; ml load iimpi/2016b Boost/1.61.0 CGAL/4.8.1; . /c3se/NOBACKUP/users/hani/hebbe/OpenFOAM/OpenFOAM-4.x/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI MPI_ROOT=/apps/Hebbe/software/Compiler/intel/2016.3.210-GCC-5.4.0-2.26/impi/5.1.3.181
#------------ Copy and run the damBreak tutorial --------------------
cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak/damBreak .
cd damBreak
blockMesh >& log_blockMesh #Generates the grid
setFields >& log_setFields #Sets the initial fields
decomposePar >& log_decomposePar #Decomposes for parallel simulation
mpirun interFoam -parallel >& log #Runs interFoam in parallel on 4 cores (flags -N 1 -n 4 above)
reconstructPar >& log_reconstructPar #Collects the decomposed results
rm -rf processor* #Deletes the decomposed files
Example of submitscript for sequential simulation:
#!/usr/bin/env bash
#SBATCH -N 1 -n 1
#SBATCH -p hebbe
#SBATCH -t 00:01:00
#---------------- Uncomment ONE module line -------------------------
. /apps/new_modules.sh; ml load iimpi/2016b Boost/1.61.0 CGAL/4.8.1; . /c3se/NOBACKUP/users/hani/hebbe/OpenFOAM/OpenFOAM-4.x/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI MPI_ROOT=/apps/Hebbe/software/Compiler/intel/2016.3.210-GCC-5.4.0-2.26/impi/5.1.3.181
#------------ Copy and run the damBreak tutorial --------------------
cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak/damBreak .
cd damBreak
blockMesh >& log_blockMesh #Generates the grid
setFields >& log_setFields #Sets the initial fields
interFoam >& log #Runs interFoam in sequential mode
Run at log-in node (only for very short tests and small cases): Add to the end of your $HOME/.bashrc file:
. /c3se/NOBACKUP/users/hani/hebbe/addToBashrc
Then open a new window at Hebbe and type:
OF4x
mkdir -p $FOAM_RUN
cd $FOAM_RUN
cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak/damBreak .
cd damBreak
blockMesh >& log_blockMesh #Generates the grid
setFields >& log_setFields #Sets the initial fields
interFoam >& log& #Runs interFoam in sequential mode
The rest is not tested!!!
Paraview is already available in two versions. You see them by typing:
module load extras
module avail
Then either:
module load paraview/3.14
or:
module load paraview/4.4.0
and then:
paraFoam -nativeReader
HP SL230s compute nodes, equipped with two Intel E5-2660 (2.2 GHz Sandybridge) processors with 8 cores each, i.e 16 cores per node. Hyper-threading is not enabled. Infiniband from Mellanox (FDR IB, 56 Gb/s)
I do not bother about paraview at the moment
mkdir -p /proj/hani/OF_inst/Triolith/OpenFOAM
cd /proj/hani/OF_inst/Triolith/OpenFOAM
wget "http://downloads.sourceforge.net/openfoamplus/files/OpenFOAM-v1606%2B.tgz?use_mirror=mesh" -O OpenFOAM-v1606+.tgz
wget "http://downloads.sourceforge.net/openfoamplus/files/ThirdParty-v1606%2B.tgz?use_mirror=mesh" -O ThirdParty-v1606+.tgz
tar -xzf OpenFOAM-v1606+.tgz
tar -xzf ThirdParty-v1606+.tgz
rm OpenFOAM-v1606+.tgz
rm ThirdParty-v1606+.tgz
cp -r $FOAM_INST_DIR/OpenFOAM-v1606+/wmake/rules/linux64Icc $FOAM_INST_DIR/OpenFOAM-v1606+/wmake/rules/linux64IccHani
sed -i s/fPIC/"fPIC -Nmpi -Nmkl -xavx"/g $FOAM_INST_DIR/OpenFOAM-v1606+/wmake/rules/linux64IccHani/c++
Used the currently recommended version at the cluster for impi, intel compiler, and mkl. Get the recommended version numbers by:
module load impi
module load intel
module load mkl
It was realized that mkl was needed by a compilation error message. I don't know what it is used for.
alias OF1606+='export FOAM_INST_DIR=/proj/hani/OF_inst/Triolith/OpenFOAM; module load impi/5.0.2.044 intel/15.0.1.133 mkl/11.2.1; export MPI_ROOT=$I_MPI_ROOT; . $FOAM_INST_DIR/OpenFOAM-v1606+/etc/bashrc WM_COMPILER=IccHani WM_MPLIB=INTELMPI'
OF1606+
foam
Remove two lines with InterfaceForce in src/lagrangian/intermediate/parcels/include/makeParcelForces.H and the final backslash on the preceding line. There was an error message about the templating. We are currently not interested in that code, so let's leave it for now.
./Allwmake >& log&
Make it available:
cd ..
chmod -R go+Xr OpenFOAM-v1606+ ThirdParty-v1606+
chgrp -R pg_hani OpenFOAM-v1606+ ThirdParty-v1606+
Test by submitting the damBreak case. First create a submit script:
#!/bin/bash
#
#SBATCH -J damBreak #Name of job
#SBATCH -t 00:30:00 #HH:MM:SS
#SBATCH -N 1
#SBATCH --exclusive
#
#Same as alias OF1606+:
export FOAM_INST_DIR=/proj/hani/OF_inst/Triolith/OpenFOAM
module load impi/5.0.2.044 intel/15.0.1.133 mkl/11.2.1
export MPI_ROOT=$I_MPI_ROOT
. $FOAM_INST_DIR/OpenFOAM-v1606+/etc/bashrc WM_COMPILER=IccHani WM_MPLIB=INTELMPI
#Prepare
cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak/damBreak .
blockMesh -case damBreak
setFields -case damBreak
decomposePar -case damBreak
#--nranks is used when less than the number of available cores is used
mpprun --nranks=4 `which interFoam` -parallel -case damBreak >& log
# Script ends here
Submit by:
sbatch submitDambreak
Check if it is running (change x_hakni to your user name):
squeue -u x_hakni
This is work in progress. I have come to the point where you see a marker text.
For next time, first have a look at the following link, which has most of the information in a different way: [1]
cd /chalmers/sw/unsup64/OpenFOAM
git clone git://github.com/OpenFOAM/OpenFOAM-4.x.git
git clone git://github.com/OpenFOAM/ThirdParty-4.x.git
cp OpenFOAM-4.x/etc/bashrc OpenFOAM-4.x/etc/bashrcHani
cp OpenFOAM-4.x/etc/config.sh/settings OpenFOAM-4.x/etc/config.sh/settingsHani
sed -i s/settings/settingsHani/g OpenFOAM-4.x/etc/bashrcHani
Check available versions:
gcc --version #Gives 4.4.7. Required: 4.7+. NOT OK! - using ThirdParty Gcc
flex --version #Gives 2.5.35. Seems to have worked ok!
Check out the makeGcc?? files and Gcc in settings.sh to decide which gcc version to use - in this case we need to upgrade through ThirdParty (Gcc version can also be checked in etc/config.sh/compiler)
sed -i s/"WM_COMPILER_TYPE=system"/"WM_COMPILER_TYPE=ThirdParty"/g OpenFOAM-4.x/etc/bashrcHani
sed -i s/"bashrc"/"bashrcHani"/g OpenFOAM-4.x/etc/config.sh/aliases
The following line destroyed the original file when installing 2.4.x (never happened before or after). Make a back-up copy first!
echo "alias OF4x='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. \$FOAM_INST_DIR/OpenFOAM-4.x/etc/bashrcHani'" >> addToBashrc
Ignore the warning after the following line is executed (we use system Gcc for compilation of Gcc etc.):
export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. $FOAM_INST_DIR/OpenFOAM-4.x/etc/bashrcHani
See ThirdParty-4.x/makeGcc for which versions of dependent packages you need.
cd ThirdParty-4.x
wget http://ftp.gnu.org/gnu/gmp/gmp-5.1.2.tar.bz2
bunzip2 gmp-5.1.2.tar.bz2
tar xf gmp-5.1.2.tar
rm gmp-5.1.2.tar
wget http://www.mpfr.org/mpfr-3.1.2/mpfr-3.1.2.tar.gz
tar xzf mpfr-3.1.2.tar.gz
rm mpfr-3.1.2.tar.gz
wget http://www.multiprecision.org/mpc/download/mpc-1.0.1.tar.gz
tar xzf mpc-1.0.1.tar.gz
rm mpc-1.0.1.tar.gz
wget https://ftp.gnu.org/gnu/gcc/gcc-4.8.5/gcc-4.8.5.tar.gz
tar xzf gcc-4.8.5.tar.gz
rm gcc-4.8.5.tar.gz
./makeGcc gcc-4.8.5 >& log_makeGcc
rm log_makeGcc #After looking at it, making sure that it went through
Check that gcc is installed:
which gcc
Returns /chalmers/sw/unsup64/OpenFOAM/ThirdParty-3.0.x/platforms/linux64/gcc-4.8.4/bin/gcc
(For the following, see http://openfoamwiki.net/index.php/Installation/Linux/OpenFOAM-4.x/CentOS_SL_RHEL):
wget http://www.cmake.org/files/v3.2/cmake-3.2.1.tar.gz
wget https://github.com/CGAL/cgal/releases/download/releases%2FCGAL-4.8/CGAL-4.8.tar.xz
wget http://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_55_0.tar.bz2
tar xzf cmake-3.2.1.tar.gz
tar -xJf CGAL-4.8.tar.xz
tar -xjf boost_1_55_0.tar.bz2
rm cmake-3.2.1.tar.gz
rm CGAL-4.8.tar.xz
rm boost_1_55_0.tar.bz2
sed -i s/"config.sh\/CGAL"/"config.sh\/CGALHani"/g ../OpenFOAM-4.x/etc/bashrcHani
sed -i s/"config.sh\/CGAL"/"config.sh\/CGALHani"/g makeCGAL
cp ../OpenFOAM-4.x/etc/config.sh/CGAL ../OpenFOAM-4.x/etc/config.sh/CGALHani
sed -i -e 's/\(boost_version=\)boost-system/\1boost_1_55_0/' ../OpenFOAM-4.x/etc/config.sh/CGALHani
sed -i -e 's/\(cgal_version=\)cgal-system/\1CGAL-4.8/' ../OpenFOAM-4.x/etc/config.sh/CGALHani
./makeCmake >& log_makeCmake
rm log_makeCmake
wmRefresh
wget "https://raw.github.com/wyldckat/ThirdParty-2.0.x/binutils/makeBinutils"
wget "https://raw.github.com/wyldckat/ThirdParty-2.0.x/binutils/getBinutils"
chmod +x get* make*
./getBinutils
./makeBinutils gcc-4.8.5 > log.makeBinutils 2>&1
rm log.makeBinutils
#CGAL compilation did not work the first attempt, but after opening a new terminal window and sourcing it worked:
./makeCGAL boost_1_55_0 gmp-5.1.2 mpfr-3.1.2 >& log_makeCGAL #Makes sure to use the ones we installed
rm log_makeCGAL
wmRefresh
ParaView 5.0.1 needs at least Qt 4.7. Therefore, we'll need to do a custom build of Qt 4.8.6. This failed the first attempt. Second attemt worked, after opening a new terminal window, sourcing, and compiling CGAL.
While compiling Qt it is also possible to compile OpenFOAM (Allwmake below)
cd $WM_THIRD_PARTY_DIR
wget https://github.com/wyldckat/scripts4OpenFOAM3rdParty/raw/master/getQt
wget https://github.com/OpenFOAM/ThirdParty-2.4.x/raw/master/makeQt
wget -P etc/tools/ https://github.com/OpenFOAM/ThirdParty-2.4.x/raw/master/etc/tools/QtFunctions
chmod +x getQt makeQt
sed -i -e 's=4\.6=4.8=' -e 's=4\.8\.4=4.8.6=' -e 's=/\$major/\$tarFile=/$major/$version/$tarFile=' getQt
./getQt
./makeQt qt-4.8.6 > log.makeQt 2>&1
rm log.makeQt #After looking at it, making sure that it went through
Had a problem with MPI when compiling OF (try it first by compiling OF), so I did this to switch to an older mpi version that will be compiled as ThirdParty:
foam
cp -r ../ThirdParty-2.3.x/openmpi-1.6.5 ../ThirdParty-4.x
sed -i s/"openmpi-1.10.2"/"openmpi-1.6.5"/g etc/config.sh/mpi
sed -i s/SYSTEMOPENMPI/OPENMPI/g etc/bashrcHani
Update the shell environment
wmRefresh
Check that the correct gcc is used (the one just installed):
which gcc
foam
./Allwmake -j 4 >& log&
./Allwmake -j 4 >& log&
rm log #Had several errors first time. Second time CGAL was compiled, and only error with conformalVoronioMesh (foamymesh) which I do not care about.
I am here.
Compile ParaView with python and mpi (this requires mpi to be compiled first)
./makeParaView -qmake $WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/qt-4.8.6/bin/qmake -mpi -python > log.makePV 2>&1
wmRefresh
Compile ParaFoam reader:
cd $FOAM_UTILITIES/postProcessing/graphics/PVReaders
./Allwclean
./Allwmake
Make the installation available
foam
cd ..
chmod -R go+Xr OpenFOAM-4.x
chmod -R go+Xr ThirdParty-4.x
This is work in progress
cd /chalmers/sw/unsup64/OpenFOAM
git clone git://git.code.sf.net/p/foam-extend/foam-extend-3.2 foam-extend-4.0
cd foam-extend-4.0
git checkout nextRelease
Do required modifications to the installation procedure, in this case point at the Qt installation of FOAM-extend-3.1, since the system installed is below the required Qt version 4.7.0, by ParaView-4.3.1 (check etc/prefs.sh-EXAMPLES):
echo "export QT_DIR=/chalmers/sw/unsup64/OpenFOAM/foam-extend-3.1/ThirdParty/packages/qt-everywhere-opensource-src-4.8.5/platforms/linux64GccDPOpt" > etc/prefs.sh
echo "export QT_BIN_DIR=\$QT_DIR/bin" >> etc/prefs.sh
alias OFextend40='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. $FOAM_INST_DIR/foam-extend-4.0/etc/bashrc'
OFextend40
foam
Compile all:
./Allwmake.firstInstall >& log&
libccmio failed compiling since the tarball was not found. I changed in AllMake.stage3 to ftp://www.daba.lv/pub/TIS/bibliotekas/dazadas/libccmio-2.6.1.tar.gz and recompiled successfully.
Make it available:
chmod -R go+Xr .
Updated 161123
Made a tar file of the existing installation, as a backup
Checked modifications done by me on original files, using 'git status'
Moved ThirdParty/AllMake.stage3 to a backup file, since it was modified
Checked difference between my installation and remote:
git fetch origin
git diff HEAD...origin/nextRelease
Downloaded modifications:
git pull
Compiled:
./Allwmake (not running installations of ThridParty again)
Made it available:
chmod -R go+Xr .
Compile Debug version, without recompiling ThirdParty:
cd /chalmers/sw/unsup64/OpenFOAM
echo "alias OFextend40Debug='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. \$FOAM_INST_DIR/foam-extend-4.0/etc/bashrc WM_COMPILE_OPTION=Debug'" >> addToBashrc
Open new terminal window
OFextend40Debug
cd $WM_THIRD_PARTY_DIR/packages/cmake-3.2.2/platforms
ln -s linux64GccDPOpt linux64GccDPDebug
cd $WM_THIRD_PARTY_DIR/packages/hwloc-1.10.1/platforms
ln -s linux64GccDPOpt linux64GccDPDebug
cd $WM_THIRD_PARTY_DIR/packages/libccmio-2.6.1/platforms
ln -s linux64GccDPOpt linux64GccDPDebug
cd $WM_THIRD_PARTY_DIR/packages/mesquite-2.1.2/platforms
ln -s linux64GccDPOpt linux64GccDPDebug
cd $WM_THIRD_PARTY_DIR/packages/metis-5.1.0/platforms
ln -s linux64GccDPOpt linux64GccDPDebug
cd $WM_THIRD_PARTY_DIR/packages/openmpi-1.6.5/platforms
ln -s linux64GccDPOpt linux64GccDPDebug
cd $WM_THIRD_PARTY_DIR/packages/ParaView-4.3.1/platforms
ln -s linux64GccDPOpt linux64GccDPDebug
cd $WM_THIRD_PARTY_DIR/packages/parmetis-4.0.3/platforms
ln -s linux64GccDPOpt linux64GccDPDebug
cd $WM_THIRD_PARTY_DIR/packages/ParMGridGen-1.0/platforms
ln -s linux64GccDPOpt linux64GccDPDebug
cd $WM_THIRD_PARTY_DIR/packages/scotch-6.0.4/platforms
ln -s linux64GccDPOpt linux64GccDPDebug
Open a new terminal:
OFextend40Debug
foam
./Allwmake >& log&
Made it available:
chmod -R go+Xr .
cd /chalmers/sw/unsup64/OpenFOAM
git clone git://github.com/OpenFOAM/OpenFOAM-3.0.x.git
#Try this next time (but then additional tar files need to be downloaded, see wiki link below): git clone https://github.com/OpenFOAM/ThirdParty-3.0.x.git
Maybe necessary to rename the downloaded file to ThirdParty-3.0.0.tgz
tar xzf ThirdParty-3.0.0.tgz
rm ThirdParty-3.0.0.tgz
mv ThirdParty-3.0.0 ThirdParty-3.0.x
cp OpenFOAM-3.0.x/etc/bashrc OpenFOAM-3.0.x/etc/bashrcHani
cp OpenFOAM-3.0.x/etc/config/settings.sh OpenFOAM-3.0.x/etc/config/settingsHani.sh
sed -i s/settings.sh/settingsHani.sh/g OpenFOAM-3.0.x/etc/bashrcHani
Check available versions:
gcc --version #Gives 4.4.7. Required: 4.7+. NOT OK! - using ThirdParty Gcc
flex --version #Gives 2.5.35. Seems to have worked ok!
Check out the makeGcc?? files and Gcc in settings.sh to decide which gcc version to use - in this case we need to upgrade through ThirdParty
sed -i s/"foamCompiler=system"/"foamCompiler=ThirdParty"/g OpenFOAM-3.0.x/etc/bashrcHani
sed -i s/". \$WM\_PROJECT\_DIR\/etc\/bashrc"/". \$WM\_PROJECT\_DIR\/etc\/bashrcHani"/g OpenFOAM-3.0.x/etc/config/aliases.sh
The following line destroyed the original file when installing 2.4.x (never happened before or after). Make a back-up copy first!
echo "alias OF30x='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. \$FOAM_INST_DIR/OpenFOAM-3.0.x/etc/bashrcHani'" >> addToBashrc
Ignore the warning after the following line is executed (we use system Gcc for compilation of Gcc etc.):
export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. $FOAM_INST_DIR/OpenFOAM-3.0.x/etc/bashrcHani WM_NCOMPPROCS=4
See ThirdParty-3.0.x/makeGcc for which versions of dependent packages you need.
cd ThirdParty-3.0.x
wget http://ftp.gnu.org/gnu/gmp/gmp-5.1.2.tar.bz2
bunzip2 gmp-5.1.2.tar.bz2
tar xf gmp-5.1.2.tar
rm gmp-5.1.2.tar
wget http://www.mpfr.org/mpfr-3.1.2/mpfr-3.1.2.tar.gz
tar xzf mpfr-3.1.2.tar.gz
rm mpfr-3.1.2.tar.gz
wget http://www.multiprecision.org/mpc/download/mpc-1.0.1.tar.gz
tar xzf mpc-1.0.1.tar.gz
rm mpc-1.0.1.tar.gz
wget https://ftp.gnu.org/gnu/gcc/gcc-4.8.4/gcc-4.8.4.tar.gz
tar xzf gcc-4.8.4.tar.gz
rm gcc-4.8.4.tar.gz
./makeGcc gcc-4.8.4 >& log_makeGcc
rm log_makeGcc #After looking at it, making sure that it went through
Check that gcc is installed:
which gcc
Returns /chalmers/sw/unsup64/OpenFOAM/ThirdParty-3.0.x/platforms/linux64/gcc-4.8.4/bin/gcc
(For the following, see http://openfoamwiki.net/index.php/Installation/Linux/OpenFOAM-3.0.x/CentOS_SL_RHEL#CentOS_6.7):
(I complied CGAL before cmake contrary to the description in the wiki)
wget http://www.cmake.org/files/v3.2/cmake-3.2.1.tar.gz
tar xzf cmake-3.2.1.tar.gz
rm cmake-3.2.1.tar.gz
wget http://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_55_0.tar.bz2
tar -xjf boost_1_55_0.tar.bz2
rm boost_1_55_0.tar.bz2
sed -i s/"CGAL.sh"/"CGALHani.sh"/g ../OpenFOAM-3.0.x/etc/bashrcHani
cp ../OpenFOAM-2.4.x/etc/config/CGAL.sh ../OpenFOAM-3.0.x/etc/config/CGALHani.sh
sed -i -e 's=boost-system=boost_1_55_0=' ../OpenFOAM-3.0.x/etc/config/CGALHani.sh
./makeCGAL boost_1_55_0 gmp-5.1.2 mpfr-3.1.2 >& log_makeCGAL #Makes sure that also boost, gmp and mpfr are compiled
rm log_makeCGAL
./makeCmake >& log_makeCmake
rm log_makeCmake
ParaView 4.4.0 needs at least Qt 4.7. Therefore, we'll need to do a custom build of Qt 4.8.6
wget https://github.com/wyldckat/scripts4OpenFOAM3rdParty/raw/master/getQt
chmod +x getQt
sed -i -e 's=4\.6=4.8=' -e 's=4\.8\.4=4.8.6=' -e 's=/\$major/\$tarFile=/$major/$version/$tarFile=' getQt
./getQt
#makeQt missing in download. Get it from old installation:
cp ../ThirdParty-2.4.x/makeQt .
cp ../ThirdParty-2.4.x/etc/tools/QtFunctions etc/tools/
./makeQt qt-4.8.6 >& log_makeQt
#YOU HAVE TO TYPE "yes" TO ACCEPT THE LGPL LICENCE
rm log_makeQt #After looking at it, making sure that it went through
Had a problem with MPI when compiling OF (try it first by compiling OF), so I did this to switch to an older mpi version that will be compiled as ThirdParty:
foam
cp -r ../ThirdParty-2.3.x/openmpi-1.6.5 ../ThirdParty-3.0.x
sed -i s/"openmpi-1.10.0"/"openmpi-1.6.5"/g etc/config/settingsHani.sh
sed -i s/SYSTEMOPENMPI/OPENMPI/g etc/bashrcHani
Update the shell environment
wmSET $FOAM_SETTINGS
Check that the correct gcc is used (the one just installed):
which gcc
foam
./Allwmake -j 4 >& log&
./Allwmake -j 4 >& log&
rm log
Compile ParaView with python and mpi (this requires mpi to be compiled first)
#We need to apply a fix for a problem with reading multiple solids in an STL
wget https://raw.githubusercontent.com/Kitware/VTK/40937e934308e5009e80769dc0c451ee4f157749/IO/Geometry/vtkSTLReader.cxx -O ParaView-4.4.0/VTK/IO/Geometry/vtkSTLReader.cxx
#this will take a while... somewhere between 30 minutes to 2 hours or more
./makeParaView4 -qmake $WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/qt-4.8.6/bin/qmake -mpi -python >& log_makeParaView4
Compile ParaFoam reader:
cd $FOAM_UTILITIES/postProcessing/graphics/PV4Readers
wmSET $FOAM_SETTINGS
./Allwclean
./Allwmake
Make the installation available
foam
cd ..
chmod -R go+Xr OpenFOAM-3.0.x
chmod -R go+Xr ThirdParty-3.0.x
This was done after completing the OpenFOAM-2.4.x installation (see below).
See e.g. http://www.cfd-online.com/Forums/openfoam-meshing-utilities/160757-cfmesh-v1-1-1-a.html
OF24x
foam
cd ../ThirdParty-2.4.x/
mv cfMesh-v1.1.1.tgz\?r\=http\:%2F%2Fsourceforge.net%2Fprojects%2Fcfmesh%2F cfMesh-v1.1.1.tgz
tar xzf cfMesh-v1.1.1.tgz
rm cfMesh-v1.1.1.tgz
cd cfMesh-v1.1.1
Prepare for site installation:
sed -i s/FOAM_USER_LIBBIN/FOAM_LIBBIN/g meshLibrary/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g executables/cartesian2DMesh/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g executables/cartesianMesh/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g executables/pMesh/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g executables/tetMesh/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/checkSurfaceMesh/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/copySurfaceParts/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/extrudeEdgesInto2DSurface/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/FLMAToSurface/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/FMSToSurface/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/FMSToVTK/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/generateBoundaryLayers/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/importSurfaceAsSubset/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/improveMeshQuality/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/improveSymmetryPlanes/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/mergeSurfacePatches/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/meshToFPMA/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/patchesToSubsets/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/preparePar/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/removeSurfaceFacets/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/scaleMesh/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/scaleSurfaceMesh/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/subsetToPatch/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/surfaceFeatureEdges/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/surfaceGenerateBoundingBox/Make/files
sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/surfaceToFMS/Make/files
Compile:
./Allwmake
Change rights:
cd ..
chmod -R go+Xr cfMesh-v1.1.1
foam
chmod -R go+Xr platforms
After the compilation with Intel (below) I did the following:
Set the alias in the bashrc file:
alias OFextend32Gcc="module load gcc/4.9/4.9.2 openmpi/1.10.0; export FOAM_INST_DIR=/c3se/NOBACKUP/users/hani/hebbe/foam;. \$FOAM_INST_DIR/foam-extend-3.2/etc/bashrc WM_MPLIB=SYSTEMOPENMPI PARAVIEW_SYSTEM=1 FOAM_VERBOSE=1; unset WM_THIRD_PARTY_USE_OPENMPI_165; unset WM_THIRD_PARTY_USE_PARAVIEW_431"
Compile, in a new terminal window:
OFextend32Gcc
foam
./Allwmake.firstInstall >& log_Allwmake.firstInstall&
chgrp -R c3-fluiddyn /c3se/NOBACKUP/users/hani/hebbe
chmod -R g-w+Xr /c3se/NOBACKUP/users/hani/hebbe
chmod -R o-wrx /c3se/NOBACKUP/users/hani/hebbe
Exchange the "module" line in the submit script with the corresponding parts of the above alias.
mkdir -p /c3se/NOBACKUP/users/hani/hebbe/foam
cd /c3se/NOBACKUP/users/hani/hebbe/foam
git clone git://git.code.sf.net/p/foam-extend/foam-extend-3.2 foam-extend-3.2
cd foam-extend-3.2
git checkout nextRelease
echo 'alias OFextend32="module load intel-compilers/15.0/090 intel-mpi/5.0.3.048; export FOAM_INST_DIR=/c3se/NOBACKUP/users/hani/hebbe/foam;. \$FOAM_INST_DIR/foam-extend-3.2/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI MPI_ROOT=/c3se/apps/Common/intel/impi/5.0.3.048 PARAVIEW_SYSTEM=1 FOAM_VERBOSE=1; unset WM_THIRD_PARTY_USE_OPENMPI_165; unset WM_THIRD_PARTY_USE_PARAVIEW_431; export OPENMPI_DIR=/c3se/apps/Common/intel/impi/5.0.3.048/intel64"' >> ~/.bashrc
(OPENMPI_DIR is needed for parMetisDecomp, to find the lib directory of INTELMPI - ticket filed)
In a new terminal window:
OFextend32
foam
./Allwmake.firstInstall >& log_Allwmake.firstInstall&
chgrp -R c3-fluiddyn /c3se/NOBACKUP/users/hani/hebbe
chmod -R g-w+Xr /c3se/NOBACKUP/users/hani/hebbe
chmod -R o-wrx /c3se/NOBACKUP/users/hani/hebbe
Example of a submit script for parallel simulation (submit by "sbatch scriptName -A PROJ_NUMBER"):
#!/usr/bin/env bash
#SBATCH -N 1 -n 4 #Will use only 4 cores on 1 node (there are 20 cores per node on Hebbe)
#SBATCH -p hebbe
#SBATCH -t 00:01:00
#---------------- Uncomment ONE module line -------------------------
module load intel-compilers/15.0/090 intel-mpi/5.0.3.048; export FOAM_INST_DIR=/c3se/NOBACKUP/users/hani/hebbe/foam;. $FOAM_INST_DIR/foam-extend-3.2/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI MPI_ROOT=/c3se/apps/Common/intel/impi/5.0.3.048 PARAVIEW_SYSTEM=1 FOAM_VERBOSE=1; unset WM_THIRD_PARTY_USE_OPENMPI_165; unset WM_THIRD_PARTY_USE_PARAVIEW_431; export OPENMPI_DIR=/c3se/apps/Common/intel/impi/5.0.3.048/intel64
#------------ Copy and run the damBreak tutorial --------------------
cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak .
cd damBreak
blockMesh >& log_blockMesh #Generates the grid
setFields >& log_setFields #Sets the initial fields
decomposePar >& log_decomposePar #Decomposes for parallel simulation
mpirun interFoam -parallel >& log #Runs interFoam in parallel on 4 cores (flags -N 1 -n 4 above)
reconstructPar >& log_reconstructPar #Collects the decomposed results
rm -rf processor* #Deletes the decomposed files
Example of submitscript for sequential simulation:
#!/usr/bin/env bash
#SBATCH -N 1 -n 1
#SBATCH -p hebbe
#SBATCH -t 00:01:00
#---------------- Uncomment ONE module line -------------------------
module load intel-compilers/15.0/090 intel-mpi/5.0.3.048; export FOAM_INST_DIR=/c3se/NOBACKUP/users/hani/hebbe/foam;. $FOAM_INST_DIR/foam-extend-3.2/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI MPI_ROOT=/c3se/apps/Common/intel/impi/5.0.3.048 PARAVIEW_SYSTEM=1 FOAM_VERBOSE=1; unset WM_THIRD_PARTY_USE_OPENMPI_165; unset WM_THIRD_PARTY_USE_PARAVIEW_431; export OPENMPI_DIR=/c3se/apps/Common/intel/impi/5.0.3.048/intel64
#------------ Copy and run the damBreak tutorial --------------------
cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak .
cd damBreak
blockMesh >& log_blockMesh #Generates the grid
setFields >& log_setFields #Sets the initial fields
interFoam >& log #Runs interFoam in sequential mode
This should probably also work, to compile with system open-mpi, but I didn't verify it completely:
echo 'alias OFextend32="module load intel-compilers/15.0/090 openmpi/1.8.3-icc; export FOAM_INST_DIR=/c3se/NOBACKUP/users/hani/hebbe/foam;. \$FOAM_INST_DIR/foam-extend-3.2/etc/bashrc WM_COMPILER=Icc WM_MPLIB=SYSTEMOPENMPI OPENMPI_DIR=/c3se/apps/Hebbe/openmpi/1.8.3 OPENMPI_BIN_DIR=/c3se/apps/Hebbe/openmpi/1.8.3/bin PARAVIEW_SYSTEM=1 FOAM_VERBOSE=1; unset WM_THIRD_PARTY_USE_OPENMPI_165; unset WM_THIRD_PARTY_USE_PARAVIEW_431"' >> ~/.bashrc
Paraview is already available in two versions. You see them by typing:
module load extras
module avail
Then either:
module load paraview/3.14
or:
module load paraview/4.4.0
and then:
paraFoam -nativeReader
cd /chalmers/sw/unsup64/OpenFOAM
Note: downloaded after official release of 3.2, but still as branch in 3.1 (can also be done as git clone --branch v3.2 http://git.code.sf.net/p/openfoam-extend/foam-extend-3.1 foam-extend-3.2)
git clone http://git.code.sf.net/p/openfoam-extend/foam-extend-3.1 foam-extend-3.2
cd foam-extend-3.2
git checkout v3.2
alias OFextend32='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. $FOAM_INST_DIR/foam-extend-3.2/etc/bashrc'
OFextend32
foam
Do required modifications to the installation procedure, in this case point at the Qt installation of FOAM-extend-3.1, since the system installed is below the required Qt version 4.7.0, by ParaView-4.3.1 (check etc/prefs.sh-EXAMPLES):
echo "export QT_DIR=/chalmers/sw/unsup64/OpenFOAM/foam-extend-3.1/ThirdParty/packages/qt-everywhere-opensource-src-4.8.5/platforms/linux64GccDPOpt" > etc/prefs.sh
echo "export QT_BIN_DIR=\$QT_DIR/bin" >> etc/prefs.sh
Compile all:
./Allwmake.firstInstall >& log&
Make it available:
chmod -R go+Xr .
paraFoam did not work remotely, but I will test locally later.
Old notes, to check:
ParaView probably fails compiling. If so, do:
export WM_NCOMPPROCS=1
./Allwmake.firstInstall >& log&
rm log
chmod -R go+Xr .
git clone git://github.com/OpenFOAM/OpenFOAM-2.4.x.git
Maybe necessary to rename the downloaded file to ThirdParty-2.4.0.tgz
tar xzf ThirdParty-2.4.0.tgz
rm ThirdParty-2.4.0.tgz
mv ThirdParty-2.4.0 ThirdParty-2.4.x
cp OpenFOAM-2.4.x/etc/bashrc OpenFOAM-2.4.x/etc/bashrcHani
cp OpenFOAM-2.4.x/etc/config/settings.sh OpenFOAM-2.4.x/etc/config/settingsHani.sh
sed -i s/settings.sh/settingsHani.sh/g OpenFOAM-2.4.x/etc/bashrcHani
Check available versions:
gcc --version #Gives 4.4.7. Required: 4.7+. NOT OK! - using ThirdParty Gcc
flex --version #Gives 2.5.35. Seems to have worked ok!
Check out the makeGcc?? files and Gcc in settings.sh to decide which gcc version to use - in this case we need to upgrade through ThirdParty
sed -i s/"foamCompiler=system"/"foamCompiler=ThirdParty"/g OpenFOAM-2.4.x/etc/bashrcHani
sed -i s/". \$WM\_PROJECT\_DIR\/etc\/bashrc"/". \$WM\_PROJECT\_DIR\/etc\/bashrcHani"/g OpenFOAM-2.4.x/etc/config/aliases.sh
The following line destroyed the original file for some strange reason (never happened before). Make a back-up copy first!
echo "alias OF24x='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. \$FOAM_INST_DIR/OpenFOAM-2.4.x/etc/bashrcHani'" >> addToBashrc
Ignore the warning after the following line is executed (we use system Gcc for compilation of Gcc etc.):
export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. $FOAM_INST_DIR/OpenFOAM-2.4.x/etc/bashrcHani WM_NCOMPPROCS=4
See ThirdParty-2.4.x/makeGcc for which versions of depandent packages you need.
cd ThirdParty-2.4.x
wget http://ftp.gnu.org/gnu/gmp/gmp-5.1.2.tar.bz2
bunzip2 gmp-5.1.2.tar.bz2
tar xf gmp-5.1.2.tar
rm gmp-5.1.2.tar
wget http://www.mpfr.org/mpfr-3.1.2/mpfr-3.1.2.tar.gz
tar xzf mpfr-3.1.2.tar.gz
rm mpfr-3.1.2.tar.gz
wget http://www.multiprecision.org/mpc/download/mpc-1.0.1.tar.gz
tar xzf mpc-1.0.1.tar.gz
rm mpc-1.0.1.tar.gz
wget https://ftp.gnu.org/gnu/gcc/gcc-4.8.2/gcc-4.8.2.tar.gz
tar xzf gcc-4.8.2.tar.gz
rm gcc-4.8.2.tar.gz
./makeGcc gcc-4.8.2 >& log_makeGcc
rm log_makeGcc #After looking at it, making sure that it went through
Check that gcc is installed:
which gcc
Returns /chalmers/sw/unsup64/OpenFOAM/ThirdParty-2.4.x/platforms/linux64/gcc-4.8.2/bin/gcc (For the following, see http://openfoamwiki.net/index.php/Installation/Linux/OpenFOAM-2.4.0/CentOS_SL_RHEL#CentOS_6.5):
sed -i s/"CGAL.sh"/"CGALHani.sh"/g ../OpenFOAM-2.4.x/etc/bashrcHani
cp ../OpenFOAM-2.4.x/etc/config/CGAL.sh ../OpenFOAM-2.4.x/etc/config/CGALHani.sh
wget "https://raw.github.com/wyldckat/scripts4OpenFOAM3rdParty/master/getBoost"
chmod +x getBoost
./getBoost
sed -i -e 's=boost-system=boost_1_54_0=' ../OpenFOAM-2.4.x/etc/config/CGALHani.sh
./makeCGAL gmp-5.1.2 mpfr-3.1.2
ParaView 4.1.0 needs at least Qt 4.7. Therefore, we'll need to do a custom build of Qt 4.8.4
wget "https://raw.github.com/wyldckat/scripts4OpenFOAM3rdParty/master/getQt"
chmod +x getQt
sed -i -e 's=4\.6=4.8=' -e 's=4\.8\.4=4.8.6=' -e 's=/\$major/\$tarFile=/$major/$version/$tarFile=' getQt
./getQt
./makeQt qt-4.8.6 > log_makeQt 2>&1
rm log_makeQt #After looking at it, making sure that it went through
CentOS 6.5 comes with CMake 2.6.4, but ParaView 4.1.0 needs a one of the versions from the more recent CMake 2.8.x series and CGAL also needs CMake.
cd $WM_THIRD_PARTY_DIR
wget "https://raw.github.com/wyldckat/scripts4OpenFOAM3rdParty/master/getCmake"
sed -i -e 's=version}\.3=version}.12.1=' getCmake
chmod +x getCmake
./getCmake
./makeCmake
wmSET $FOAM_SETTINGS
Close the terminal window and open a new one, then type
OF24x
. $FOAM_INST_DIR/OpenFOAM-2.4.x/etc/bashrcHani WM_NCOMPPROCS=4
Check that the correct gcc is used (the one just installed):
which gcc
foam
./Allwmake >& log&
./Allwmake >& log&
SOME PROBLEMS RELATED TO MPI, SEE END OF SECTION!!!
rm log
ParaFoam:
cd $WM_THIRD_PARTY_DIR
./makeParaView4 -qmake $WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/qt-4.8.6/bin/qmake PARAVIEW_INSTALL_THIRD_PARTY_LIBRARIES:BOOL=ON
cd $FOAM_UTILITIES/postProcessing/graphics/PV4Readers
wmSET $FOAM_SETTINGS
./Allwclean
./Allwmake
foam
cd ..
chmod -R go+Xr ThirdParty-2.4.x OpenFOAM-2.4.x
Compile Debug version, without recompiling ThirdParty:
cd /chalmers/sw/unsup64/OpenFOAM
echo "alias OF24xDebug='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. \$FOAM_INST_DIR/OpenFOAM-2.4.x/etc/bashrcHani WM_COMPILE_OPTION=Debug'" >> addToBashrc
Open new terminal window
OF24xDebug
cd $WM_THIRD_PARTY_DIR/platforms
ln -s linux64GccDPOpt linux64GccDPDebug
Open a new terminal:
OF24xDebug
foam
./Allwmake >& log&
./Allwmake >& log&
cd $FOAM_UTILITIES/postProcessing/graphics/PV4Readers
#Skip wmSET since it will not set Debug, see alias wmSET
./Allwclean
./Allwmake
foam
cd ..
chmod -R go+Xr OpenFOAM-2.4.x;echo "done: chmod -R go+Xr OpenFOAM-2.4.x"; chmod -R go+Xr ThirdParty-2.4.x; echo "done: chmod -R go+Xr ThirdParty-2.4.x" >& log_chmod&
The problem with MPI was solved after the above using:
OF24x
foam
cp -r ../ThirdParty-2.3.x/openmpi-1.6.5 ../ThirdParty-2.4.x
sed -i s/"openmpi-1.8.5"/"openmpi-1.6.5"/g etc/config/settingsHani.sh
sed -i s/SYSTEMOPENMPI/OPENMPI/g etc/bashrcHani
New terminal window
OF24x
foam
./Allwmake >& log&
./Allwmake >& log&
New terminal window
OF24xDebug
foam
./Allwmake >& log&
./Allwmake >& log&
cd ..
chmod -R go+Xr OpenFOAM-2.4.x;echo "done: chmod -R go+Xr OpenFOAM-2.4.x"; chmod -R go+Xr ThirdParty-2.4.x; echo "done: chmod -R go+Xr ThirdParty-2.4.x" >& log_chmod&
I later also added cfMesh-1.1.1. See separate section above.
NOTE:
This installation did not complete zoltanRenumber since a fix for that is not possible without root access:
The Boost C++ library is installed automatically for GCAL under the name libboost_thread-mt.so, but it requires the library to be named libboost_thread.so. To work around this inconsistency, a soft link should be created as follows:
ln -s /usr/lib64/libboost_thread-mt.so /usr/lib64/libboost_thread.so
INSTALLATION NOTES START HERE:
git clone git://github.com/OpenFOAM/OpenFOAM-2.3.x.git
Mayby necessary to rename the downloaded file to ThirdParty-2.3.0.tgz
tar xzf ThirdParty-2.3.0.tgz
rm ThirdParty-2.3.0.tgz
mv ThirdParty-2.3.0 ThirdParty-2.3.x
cp OpenFOAM-2.3.x/etc/bashrc OpenFOAM-2.3.x/etc/bashrcHani
cp OpenFOAM-2.3.x/etc/config/settings.sh OpenFOAM-2.3.x/etc/config/settingsHani.sh
sed -i s/settings.sh/settingsHani.sh/g OpenFOAM-2.3.x/etc/bashrcHani
Check out the makeGcc?? files and Gcc in settings.sh to decide which gcc version to use
sed -i s/"foamCompiler=system"/"foamCompiler=ThirdParty"/g OpenFOAM-2.3.x/etc/bashrcHani
sed -i s/". \$WM\_PROJECT\_DIR\/etc\/bashrc"/". \$WM\_PROJECT\_DIR\/etc\/bashrcHani"/g OpenFOAM-2.3.x/etc/config/aliases.sh
echo "alias OF23x='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. \$FOAM_INST_DIR/OpenFOAM-2.3.x/etc/bashrcHani'" >> addToBashrc
Check versions:
gcc --version #Gives 4.4.7. Required: 4.7+. NOT OK! - using ThirdParty Gcc
flex --version #Gives 2.5.35. Seems to have worked ok!
export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. $FOAM_INST_DIR/OpenFOAM-2.3.x/etc/bashrcHani WM_NCOMPPROCS=4
cd ThirdParty-2.3.x
wget http://ftp.gnu.org/gnu/gmp/gmp-5.1.2.tar.bz2
bunzip2 gmp-5.1.2.tar.bz2
tar xf gmp-5.1.2.tar
rm gmp-5.1.2.tar
wget http://www.mpfr.org/mpfr-3.1.2/mpfr-3.1.2.tar.gz
tar xzf mpfr-3.1.2.tar.gz
rm mpfr-3.1.2.tar.gz
wget http://www.multiprecision.org/mpc/download/mpc-1.0.1.tar.gz
tar xzf mpc-1.0.1.tar.gz
rm mpc-1.0.1.tar.gz
wget https://ftp.gnu.org/gnu/gcc/gcc-4.8.2/gcc-4.8.2.tar.gz
tar xzf gcc-4.8.2.tar.gz
rm gcc-4.8.2.tar.gz
./makeGcc48
(see http://openfoamwiki.net/index.php/Installation/Linux/OpenFOAM-2.3.0/CentOS_SL_RHEL#CentOS_6.5):
sed -i s/"CGAL.sh"/"CGALHani.sh"/g ../OpenFOAM-2.3.x/etc/bashrcHani
cp OpenFOAM-2.3.x/etc/config/CGAL.sh ../OpenFOAM-2.3.x/etc/config/CGALHani.sh
wget "https://raw.github.com/wyldckat/scripts4OpenFOAM3rdParty/master/getBoost"
chmod +x getBoost
./getBoost
sed -i -e 's=boost-system=boost_1_54_0=' ../OpenFOAM-2.3.x/etc/config/CGALHani.sh
./makeCGAL gmp-5.1.2 mpfr-3.1.2
ParaView 4.1.0 needs at least Qt 4.7. Therefore, we'll need to do a custom build of Qt 4.8.4
wget "https://raw.github.com/wyldckat/scripts4OpenFOAM3rdParty/master/getQt"
chmod +x getQt
sed -i -e 's=4\.6=4.8=' -e 's=/\$major/\$tarFile=/$major/$version/$tarFile=' getQt
./getQt
./makeQt qt-4.8.4 >& log_makeQt & (NOTE: Have to say "y")
CentOS 6.5 comes with CMake 2.6.4, but ParaView 4.1.0 needs a one of the versions from the more recent CMake 2.8.x series and CGAL also needs CMake.
cd $WM_THIRD_PARTY_DIR
wget "https://raw.github.com/wyldckat/scripts4OpenFOAM3rdParty/master/getCmake"
sed -i -e 's=version}\.3=version}.12.1=' getCmake
chmod +x getCmake
./getCmake
./makeCmake
wmSET $FOAM_SETTINGS
. $FOAM_INST_DIR/OpenFOAM-2.3.x/etc/bashrcHani
Check that the correct gcc is used (the one just installed):
which gcc
foam
./Allwmake
./Allwmake
ParaFoam:
cd $WM_THIRD_PARTY_DIR
./makeParaView4 -qmake $WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/qt-4.8.4/bin/qmake PARAVIEW_INSTALL_THIRD_PARTY_LIBRARIES:BOOL=ON
cd $FOAM_UTILITIES/postProcessing/graphics/PV4Readers
wmSET $FOAM_SETTINGS
./Allwclean
./Allwmake
cd ..
chmod -R go+Xr ThirdParty-2.3.x OpenFOAM-2.3.x
Can't run paraFoam remotely:
/nfs4/sol.ita.chalmers.se/ufs/groups/manual03/unsup/unsup64/OpenFOAM/ThirdParty-2.3.x/platforms/linux64Gcc/ParaView-4.1.0/lib/paraview-4.1/paraview: symbol lookup error: /usr/lib64/libGL.so.1: undefined symbol: _XGetRequest
See also:
http://www.cfd-online.com/Forums/blogs/wyldckat/1203-paraview-3-12-0-superbuild-openfoam.html
Compile Debug version, without recompiling ThirdParty:
cd /chalmers/sw/unsup64/OpenFOAM
echo "alias OF23xDebug='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. \$FOAM_INST_DIR/OpenFOAM-2.3.x/etc/bashrcHani WM_COMPILE_OPTION=Debug'" >> addToBashrc
cd $WM_THIRD_PARTY_DIR/platforms
ln -s linux64GccDPOpt linux64GccDPDebug
Open a new terminal:
OF23xDebug
foam
./Allwmake
cd $FOAM_UTILITIES/postProcessing/graphics/PV3Readers
#Skip wmSET since it will not set Debug, see alias wmSET
./Allwclean
./Allwmake
foam
cd ..
chmod -R go+Xr OpenFOAM-2.3.x;echo "done: chmod -R go+Xr OpenFOAM-2.3.x"; chmod -R go+Xr ThirdParty-2.3.x; echo "done: chmod -R go+Xr ThirdParty-2.3.x" >& log_chmod&