Memo

I often spend lots of time on setting up the same software over and over again for different purposes. In order to save time, I decide to write down some useful procedures or hints here. Hope that they could be helpful to myself and others who visit here.

Suggestions to avoid ARPACK++ compilation errors

posted Jan 21, 2015, 7:24 AM by Teng-Yok Lee   [ updated Jan 21, 2015, 6:42 PM ]

I plan to put my patches to ARPACK++(http://www.ime.unicamp.br/~chico/arpack++/) to my Google Code repo at the end of Jan. 2015. Here I manually list the fixes for my applications. Note that my applications only use ardsmat.h and ardssym.h so there could be more errors, but I guess that the fixes for other parts should be similar.

arch.h

Comment arcomp.h (Otherwise, arcomple<float>/arcomple<double> will be declared inside a extern "C" closure, which is not allowed since C does not understand C++ template).

// #include "arcomp.h"

Also, replace the generic.h

#include <generic.h>

by the only needed macro name2:

// REF: http://www-d0.fnal.gov/KAI/doc/migrate/gnu_generic.h
#define name2(a,b) gEnErIc2(a,b)
#define gEnErIc2(a,b) a ## b

ardssym.h

Replace
DefineParameters(A.ncols(), nevp, &A, ARdsSymMatrix<FLOAT>::MultMv,
by
DefineParameters(A.ncols(), nevp, &A, &ARdsSymMatrix<FLOAT>::MultMv,

arerror.h

Replace
#include <iostream.h>

by
#include <iostream>

arpackf.h

Change the sentences at the end (Because arcomp.h is not included, '}' will be ignored and thus the entern closure is not completed).

}
#endif // ARCOMP_H

to
#endif // ARCOMP_H
} // extern "C" {


Build ARPACK on Windows for Visual Studio

posted Jan 19, 2015, 11:33 PM by Teng-Yok Lee   [ updated Jan 21, 2015, 7:27 AM ]

This is a revision of Jernej Barbic's steps to build ARPACK for Visual Studio. The original steps are in the URL below:

http://www-bcf.usc.edu/~jbarbic/arpack.html

I met issues when executing step 3 so I decide to put my modified procedure here. For 32-bit, you can try both gfortran or g77. For 64-bit, only gfortran can work.

My recommendation is to use the MinGW64 in CYGWIN64.

Use gfortran (Win32)

  1. Modify ARmake.inc in the source code root. I extracted the source code of ARPACK to D:\src\ARPACK. Also I cannot find f77 in the latest MinGW so I used gfortran instead. Thus the following variables should be changed accordingly:
    home = /d/src/ARPACK
    PLAT = win32
    FC      = gfortran
    FFLAGS    = -O
  2. Open a MSYS window and:
    cd /d/src/ARPACK.
  3. Compile the .f to .o:
    make lib
  4. Wrap the *.o to .dll. I need to add the library gfortran:
    dllwrap --export-all-symbols BLAS/*.o LAPACK/*.o SRC/*.o UTIL/*.o -lg2c -lgfortran --output-def arpack_win32.def -o arpack_win32.dll
  5. Open a Visual Studio Command Prompt and:
    cd d:\src\ARPACK
  6. Generate the library:
    lib.exe /machine:i386 /def:arpack_win32.def

Use g77 (Win32)

  1. Modify ARmake.inc in the source code root. I extracted the source code of ARPACK to D:\src\ARPACK. Also I cannot find f77 in the latest MinGW so I used gfortran instead. Thus the following variables should be changed accordingly:
    home = /d/src/ARPACK
    PLAT = win32
    FC      = g77
    FFLAGS    = -O
  2. Open a MSYS window and:
    cd /d/src/ARPACK.
  3. Compile the .f to .o:
    make lib
  4. Wrap the *.o to .dll.
    dllwrap --export-all-symbols BLAS/*.o LAPACK/*.o SRC/*.o UTIL/*.o -lg2c --output-def arpack_win32.def -o arpack_win32.dll
  5. Open a Visual Studio Command Prompt and:
    cd d:\src\ARPACK
  6. Generate the library:
    lib.exe /machine:i386 /def:arpack_win32.def

Use gfortran (Win 64) in MSYS

Before the procedure, install MinGW 64. In my case, it is installed to C:\Program Files (x86)\mingw-w64, and gfortran.exe is in C:\Program Files (x86)\mingw-w64\i686-4.9.2-posix-dwarf-rt_v3-rev1\mingw32\bin.
  1. Modify ARmake.inc in the source code root. I extracted the source code of ARPACK to D:\src\ARPACK. Also I cannot find f77 in the latest MinGW so I used gfortran instead. Thus the following variables should be changed accordingly:
    home = /d/src/ARPACK
    PLAT = x64
    FC = /c/Program\ Files\ \(x86\)/mingw-w64/i686-4.9.2-posix-dwarf-rt_v3-rev1/mingw32/bin/gfortran.exe
    FFLAGS    = -O
    RANLIB = /c/Program\ Files\ \(x86\)/mingw-w64/i686-4.9.2-posix-dwarf-rt_v3-rev1/mingw32/bin/ranlib.exe

  2. Open a MSYS window and:
    cd /d/src/ARPACK
  3. Extend PATH:
    export PATH=$PATH:/c/Program\ Files\ \(x86\)/mingw-w64/i686-4.9.2-posix-dwarf-rt_v3-rev1/mingw32/opt/bin
  4. Change UTIL/second. Remove the sentence: EXTERNAL           ETIME.
  5. Compile the .f to .o:
    make lib
  6. Wrap the *.o to .dll:
    /c/Program\ Files\ \(x86\)/mingw-w64/i686-4.9.2-posix-dwarf-rt_v3-rev1/mingw32/bin/dllwrap.exe --export-all-symbols BLAS/*.o LAPACK/*.o SRC/*.o UTIL/*.o -lgfortran --output-def arpack_x64.def -o arpack_x64.dll
  7. Open a Visual Studio (64-bit) Command Prompt and:
    cd d:\src\ARPACK
  8. Generate the library:
    lib.exe /machine:X64 /def:arpack_x64.def

Use gfortran (Win 64) in cygwin

Later I found later MinGW is also available in CYGWIN, which make the commands shorter.

Before the procedure, install mingw64-x86_64-gcc in CYGWIN 64.

  1. Modify ARmake.inc in the source code root. I extracted the source code of ARPACK to D:\src\ARPACK. Also I cannot find f77 in the latest MinGW so I used gfortran instead. Thus the following variables should be changed accordingly:
    home = /cygdrive/d/src/ARPACK
    PLAT = x64
    FC = /usr/bin/x86_64-w64-mingw32-gfortran.exe
    FFLAGS    = -O
    RANLIB = /usr/bin/x86_64-w64-mingw32-ranlib.exe

  2. Open a MSYS window and:
    cd /cygdrive/d/src/ARPACK
  3. Change UTIL/second. Remove the sentence: EXTERNAL           ETIME.
  4. Compile the .f to .o:
    make lib
  5. Wrap the *.o to .dll:
    /usr/bin/x86_64-w64-mingw32-dllwrap.exe --export-all-symbols BLAS/*.o LAPACK/*.o SRC/*.o UTIL/*.o -lm -lgfortran --output-def arpack_x64.def -o arpack_x64.dll
  6. Open a Visual Studio (64-bit) Command Prompt and:
    cd d:\src\ARPACK
  7. Generate the library:
    lib.exe /machine:X64 /def:arpack_x64.def

Link the lib

After building .lib and .dll, adding the directories of related dll. to your path. For CYGWIN, the path is

C:\cygwin64\usr\x86_64-w64-mingw32\sys-root\mingw\bin

For MinGW, it should be the path to the 64-bit.

If the .dll does not match to the linked library, it might show error The application was unable to start correctly (0xc000007b). I saw it when my system had both 32-bit and 64-bit versions of MinGW. After I uninstalled both and just kept CYGWIN 64, the problem was solved.

References

  1. External issue of etime: https://gcc.gnu.org/ml/fortran/2007-03/msg00305.html
  2. Use CYGWIN and MinGW64 to build Visual Studio libraries: https://github.com/arrayfire/arrayfire/wiki/CBLAS-for-Windows
  3. Cannot start my application: http://stackoverflow.com/questions/25124182/mingw-gcc-the-application-was-unable-to-start-correctly-0xc000007b

A dirty trick to install numpy via distutils without gcc 4.9

posted Dec 15, 2014, 12:17 AM by Teng-Yok Lee   [ updated Dec 15, 2014, 12:17 AM ]

After python 2.7.8, its distutils use gcc 4.9 as the C compiler, which introduces a new flag -fstack-protector-strong. Also, the AR executable is changed. However, on a system without gcc 4.9, this can fail. Consequently, numpy cannot be built.

While not all package management system has gcc 4.9 (e.g. old Ubuntu AMI from AWS), a quick trick is to modify the following file:

/usr/lib/python2.7/plat-x86_64-linux-gnu/_sysconfigdata_nd.py

First, remove -fstack-protector-strong in this file. Second, change the value of AR to 'ar'.


Export the link to pdf format of Google Doc

posted Nov 15, 2014, 4:12 PM by Teng-Yok Lee   [ updated Dec 15, 2014, 12:20 AM ]

As I use Google Doc to edit my CV, before I needed to download the file as pdf, and upload to my webpage. Nevertheless, the link to google doc is very simple: First, edit the file to get the URL. Ten change /edit in the url to /export?format=pdf.

NOTE: To make the links accessible to others, the doc link should be viewable to others.

REF: http://webapps.stackexchange.com/questions/8106/link-to-view-pdf-version-of-a-google-doc 

Disable GPU Acceleration on Firefox

posted Sep 14, 2014, 8:13 AM by Teng-Yok Lee   [ updated Dec 15, 2014, 12:19 AM ]

  1. Type about:config as the URL.
  2. Search for gfx.direct2d.disabled
  3. Double click to make it "true".

REF: https://support.mozilla.org/en-US/questions/922995

PyDev: My FAQ

posted Aug 15, 2014, 10:55 AM by Teng-Yok Lee   [ updated Aug 15, 2014, 1:57 PM ]

Q: What to do when receive the error messages "Failed to read server's response: Connection refused: connect"?
A: Increase the connection attempt.
REF: https://www.mail-archive.com/pydev-users@lists.sourceforge.net/msg03298.html

Generate histograms from bash commands

posted Jun 22, 2014, 8:01 PM by Teng-Yok Lee

REF: http://www.smallmeans.com/notes/shell-history/

For instance, to find the histogram of commands:
history|awk '{print $2}'|sort|uniq -c

Install mpi4py on Amazon EC2

posted May 29, 2014, 8:00 AM by Teng-Yok Lee   [ updated May 29, 2014, 8:36 AM ]

It sounds very simple

# REF: http://mpi4py.scipy.org/docs/usrman/install.html
$ sudo pip install mpi4py

Nevertheless, as my EC2 instance use openmpi-devel installed via yum, it cannot find mpi.h. Later I searched for the installation path and found it:

# REF: http://stackoverflow.com/questions/1766380/determining-the-path-that-a-yum-package-installed-to
$ rpm -ql openmpi-devel | grep mpi.h
/usr/include/openmpi-x86_64/mpi.h

Then how to run pip with the specified mpi.h?

NOTE: mpi4py will look for pyconfig.h and Python.h, which are only available in python2.7.

As a result, build from the source is needed. The following steps are my procedure:
$ wget http://mpi4py.googlecode.com/files/mpi4py-1.3.1.tar.gz
$ tar zxvf mpi4py-1.3.1.tar.gz
# Edit the mpi section in mpi4py-1.3.1/mpi.cfg as follows

## define_macros        =
## undef_macros         =
## include_dirs         = %(mpi_dir)s/include
include_dirs         = /usr/include/openmpi-x86_64:/usr/include/python2.7
## libraries            = mpi
libraries            = mpi
## library_dirs         = %(mpi_dir)s/lib
library_dirs         = /usr/lib64/openmpi/lib
## runtime_library_dirs = %(mpi_dir)s/lib
runtime_library_dirs = /usr/lib64/openmpi/lib
$ python2.7 setup.py build
$ sudo python2.7 setup.py install

# Edit ~/.bashrc by adding the following line:
# export PYTHONPATH=$PYTHONPATH:/usr/lib64/python2.7/site-packages/

Meanwhile, I also use mpirun+NFS to install mpi4py to all nodes.

$ cd /mnt/share/mpi4py-1.3.1
$ mpirun -hostfile ~/hostfile -n 2 -npernode 1 sudo python2.7 setup.py install

NOTE: To make this work, root need the permission to write to the NFS (here /mnt/share). Then in /etc/exports, the permission for each drive should be (rw,no_root_squash) (REF: http://superuser.com/questions/605464/unable-to-write-as-root-but-can-as-user)

My memo on python

posted May 17, 2014, 8:56 AM by Teng-Yok Lee   [ updated May 27, 2014, 1:22 AM ]

My tricks

Write a script that can be imported

I usually write python scripts for small task. Scripts can be executed quickly (without setting program variables), but they are hard to debug when scripts are calling scripts. In such a case, the ability to run scripts like a module will be better.

Here is my template to write a python script so it can be also called as a module. In each script, I just define a method main(), which define the main computation.

from optparse import OptionParser;

# Define the parser.
parser = OptionParser()
...

def main(arguments):
    (options, args) = parser.parse_args(arguments)
    ...

    return 0;

Then I will check whether this script is directly called by python. If yes, immediately call main():

# Entry point of the script.
import os;
import sys;
...
if( os.path.basename(sys.argv[0]) == os.path.basename(__file__) ):
    exit(main(sys.argv));

To call this script from another script (says my_script), now just called its main():

import my_script;
...

# Setup the argument.
arguments = [
...];

my_script.main(arguments);

Useful links about advanced python usage:

How to get the python.exe location programmatically?

import sys;
...
python_exec = sys.executable;

Usage: On Windows, I use WinPython + Eclipse for developing. If I want to call another python scripts, I need to pass the correct python interpreter. Otherwise, the default python might not have the packages I needed (such as numpy).

In Python, how do I get the path and name of the file that is currently executing?

Use the keyword __file__.


[FW] Text Coding in emacs

posted May 7, 2014, 12:40 AM by Teng-Yok Lee

REF: https://www.gnu.org/software/emacs/manual/html_node/emacs/Text-Coding.html

This will be extremely help when using a DOS-encoded files on Unix. In such a case, put coding as unix.

C-x <RET> f coding <RET>

1-10 of 98