Memo

I often spend lots of time on setting up the same software over and over again for different purposes. In order to save time, I decide to write down some useful procedures or hints here. Hope that they could be helpful to myself and others who visit here.

Use libSDF to read SDF format on Windows

posted Mar 20, 2015, 9:46 AM by Teng-Yok Lee   [ updated Mar 20, 2015, 9:47 AM ]

This is my memo to read the data for IEEE SciVis 2015 Contest. The file is in SDF format. There is a C/C++ library libSDF to open the files, but some instructions are unclear, especially for the porting on Windows. Also, I cannot find examples about its usage so I write one:

Prerequisites

To build libSDF for Visual Studio 2010, I use cygwin and mingw x64. The procedure to install them can be seen here:

http://www.recheliu.org/memo/suggestionstoavoidarpackcompilationerrors

Build libSDF for Windows x64 platform

  • Edit SDFfuncs.c: Remove the preprocessor USE_ALLOCA.
  • Edit utils.c: Change the function MPMU_Fopen() to open the file as binary (otherwise not all bytes can be read):
MPMYFile *MPMY_Fopen(const char *path, int mpmy_flags)
{
    MPMYFile *fp;
    int iomode = MPMY_SINGL;
    char mode[8] = {[0] = 'r'};

    Msgf("Fopen %s\n", path);
    if (mpmy_flags & MPMY_RDONLY) mode[0] = 'r'; /* MPMY_RDONLY is 0 since O_RDONLY is (stupidly) 0 */
    if (mpmy_flags & MPMY_WRONLY) mode[0] = 'w';
    if (mpmy_flags & MPMY_APPEND) mode[0] = 'a';
    // TEST-ADD-BEGIN
    switch(mode[0]) {
    case 'r':
        strcpy(mode, "r+b");
        break;
    case 'w':
        strcpy(mode, "w+b");
        break;
    case 'a':
        strcpy(mode, "a+b");
        break;
    }
    // TEST-ADD-END
    ...

  • Edit Makefile: Change CC from gcc to the one for mingw.
CC=/usr/bin/x86_64-w64-mingw32-gcc.exe
  • Use cygwin to build the library.
$ make libSDF.a
$ /usr/bin/x86_64-w64-mingw32-dllwrap.exe --export-all-symbols *.o -lm --output-def libSDF_x64.def -o libSDF_x64.dll

  • Use Visual Studio (64-bit) Command Prompt to build the .dll

lib.exe /machine:X64 /def:libSDF_x64.def

My quick example to read the array "x"

    #define LOG_VAR(x) cout<<x<<endl;

    char* szSdfFilepath = "F:/data/viscontest/scivis2015/ds14_scivis_0128_e4_dt04_0.0200";
    LOG_VAR(SDFissdf(szSdfFilepath));

    SDF *sdf = SDFopen(szSdfFilepath, "");
    SDFdebug(1); // Put it 0 to disable debug information.

    int64_t uNrOfRecs = SDFnrecs("x", sdf);
    LOG_VAR(uNrOfRecs);
    vector<float> vfData(uNrOfRecs);
    SDFrdvecs(sdf, "x", uNrOfRecs, vfData.data(), 0, NULL);
    SDFclose(sdf);

    for(size_t i = 0; i < uNrOfRecs; i++)
    {
        if( 0.0f != vfData[i] )
        {
            LOG_VAR(vfData[i]);
        }
    }


Suggestions to avoid ARPACK++ compilation errors

posted Jan 21, 2015, 7:24 AM by Teng-Yok Lee   [ updated Jan 21, 2015, 6:42 PM ]

I plan to put my patches to ARPACK++(http://www.ime.unicamp.br/~chico/arpack++/) to my Google Code repo at the end of Jan. 2015. Here I manually list the fixes for my applications. Note that my applications only use ardsmat.h and ardssym.h so there could be more errors, but I guess that the fixes for other parts should be similar.

arch.h

Comment arcomp.h (Otherwise, arcomple<float>/arcomple<double> will be declared inside a extern "C" closure, which is not allowed since C does not understand C++ template).

// #include "arcomp.h"

Also, replace the generic.h

#include <generic.h>

by the only needed macro name2:

// REF: http://www-d0.fnal.gov/KAI/doc/migrate/gnu_generic.h
#define name2(a,b) gEnErIc2(a,b)
#define gEnErIc2(a,b) a ## b

ardssym.h

Replace
DefineParameters(A.ncols(), nevp, &A, ARdsSymMatrix<FLOAT>::MultMv,
by
DefineParameters(A.ncols(), nevp, &A, &ARdsSymMatrix<FLOAT>::MultMv,

arerror.h

Replace
#include <iostream.h>

by
#include <iostream>

arpackf.h

Change the sentences at the end (Because arcomp.h is not included, '}' will be ignored and thus the entern closure is not completed).

}
#endif // ARCOMP_H

to
#endif // ARCOMP_H
} // extern "C" {


Build ARPACK on Windows for Visual Studio

posted Jan 19, 2015, 11:33 PM by Teng-Yok Lee   [ updated Jan 21, 2015, 7:27 AM ]

This is a revision of Jernej Barbic's steps to build ARPACK for Visual Studio. The original steps are in the URL below:

http://www-bcf.usc.edu/~jbarbic/arpack.html

I met issues when executing step 3 so I decide to put my modified procedure here. For 32-bit, you can try both gfortran or g77. For 64-bit, only gfortran can work.

My recommendation is to use the MinGW64 in CYGWIN64.

Use gfortran (Win32)

  1. Modify ARmake.inc in the source code root. I extracted the source code of ARPACK to D:\src\ARPACK. Also I cannot find f77 in the latest MinGW so I used gfortran instead. Thus the following variables should be changed accordingly:
    home = /d/src/ARPACK
    PLAT = win32
    FC      = gfortran
    FFLAGS    = -O
  2. Open a MSYS window and:
    cd /d/src/ARPACK.
  3. Compile the .f to .o:
    make lib
  4. Wrap the *.o to .dll. I need to add the library gfortran:
    dllwrap --export-all-symbols BLAS/*.o LAPACK/*.o SRC/*.o UTIL/*.o -lg2c -lgfortran --output-def arpack_win32.def -o arpack_win32.dll
  5. Open a Visual Studio Command Prompt and:
    cd d:\src\ARPACK
  6. Generate the library:
    lib.exe /machine:i386 /def:arpack_win32.def

Use g77 (Win32)

  1. Modify ARmake.inc in the source code root. I extracted the source code of ARPACK to D:\src\ARPACK. Also I cannot find f77 in the latest MinGW so I used gfortran instead. Thus the following variables should be changed accordingly:
    home = /d/src/ARPACK
    PLAT = win32
    FC      = g77
    FFLAGS    = -O
  2. Open a MSYS window and:
    cd /d/src/ARPACK.
  3. Compile the .f to .o:
    make lib
  4. Wrap the *.o to .dll.
    dllwrap --export-all-symbols BLAS/*.o LAPACK/*.o SRC/*.o UTIL/*.o -lg2c --output-def arpack_win32.def -o arpack_win32.dll
  5. Open a Visual Studio Command Prompt and:
    cd d:\src\ARPACK
  6. Generate the library:
    lib.exe /machine:i386 /def:arpack_win32.def

Use gfortran (Win 64) in MSYS

Before the procedure, install MinGW 64. In my case, it is installed to C:\Program Files (x86)\mingw-w64, and gfortran.exe is in C:\Program Files (x86)\mingw-w64\i686-4.9.2-posix-dwarf-rt_v3-rev1\mingw32\bin.
  1. Modify ARmake.inc in the source code root. I extracted the source code of ARPACK to D:\src\ARPACK. Also I cannot find f77 in the latest MinGW so I used gfortran instead. Thus the following variables should be changed accordingly:
    home = /d/src/ARPACK
    PLAT = x64
    FC = /c/Program\ Files\ \(x86\)/mingw-w64/i686-4.9.2-posix-dwarf-rt_v3-rev1/mingw32/bin/gfortran.exe
    FFLAGS    = -O
    RANLIB = /c/Program\ Files\ \(x86\)/mingw-w64/i686-4.9.2-posix-dwarf-rt_v3-rev1/mingw32/bin/ranlib.exe

  2. Open a MSYS window and:
    cd /d/src/ARPACK
  3. Extend PATH:
    export PATH=$PATH:/c/Program\ Files\ \(x86\)/mingw-w64/i686-4.9.2-posix-dwarf-rt_v3-rev1/mingw32/opt/bin
  4. Change UTIL/second. Remove the sentence: EXTERNAL           ETIME.
  5. Compile the .f to .o:
    make lib
  6. Wrap the *.o to .dll:
    /c/Program\ Files\ \(x86\)/mingw-w64/i686-4.9.2-posix-dwarf-rt_v3-rev1/mingw32/bin/dllwrap.exe --export-all-symbols BLAS/*.o LAPACK/*.o SRC/*.o UTIL/*.o -lgfortran --output-def arpack_x64.def -o arpack_x64.dll
  7. Open a Visual Studio (64-bit) Command Prompt and:
    cd d:\src\ARPACK
  8. Generate the library:
    lib.exe /machine:X64 /def:arpack_x64.def

Use gfortran (Win 64) in cygwin

Later I found later MinGW is also available in CYGWIN, which make the commands shorter.

Before the procedure, install mingw64-x86_64-gcc in CYGWIN 64.

  1. Modify ARmake.inc in the source code root. I extracted the source code of ARPACK to D:\src\ARPACK. Also I cannot find f77 in the latest MinGW so I used gfortran instead. Thus the following variables should be changed accordingly:
    home = /cygdrive/d/src/ARPACK
    PLAT = x64
    FC = /usr/bin/x86_64-w64-mingw32-gfortran.exe
    FFLAGS    = -O
    RANLIB = /usr/bin/x86_64-w64-mingw32-ranlib.exe

  2. Open a MSYS window and:
    cd /cygdrive/d/src/ARPACK
  3. Change UTIL/second. Remove the sentence: EXTERNAL           ETIME.
  4. Compile the .f to .o:
    make lib
  5. Wrap the *.o to .dll:
    /usr/bin/x86_64-w64-mingw32-dllwrap.exe --export-all-symbols BLAS/*.o LAPACK/*.o SRC/*.o UTIL/*.o -lm -lgfortran --output-def arpack_x64.def -o arpack_x64.dll
  6. Open a Visual Studio (64-bit) Command Prompt and:
    cd d:\src\ARPACK
  7. Generate the library:
    lib.exe /machine:X64 /def:arpack_x64.def

Link the lib

After building .lib and .dll, adding the directories of related dll. to your path. For CYGWIN, the path is

C:\cygwin64\usr\x86_64-w64-mingw32\sys-root\mingw\bin

For MinGW, it should be the path to the 64-bit.

If the .dll does not match to the linked library, it might show error The application was unable to start correctly (0xc000007b). I saw it when my system had both 32-bit and 64-bit versions of MinGW. After I uninstalled both and just kept CYGWIN 64, the problem was solved.

References

  1. External issue of etime: https://gcc.gnu.org/ml/fortran/2007-03/msg00305.html
  2. Use CYGWIN and MinGW64 to build Visual Studio libraries: https://github.com/arrayfire/arrayfire/wiki/CBLAS-for-Windows
  3. Cannot start my application: http://stackoverflow.com/questions/25124182/mingw-gcc-the-application-was-unable-to-start-correctly-0xc000007b

A dirty trick to install numpy via distutils without gcc 4.9

posted Dec 15, 2014, 12:17 AM by Teng-Yok Lee   [ updated Dec 15, 2014, 12:17 AM ]

After python 2.7.8, its distutils use gcc 4.9 as the C compiler, which introduces a new flag -fstack-protector-strong. Also, the AR executable is changed. However, on a system without gcc 4.9, this can fail. Consequently, numpy cannot be built.

While not all package management system has gcc 4.9 (e.g. old Ubuntu AMI from AWS), a quick trick is to modify the following file:

/usr/lib/python2.7/plat-x86_64-linux-gnu/_sysconfigdata_nd.py

First, remove -fstack-protector-strong in this file. Second, change the value of AR to 'ar'.


Export the link to pdf format of Google Doc

posted Nov 15, 2014, 4:12 PM by Teng-Yok Lee   [ updated Dec 15, 2014, 12:20 AM ]

As I use Google Doc to edit my CV, before I needed to download the file as pdf, and upload to my webpage. Nevertheless, the link to google doc is very simple: First, edit the file to get the URL. Ten change /edit in the url to /export?format=pdf.

NOTE: To make the links accessible to others, the doc link should be viewable to others.

REF: http://webapps.stackexchange.com/questions/8106/link-to-view-pdf-version-of-a-google-doc 

Disable GPU Acceleration on Firefox

posted Sep 14, 2014, 8:13 AM by Teng-Yok Lee   [ updated Dec 15, 2014, 12:19 AM ]

  1. Type about:config as the URL.
  2. Search for gfx.direct2d.disabled
  3. Double click to make it "true".

REF: https://support.mozilla.org/en-US/questions/922995

PyDev: My FAQ

posted Aug 15, 2014, 10:55 AM by Teng-Yok Lee   [ updated Aug 15, 2014, 1:57 PM ]

Q: What to do when receive the error messages "Failed to read server's response: Connection refused: connect"?
A: Increase the connection attempt.
REF: https://www.mail-archive.com/pydev-users@lists.sourceforge.net/msg03298.html

Generate histograms from bash commands

posted Jun 22, 2014, 8:01 PM by Teng-Yok Lee

REF: http://www.smallmeans.com/notes/shell-history/

For instance, to find the histogram of commands:
history|awk '{print $2}'|sort|uniq -c

Install mpi4py on Amazon EC2

posted May 29, 2014, 8:00 AM by Teng-Yok Lee   [ updated May 29, 2014, 8:36 AM ]

It sounds very simple

# REF: http://mpi4py.scipy.org/docs/usrman/install.html
$ sudo pip install mpi4py

Nevertheless, as my EC2 instance use openmpi-devel installed via yum, it cannot find mpi.h. Later I searched for the installation path and found it:

# REF: http://stackoverflow.com/questions/1766380/determining-the-path-that-a-yum-package-installed-to
$ rpm -ql openmpi-devel | grep mpi.h
/usr/include/openmpi-x86_64/mpi.h

Then how to run pip with the specified mpi.h?

NOTE: mpi4py will look for pyconfig.h and Python.h, which are only available in python2.7.

As a result, build from the source is needed. The following steps are my procedure:
$ wget http://mpi4py.googlecode.com/files/mpi4py-1.3.1.tar.gz
$ tar zxvf mpi4py-1.3.1.tar.gz
# Edit the mpi section in mpi4py-1.3.1/mpi.cfg as follows

## define_macros        =
## undef_macros         =
## include_dirs         = %(mpi_dir)s/include
include_dirs         = /usr/include/openmpi-x86_64:/usr/include/python2.7
## libraries            = mpi
libraries            = mpi
## library_dirs         = %(mpi_dir)s/lib
library_dirs         = /usr/lib64/openmpi/lib
## runtime_library_dirs = %(mpi_dir)s/lib
runtime_library_dirs = /usr/lib64/openmpi/lib
$ python2.7 setup.py build
$ sudo python2.7 setup.py install

# Edit ~/.bashrc by adding the following line:
# export PYTHONPATH=$PYTHONPATH:/usr/lib64/python2.7/site-packages/

Meanwhile, I also use mpirun+NFS to install mpi4py to all nodes.

$ cd /mnt/share/mpi4py-1.3.1
$ mpirun -hostfile ~/hostfile -n 2 -npernode 1 sudo python2.7 setup.py install

NOTE: To make this work, root need the permission to write to the NFS (here /mnt/share). Then in /etc/exports, the permission for each drive should be (rw,no_root_squash) (REF: http://superuser.com/questions/605464/unable-to-write-as-root-but-can-as-user)

My memo on python

posted May 17, 2014, 8:56 AM by Teng-Yok Lee   [ updated May 27, 2014, 1:22 AM ]

My tricks

Write a script that can be imported

I usually write python scripts for small task. Scripts can be executed quickly (without setting program variables), but they are hard to debug when scripts are calling scripts. In such a case, the ability to run scripts like a module will be better.

Here is my template to write a python script so it can be also called as a module. In each script, I just define a method main(), which define the main computation.

from optparse import OptionParser;

# Define the parser.
parser = OptionParser()
...

def main(arguments):
    (options, args) = parser.parse_args(arguments)
    ...

    return 0;

Then I will check whether this script is directly called by python. If yes, immediately call main():

# Entry point of the script.
import os;
import sys;
...
if( os.path.basename(sys.argv[0]) == os.path.basename(__file__) ):
    exit(main(sys.argv));

To call this script from another script (says my_script), now just called its main():

import my_script;
...

# Setup the argument.
arguments = [
...];

my_script.main(arguments);

Useful links about advanced python usage:

How to get the python.exe location programmatically?

import sys;
...
python_exec = sys.executable;

Usage: On Windows, I use WinPython + Eclipse for developing. If I want to call another python scripts, I need to pass the correct python interpreter. Otherwise, the default python might not have the packages I needed (such as numpy).

In Python, how do I get the path and name of the file that is currently executing?

Use the keyword __file__.


1-10 of 99