wiki.chpc.ac.za

This guide follows from GCC 4.7.1 install from source Guide, and assumes that your bash environmental are setup accordingly. Building directory

$ mkdir -p /tmp/$USER/python_build

All libaries and binaries installed in $HOME/local folder, and will require and additional 110M of disk space.

zlib is designed to be a free, general-purpose, legally unencumbered – that is, not covered by any patents – lossless data-compression library for use on virtually any computer hardware and operating system.

$ cd /tmp/$USER/python_build
$ wget http://zlib.net/zlib-1.2.7.tar.gz
$ tar -xf zlib-1.2.7.tar.gz
$ cd zlib-1.2.7
$ ./configure --prefix=$HOME/local --libdir=$HOME/local/lib64 --64
$ make
$ make check
$ make install 

tcl/tk installation. first tcl

$ cd /tmp/$USER/python_build
$ wget http://prdownloads.sourceforge.net/tcl/tcl8.5.12-src.tar.gz
$ tar -xf tcl8.5.12-src.tar.gz
$ cd tcl8.5.12
$ mkdir BUILD
$ cd BUILD
$ ../unix/configure --prefix=$HOME/local --libdir=$HOME/local/lib64 \
--enable-64bit --enable-64bit-vis 
$ make
$ make test

Summary of test results were

Tests ended at Mon Sep 10 16:00:00 SAST 2012
all.tcl:    Total  27048  Passed   25880	Skipped	1166	Failed	2
Sourced 137 Test Files.
Files with failing tests: httpold.test unixInit.test

If acceptable, proceed

$ make install

then tk

$ cd /tmp/$USER/python_build
$ wget http://prdownloads.sourceforge.net/tcl/tk8.5.12-src.tar.gz
$ tar -xf tk8.5.12-src.tar.gz
$ mkdir tk8.5.12/BUILD
$ cd tk8.5.12/BUILD
$ ../unix/configure --with-x --prefix=$HOME/local --libdir=$HOME/local/lib64  \
--enable-64bit --enable-64bit-vis
$ make
$ make install

SQLite is a in-process library that implements a self-contained, serverless, zero-configuration, transactional SQL database engine.

$ cd /tmp/$USER/python_build
$ wget http://www.sqlite.org/sqlite-autoconf-3071400.tar.gz
$ tar -xf sqlite-autoconf-3071400.tar.gz
$ cd sqlite-autoconf-3071400
$ mkdir BUILD
$ cd BUILD
$ ../configure --prefix=$HOME/local --libdir=$HOME/local/lib64 
$ make

no tests with package?

$ make install

Got warnings ldconfig: $HOME/local/lib64/libisl.so.10.0.0-gdb.py is not an ELF file - it has the wrong magic bytes at the start.

Where ELF = Executable and Linkable Format (ELF, formerly called Extensible Linking Format). This is assumed okay as these are python module files, and not shared object files. The .so.10.0.0-gdb.py extention is probabily confusing ldconfig, as least i hope.

Online docs also indicate that reconfiguring ld may fix the problem. If there is problem, try reinstalling binutils?

$ download_and_install_from_source.sh \
http://ftp.gnu.org/gnu/gdb/gdb-7.5.tar.bz2 \
/tmp/$USER/python_build "--libdir=$HOME/local/lib64 --with-python" $HOME/local

Testing took 20 minutes to complete Got quite a lot of failures here…

please…

$ cd /tmp/$USER/python_build
$ wget http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2
$ tar -xf Python-2.7.3.tar.bz2
$ mkdir Python-2.7.3/BUILD
$ cd Python-2.7.3/BUILD
$ ../configure --prefix=$HOME/local 

Does thread support work? if not add the –with-threads=no option to the above configure

$ screen 
$ make > make.out 2>&1
# ctrl-a ctrl-d to detach screen
$ tail -f make.out # to watch compilation progress

giving

Python build finished, but the necessary bits to build these modules were not
found:
bsddb185           dl        imageop
sunaudiodev   
To find the necessary bits, look in setup.py in detect_modules() for the
module's name.

Where

• dl is depreciated since Python 2.6, use ctypes instead.
• Beginning in 2.3 some Unix versions of Python may have a bsddb185 module. This is present only to allow backwards compatibility with systems which ship with the old Berkeley DB 1.85 database library. The bsddb185 module should never be used directly in new code. The module has been removed in Python 3. If you find you still need it look in PyPI.
• imageop and sunaudiodev, do not work on 64 bit systems.

$ screen
$ make test > make.test 2>&1
# ctrl-a ctrl-d to detach screen
$ tail -f make.test

Test Summary:

352 tests OK.
1 test failed:
   test_gdb
36 tests skipped:
   test_aepack test_al test_applesingle test_bsddb185 test_bsddb3
   test_cd test_cl test_codecmaps_cn test_codecmaps_hk
   test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
   test_dl test_gl test_imageop test_imgfile test_kqueue
   test_linuxaudiodev test_macos test_macostools test_msilib
   test_ossaudiodev test_scriptpackages test_smtpnet
   test_socketserver test_startfile test_sunaudiodev test_timeout
   test_tk test_ttk_guionly test_urllib2net test_urllibnet
   test_winreg test_winsound test_zipfile64
2 skips unexpected on linux2:
   test_tk test_ttk_guionly

The unexpected test were

skipped -- tk not available: no display name and no $DISPLAY environment variable

As for test_gdb, which i assume is the GNU debugger interfacing module, more information on the error can be obtained by

$ ./python ../Lib/test/regrtest.py -v test_gdb

If this is good enough for your applications proceed with the installation.

$ make install
$ ln -s ~/local/lib/libpython2.7.a ~/local/lib64/

clean up after yourself,

$ rm -r /tmp/$USER/python_build

Then you should see the following

$ python
Python 2.7.3 (default, Sep 11 2012, 11:01:22) 
[GCC 4.7.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> 

Start with python setup tools, as it contains the easy install tool.

$ cd ~/scratch
$ wget http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz
$ tar -xf setuptools-0.6c11.tar.gz
$ cd setuptools-0.6c11
$ python setup.py install

Refer to the reference and manual website for further information on using easy install.

Nose is a testing module, used by packages such as Numpy and Scipy. To install simply

$ easy_install nose

An enhanced Interactive Python shell, with lots of goodies …

$ easy_install ipython

mpi4py is python interface to openmpi which is a High Performance Message Passing Library.

To start we are going to build and test the openmpi libraries.

$ cd ~/scratch
$ wget http://www.open-mpi.org/software/ompi/v1.6/downloads/openmpi-1.6.1.tar.bz2
$ tar -xf openmpi-1.6.1.tar.bz2
$ cd openmpi-1.6.1
$ mkdir BUILD
$ cd BUILD
$ ../configure --prefix=$HOME/local --libdir=$HOME/local/lib64

configure taks a minutes or two, with lots of output ….

$ make
$ make check > make.check 2>&1  #takes a minute to complete
$ grep -i -e pass -e fail make.check
$ make install

Download mpi4py

$ cd ~/scratch
$ easy_install --editable --build-directory . mpi4py
$ cd mpi4py
$ python setup.py build 
$ python setup.py test

if no errors install

$ python setup.py install

NumPy is the fundamental package for scientific computing with Python. It contains among other things:

• a powerful N-dimensional array object
• sophisticated (broadcasting) functions
• tools for integrating C/C++ and Fortran code
• useful linear algebra, Fourier transform, and random number capabilities

Based on installation guide.

Following this guide

Download the LAPACK source and extract, setup make and build as follows,

$ cd ~/scratch
$ wget http://www.netlib.org/lapack/lapack-3.4.1.tgz
$ tar -xf lapack-3.4.1.tgz
$ cd lapack-3.4.1

Setup make,

$ cp make.inc.example make.inc
$ emacs make.inc
# change 
#  OPTS     = -O2  and, NOOPT    = -O0
# to
#  OPTS     = -O2 -fPIC and, NOOPT    = -O0 -fPIC
# save (crtl-x crtl-s) and exit (crtl-x ctrl-c)
$ make blaslib

Make LAPACK

$ screen
$ make all > make.out 2>&1
# crtl-a d
$ tail -f make.out

After 10 minutes make will finish, and you should see test results such as

	 -->   LAPACK TESTING SUMMARY  <--
     Processing LAPACK Testing output found in the TESTING  direcory
SUMMARY		       nb test run	numerical error		other error  
REAL		        1077227		0	(0.000%)	0   (0.000%)	
DOUBLE PRECISION	1078039		0	(0.000%)	0   (0.000%)	
COMPLEX	                522814		0	(0.000%)	0   (0.000%)	
COMPLEX16		552410		0	(0.000%)	0   (0.000%)	
--> ALL PRECISIONS	3230490		0	(0.000%)	0   (0.000%)

if all went well

$ cp -iv lib* $HOME/local/lib64

To convert the built static libraries to dynamic libraries, the following bash loop is used

$ for LIB in *.a ; do
    DIR=${LIB}_contents
    SOLIB=${LIB%.*}.so
    echo $SOLIB
    mkdir $DIR
    cd $DIR
    ar -x ../$LIB
    cd -
    gfortran -shared -Wl,-soname=$SOLIB -o $SOLIB ${DIR}/*.o
  done;
$ ln -snf librefblas.so libblas.so

Copy the shared object or dynamic libraries to your library folder

$ cp -iv lib*.so $HOME/local/lib64

[warning dynamic libaries not fully tested]

Download numpy

$ cd ~/scratch
$ wget   http://tenet.dl.sourceforge.net/project/numpy/NumPy/1.6.2/numpy-1.6.2.tar.gz
$ tar -xf numpy-1.6.2.tar.gz
$ cd numpy-1.6.2

$ echo "[DEFAULT]" > site.cfg
$ echo "library_dirs = $HOME/local/lib64" >> site.cfg
$ echo "include_dirs = $HOME/local/include"  >> site.cfg
$ python setup.py build --fcompiler=gnu95
$ python setup.py install

Set blas and lapack paths

$ export ATLAS=None
$ export BLAS=$HOME/local/lib64/librefblas.a
$ export LAPACK=$HOME/local/lib64/liblapack.a

Build and install

$ python setup.py build --fcompiler=gnu95

NB to build without lapack and blas libraries, do the following

$ export ATLAS=None
$ export BLAS=None
$ export LAPACK=None

then build

speed difference: using LAPACK and BLAS libraries numpy testing takes between 10 and 12s, without the libraries numpy.test() takes exactly the same amount of time?

$ mkdir -p /tmp/$USER/numpy-1.6.2_testing
$ python setup.py install --prefix=/tmp/$USER/numpy-1.6.2_testing
$ ipython
  $ import os, sys
  $ sys.path.insert(0,'/tmp/%s/numpy-1.6.2_testing/lib/python2.7/site-packages'  % os.environ['USER'])
  $ import numpy
  $ numpy.test()

If all the test were passed, rm the testing directory and install numpy

$ rm -r /tmp/$USER/numpy-1.6.2_testing
$ python setup.py install

$ cd ~/scratch
$ wget http://tenet.dl.sourceforge.net/project/scipy/scipy/0.10.1/scipy-0.10.1.tar.gz
$ tar -xf scipy-0.10.1.tar.gz
$ cd scipy-0.10.1/
$ python setup.py build #takes about 10 minutes to build

testing,

$ mkdir -p /tmp/$USER/scipy-0.10.1
$ python setup.py install --prefix=/tmp/$USER/scipy-0.10.1_testing
$ ipython
  $ import os, sys
  $ sys.path.insert(0,'/tmp/%s/scipy-0.10.1_testing/lib/python2.7/site-packages'  % os.environ['USER'])
  $ import scipy
  $ scipy.test()

which did not work 100 % :(

FAILED (KNOWNFAIL=13, SKIP=41, failures=1)
Out[2]: <nose.result.TextTestResult run=5101 errors=0 failures=1>

If testing satisfactory, rm the testing directory and install scipy

$ rm -r /tmp/$USER/scipy-0.10.1_testing
$ python setup.py install

Download latest version from source forge (version 1.1 in this case)
$  cd $MATPLOTLIB_FOLDER
$  python setup.py install