Differences

This shows you the differences between two versions of the page.

--- howto:compiling_python_2_7_3 [2012/07/27 14:19]
kevin created
+++ howto:compiling_python_2_7_3 [2021/12/09 16:42] (current)
@@ Line 1: / Line 1: @@
-====== Installing Python 2.7.3 from source on the Sun Tsessbe Cluster ======
+===== Installing Python 2.7.3 from Source =====
-Before following this guide read through the README file located in the Python Source Install Folder. Also it recommended to make use of temporary or scratch directory to do all the compilation in (i.e. ~/scratch or /tmp/$USER), as to save disk space in your home directory. This guide assumes GCC 4.6.2 has been installed in your home directory as  in [[guide:compiling_gcc|Compiling GCC from Source]]. Ensure that the following lines at the end of your ./bashrc file
+This guide follows from GCC 4.7.1 install from source Guide, and assumes that
-  export LD_LIBRARY_PATH=$HOME/lib:$LD_LIBRARY_PATH
+your bash environmental are setup accordingly. Building directory
-  export LD_LIBRARY_PATH=$HOME/lib64:$LD_LIBRARY_PATH
-  export PATH=$HOME/bin:$PATH
-  export LDFLAGS="-L$HOME/lib64 -L$HOME/lib"
-  export CPPFLAGS="-I$HOME/include -L$HOME/lib64 -L$HOME/lib"
-  export CXXFLAGS=$CPPFLAGS
-  export F77=gfortran
-log out and in again,
+  $ mkdir -p /tmp/$USER/python_build
-  $  gcc --version
-  gcc (GCC) 4.6.2
-===== Prerequisites  =====
+All libaries and binaries installed in $HOME/local folder, and will require and
+additional 110M of disk space.
-Before compiling Python, the zlib library needs to be installed.
-  $ cd ~/scratch
+===== Opptional pre-requisites =====
-  $ wget http://zlib.net/zlib-1.2.6.tar.gz
-  $ tar -xf zlib-1.2.6.tar.gz
+==== zlib ====
-  $ cd zlib-1.2.6
+[[ http://zlib.net | zlib ]] is designed to be a free, general-purpose,
+legally unencumbered -- that is, not covered by any patents -- lossless
+data-compression library for use on virtually any computer hardware and
+operating system.
+  $ cd /tmp/$USER/python_build
+  $ wget http://zlib.net/zlib-1.2.7.tar.gz
+  $ tar -xf zlib-1.2.7.tar.gz
+  $ cd zlib-1.2.7
+  $ ./configure --prefix=$HOME/local --libdir=$HOME/local/lib64 --64
   $ make
-Test it!
   $ make check
-Install if all tests pass
+  $ make install
-  $ make install
-===== Download Source =====
+==== tcl/tk ====
-  $ cd ~/scratch
+[[http://www.tcl.tk/about/ |  tcl/tk ]] installation. first tcl
-  $ wget http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz
+  $ cd /tmp/$USER/python_build
-  $ tar -xf Python-2.7.3.tgz
+  $ wget http://prdownloads.sourceforge.net/tcl/tcl8.5.12-src.tar.gz
+  $ tar -xf tcl8.5.12-src.tar.gz
-===== Configuration =====
+  $ cd tcl8.5.12
+  $ mkdir BUILD
+  $ cd BUILD
+  $ ../unix/configure --prefix=$HOME/local --libdir=$HOME/local/lib64 \
+  --enable-64bit --enable-64bit-vis
+  $ make
+  $ make test
+Summary of test results were
+  Tests ended at Mon Sep 10 16:00:00 SAST 2012
+  all.tcl:    Total  27048  Passed   25880	Skipped	1166	Failed	2
+  Sourced 137 Test Files.
+  Files with failing tests: httpold.test unixInit.test
+If acceptable, proceed
+  $ make install
-   $ cd ~/scratch/Python-2.7.3
+then tk
-   $ mkdir BUILD
+  $ cd /tmp/$USER/python_build
-   $ cd BUILD
+  $ wget http://prdownloads.sourceforge.net/tcl/tk8.5.12-src.tar.gz
-   $ ../configure --prefix=$HOME
+  $ tar -xf tk8.5.12-src.tar.gz
+  $ mkdir tk8.5.12/BUILD
-===== Build =====
+  $ cd tk8.5.12/BUILD
+  $ ../unix/configure --with-x --prefix=$HOME/local --libdir=$HOME/local/lib64  \
+  --enable-64bit --enable-64bit-vis
+  $ make
+  $ make install
-   $ make
+==== sqlite 3 ====
-If errors occur such as
+[[ http://www.sqlite.org/about.html | SQLite ]] is a in-process library that implements a self-contained,
-  * g++: Internal error: Killed (program cc1plus)
+serverless, zero-configuration, transactional SQL database engine.
-wait a while and then make again, as the process was killed by the usage limiter which runs on the login node. If nessary try the approach shown in [[guide:compiling_gcc#compiling_gcc|Compiling GCC from Source]].
-===== Check =====
+  $ cd /tmp/$USER/python_build
+  $ wget http://www.sqlite.org/sqlite-autoconf-3071400.tar.gz
+  $ tar -xf sqlite-autoconf-3071400.tar.gz
+  $ cd sqlite-autoconf-3071400
+  $ mkdir BUILD
+  $ cd BUILD
+  $ ../configure --prefix=$HOME/local --libdir=$HOME/local/lib64
+  $ make
+no tests with package?
+  $ make install
-  $ make test
+Got warnings
+ldconfig: $HOME/local/lib64/libisl.so.10.0.0-gdb.py is not an ELF file - it has the wrong magic bytes at the start.
-No news is good news: If this test command completes with out an error you know that Python has been compiled successfully.
+Where ELF = Executable and Linkable Format (ELF, formerly called Extensible
+Linking Format). This is assumed okay as these are python module files, and
+not shared object files. The .so.10.0.0-gdb.py extention is probabily
+confusing ldconfig, as least i hope.
-===== Install =====
+Online docs also indicate that reconfiguring ld may fix the problem. If there
+is problem, try reinstalling binutils?
+==== GDB : the GNU debugger ====
+  $ download_and_install_from_source.sh \
+  http://ftp.gnu.org/gnu/gdb/gdb-7.5.tar.bz2 \
+  /tmp/$USER/python_build "--libdir=$HOME/local/lib64 --with-python" $HOME/local
+Testing took 20 minutes  to complete
+Got quite a lot of failures here...
+FIXME please...
+===== Download and Configuration =====
+  $ cd /tmp/$USER/python_build
+  $ wget http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2
+  $ tar -xf Python-2.7.3.tar.bz2
+  $ mkdir Python-2.7.3/BUILD
+  $ cd Python-2.7.3/BUILD
+  $ ../configure --prefix=$HOME/local
+Does thread support work? if not add the --with-threads=no option to the above
+configure
+===== Compile =====
+  $ screen
+  $ make > make.out 2>&1
+  # ctrl-a ctrl-d to detach screen
+  $ tail -f make.out # to watch compilation progress
+giving
+  Python build finished, but the necessary bits to build these modules were not
+  found:
+  bsddb185           dl        imageop
+  sunaudiodev
+  To find the necessary bits, look in setup.py in detect_modules() for the
+  module's name.
+Where
+  * dl is depreciated since Python 2.6, use ctypes instead.
+  * Beginning in 2.3 some Unix versions of Python may have a bsddb185 module. This is present only to allow backwards compatibility with systems which ship with the old Berkeley DB 1.85 database library. The bsddb185 module should never be used directly in new code. The module has been removed in Python 3. If you find you still need it look in PyPI.
+  * imageop and sunaudiodev, do not work on 64 bit systems.
+===== Test =====
+  $ screen
+  $ make test > make.test 2>&1
+  # ctrl-a ctrl-d to detach screen
+  $ tail -f make.test
+Test Summary:
+tests OK.
+test failed:
+     test_gdb
+tests skipped:
+     test_aepack test_al test_applesingle test_bsddb185 test_bsddb3
+     test_cd test_cl test_codecmaps_cn test_codecmaps_hk
+     test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
+     test_dl test_gl test_imageop test_imgfile test_kqueue
+     test_linuxaudiodev test_macos test_macostools test_msilib
+     test_ossaudiodev test_scriptpackages test_smtpnet
+     test_socketserver test_startfile test_sunaudiodev test_timeout
+     test_tk test_ttk_guionly test_urllib2net test_urllibnet
+     test_winreg test_winsound test_zipfile64
+skips unexpected on linux2:
+     test_tk test_ttk_guionly
+The unexpected test were
+  skipped -- tk not available: no display name and no $DISPLAY environment variable
+As for test_gdb, which i assume is the GNU debugger interfacing module, more
+information on the error can be obtained by
+  $ ./python ../Lib/test/regrtest.py -v test_gdb
+If this is good enough for your applications proceed with the installation.
+===== Installation =====
   $ make install
+  $ ln -s ~/local/lib/libpython2.7.a ~/local/lib64/
-If your PATH environmental variable is configured to include $HOME/bin, then
+clean up after yourself,
+  $ rm -r /tmp/$USER/python_build
+Then you should see the following
   $ python
-  Python 2.7.3 (default, Apr 12 2012, 12:03:56)
+  Python 2.7.3 (default, Sep 11 2012, 11:01:22)
-  [GCC 4.6.2] on linux2
+  [GCC 4.7.1] on linux2
   Type "help", "copyright", "credits" or "license" for more information.
-  >>>
+   >>>
-===== Installing Packages =====
+===== Installing Some commom python packages =====
-Before beginning make sure bash environmental variables do not cause a conflict
+==== setupTools ====
-  unset CPPFLAGS
-  unset LDFLAGS
-=== iPython ===
+Start with python setup tools, as it contains the easy install tool.
   $ cd ~/scratch
-  $ wget http://archive.ipython.org/release/0.12/ipython-0.12.tar.gz
+  $ wget http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz
-  $ tar -xf ipython-0.12.tar.gz
+  $ tar -xf setuptools-0.6c11.tar.gz
-  $ cd ipython-0.12
+  $ cd setuptools-0.6c11
   $ python setup.py install
+Refer to the [[http://peak.telecommunity.com/DevCenter/EasyInstall|reference
+and manual website]] for further information on using easy install.
+==== Nose ====
+Nose is a testing module, used by packages such as Numpy and Scipy. To install
+simply
+  $ easy_install nose
+==== iPython ====
+An enhanced Interactive Python shell, with lots of goodies ...
+  $ easy_install ipython
+==== mpi4py ====
+[[https://code.google.com/p/mpi4py/|mpi4py]] is python interface to
+[[http://www.open-mpi.org/|openmpi]] which is a High Performance Message
+Passing Library.
+To start we are going to build and test the openmpi libraries.
+  $ cd ~/scratch
+  $ wget http://www.open-mpi.org/software/ompi/v1.6/downloads/openmpi-1.6.1.tar.bz2
+  $ tar -xf openmpi-1.6.1.tar.bz2
+  $ cd openmpi-1.6.1
+  $ mkdir BUILD
+  $ cd BUILD
+  $ ../configure --prefix=$HOME/local --libdir=$HOME/local/lib64
+configure taks a minutes or two, with lots of output ....
+  $ make
+  $ make check > make.check 2>&1  #takes a minute to complete
+  $ grep -i -e pass -e fail make.check
+  $ make install
+Download mpi4py
+  $ cd ~/scratch
+  $ easy_install --editable --build-directory . mpi4py
+  $ cd mpi4py
+  $ python setup.py build
+  $ python setup.py test
+if no errors install
+  $ python setup.py install
+==== NumPy and SciPy====
+[[http://numpy.scipy.org/|NumPy]] is the fundamental package for scientific computing with Python. It
+contains among other things:
+  *  a powerful N-dimensional array object
+  *  sophisticated (broadcasting) functions
+  *  tools for integrating C/C++ and Fortran code
+  *  useful linear algebra, Fourier transform, and random number capabilities
+Based on [[http://www.scipy.org/Installing_SciPy/Linux|installation guide]].
+=== BLAS and LAPACK libraries ===
+Following this [[http://www.tfinley.net/notes/blas-lapack/|guide]]
+Download the LAPACK source and extract, setup make and build as follows,
+  $ cd ~/scratch
+  $ wget http://www.netlib.org/lapack/lapack-3.4.1.tgz
+  $ tar -xf lapack-3.4.1.tgz
+  $ cd lapack-3.4.1
+Setup make,
+  $ cp make.inc.example make.inc
+  $ emacs make.inc
+  # change
+  #  OPTS     = -O2  and, NOOPT    = -O0
+  # to
+  #  OPTS     = -O2 -fPIC and, NOOPT    = -O0 -fPIC
+  # save (crtl-x crtl-s) and exit (crtl-x ctrl-c)
+  $ make blaslib
+Make LAPACK
+  $ screen
+  $ make all > make.out 2>&1
+  # crtl-a d
+  $ tail -f make.out
+After 10 minutes make will finish, and you should see test results such as
+		 -->   LAPACK TESTING SUMMARY  <--
+       Processing LAPACK Testing output found in the TESTING  direcory
+  SUMMARY		       nb test run	numerical error		other error
+  REAL		        1077227		0	(0.000%)	0   (0.000%)
+  DOUBLE PRECISION	1078039		0	(0.000%)	0   (0.000%)
+  COMPLEX	                522814		0	(0.000%)	0   (0.000%)
+  COMPLEX16		552410		0	(0.000%)	0   (0.000%)
+  --> ALL PRECISIONS	3230490		0	(0.000%)	0   (0.000%)
+if all went well
+  $ cp -iv lib* $HOME/local/lib64
-provided your $HOME/bin in your PATH variable, ipython should now be available
+To convert the built static libraries to dynamic libraries, the following bash
+loop is used
+  $ for LIB in *.a ; do
+      DIR=${LIB}_contents
+      SOLIB=${LIB%.*}.so
+      echo $SOLIB
+      mkdir $DIR
+      cd $DIR
+      ar -x ../$LIB
+      cd -
+      gfortran -shared -Wl,-soname=$SOLIB -o $SOLIB ${DIR}/*.o
+    done;
+  $ ln -snf librefblas.so libblas.so
+Copy the shared object or dynamic libraries to your library folder
+  $ cp -iv lib*.so $HOME/local/lib64
+[warning dynamic libaries not fully tested]
-=== Nose ===
+=== Numpy building, testing and installation ===
+Download numpy
   $ cd ~/scratch
-  $ wget http://pypi.python.org/packages/source/n/nose/nose-1.1.2.tar.gz#md5=144f237b615e23f21f6a50b2183aa817
+  $ wget   http://tenet.dl.sourceforge.net/project/numpy/NumPy/1.6.2/numpy-1.6.2.tar.gz
-  $ tar -xf nose-1.1.2.tar.gz
+  $ tar -xf numpy-1.6.2.tar.gz
-  $ cd nose-1.1.2
+  $ cd numpy-1.6.2
+== build using shared object libraries ==
+  $ echo "[DEFAULT]" > site.cfg
+  $ echo "library_dirs = $HOME/local/lib64" >> site.cfg
+  $ echo "include_dirs = $HOME/local/include"  >> site.cfg
+  $ python setup.py build --fcompiler=gnu95
   $ python setup.py install
-=== Numpy ===
-Before beginning compile the LAPACK and BLAS librariess as in [[guide:compiling_octave|Compiling Octave 3.4.3 from Source]], and copy libraries to $HOME.lib:
+== build using static lapack and blas libraries (for interests sake only) ==
-  $lapack-3.4.0> cp lib* $HOME/lib
-Download the last version of numpy source, in your scratch folder. Extract it, run python setup.py install. Then test it (note this requires nose)
+Set blas and lapack paths
-  $ cd ~
+  $ export ATLAS=None
+  $ export BLAS=$HOME/local/lib64/librefblas.a
+  $ export LAPACK=$HOME/local/lib64/liblapack.a
+Build and install
+  $ python setup.py build --fcompiler=gnu95
+NB to build without lapack and blas libraries, do the following
+  $ export ATLAS=None
+  $ export BLAS=None
+  $ export LAPACK=None
+then build
+speed difference: using LAPACK and BLAS libraries numpy testing takes between
+and 12s, without the libraries numpy.test() takes exactly the same amount
+of time?
+== testing ==
+  $ mkdir -p /tmp/$USER/numpy-1.6.2_testing
+  $ python setup.py install --prefix=/tmp/$USER/numpy-1.6.2_testing
   $ ipython
-  $ import numpy
+    $ import os, sys
-  $ numy.test()
+    $ sys.path.insert(0,'/tmp/%s/numpy-1.6.2_testing/lib/python2.7/site-packages'  % os.environ['USER'])
+    $ import numpy
+    $ numpy.test()
-=== matplotlib / pylab ===
+If all the test were passed, rm the testing directory and install numpy
+  $ rm -r /tmp/$USER/numpy-1.6.2_testing
+  $ python setup.py install
-  Download latest version from source forge (version 1.1 in this case)
-  $  cd $MATPLOTLIB_FOLDER
-  $  python setup.py install
-=== Scipy (minimal install) ===
-This requires that numpy already be installed and that LAPACK library are setup as in the numpy section. Before beginning the scipy installation, clear some possibly conflicting bash environmental variables
+=== Scipy, minimal install ===
-  unset CPPFLAGS LDFLAGS
-Then
   $ cd ~/scratch
   $ wget http://tenet.dl.sourceforge.net/project/scipy/scipy/0.10.1/scipy-0.10.1.tar.gz
   $ tar -xf scipy-0.10.1.tar.gz
   $ cd scipy-0.10.1/
-  $ export ATLAS=~/scratch/ATLAS/lib/Linux_HAMMER64SSE2/
+  $ python setup.py build #takes about 10 minutes to build
-  $ python setup.py build
-Once the build has been successfully completed, its time to install and test scipy
+testing,
+  $ mkdir -p /tmp/$USER/scipy-0.10.1
+  $ python setup.py install --prefix=/tmp/$USER/scipy-0.10.1_testing
+  $ ipython
+    $ import os, sys
+    $ sys.path.insert(0,'/tmp/%s/scipy-0.10.1_testing/lib/python2.7/site-packages'  % os.environ['USER'])
+    $ import scipy
+    $ scipy.test()
+which did not work 100 % :(
+  FAILED (KNOWNFAIL=13, SKIP=41, failures=1)
+  Out[2]: <nose.result.TextTestResult run=5101 errors=0 failures=1>
+If testing satisfactory, rm the testing directory and install scipy
+  $ rm -r /tmp/$USER/scipy-0.10.1_testing
   $ python setup.py install
-  $ cd ~
+==== matplotlib / pylab ====
-  $ python
-  $ import scipy
-  $ scipy.test()
-which did not work :(
-  FAILED (KNOWNFAIL=13, SKIP=41, failures=2)
-  Out[2]: <nose.result.TextTestResult run=5101 errors=0 failures=2>
+  Download latest version from source forge (version 1.1 in this case)
+  $  cd $MATPLOTLIB_FOLDER
+  $  python setup.py install

wiki.chpc.ac.za

User Tools

Site Tools

Differences

Page Tools