Adding themes to jupyter notebooks

First install https://github.com/dunovank/jupyter-themes
then do jt -t themename
to install a theme.

You can list all available themes by
jt -l

MongoDB cheat sheet

To run mongodb do

>mongod

This will start the deamon.

Then to query the DB start the client by

>mongo

 To use a DB do
use

To see the databases do
show databases

You can see the collections in a database after opening that database by
show collections

Collections correspond to Tables in mysql.

db.collection.find({key:value})

can be used to query the DB. If you want to perform "like" queries do
db.collection.find({"my key": /term/})

Do not use quotes around term in the previous example to perform like queries. If you do, then it will be an exact match.

This will not work within pymongo and you need to use $regex directive for this.
db["mycollection"].find({"key":{"$regex":"%s" % query}})

You can use count do count the number of matching records.
db.collection.count({key:value})

Python tricks and tips

If you want to assign incremental ids to a list of words then you can define a defultdict as follows:

wpids = defaultdict(lambda: len(wpids))

boost-python

Here is a minimum working example of boos-python from the official web site.

First create a hello.cpp file and put this.

#include YOU HAVE TO INCLUDE boost/python.hpp within tag markers.

char const* greet()
{
   return "hello, world";
}


BOOST_PYTHON_MODULE(hello)
{
    using namespace boost::python;
    def("greet", greet);
}

Now compline .so file by doing the following.

g++ -g -shared -fPIC -I/usr/include/python2.7 hello.cpp -lpython2.7 -lboost_python -o hello.so

Open IPython and import hello
now call hello.greet()

Debugging Python using ipdb

We can install ipdb (interactive python debugger) via pip

$ pip install ipdb

To insert a breakpoint (hard coded break points) in the code use ipdb.set_trace(). You can insert many breakpoints as you like. To run the code:

$ python mycode.py

When the breakpoint is hit, you will be dropped to ipdb prompt, similar to the ipython prompt. Use "c" (continue) command to continue the code until the next breakpoint or until the program ends.

For example,

import sparsesvd
import ipdb

def f(x):
    print x
    y = 2 *x 
    ipdb.set_trace()
    print y
    z = y * 3
    ipdb.set_trace()
    print z
    pass

if __name__ == "__main__":
    f(10)
    print "all done"

Python nose testing

Nose provide additional testing functionalities that are not provided in unittests framework. Unlike unittests which is a standard python module you will have to easy_install nose as described here.


use test_mytest.py to name a test module and test_method to name any test methods. If you do so the nosetests will be able to discover and execute those tests automatically.

You will have to use assert_equal(A, B) to test whether A is equal to B.
For numpy arrays use numpy.testing.assert_array_equal instead to compare numpy arrays.

To test the unit tests you must call nosetests as follows.

To display the name of the test functions

nosetests -v test.py

To display the print statements as well

nosetests -v -s test.py

Execute a specific test function within a class in a test module

nosetests -v -s test.py:className:test_method 

 

Synchronized decorator for Python

The following decorator provides synchronization for Python codes.
Reference

def synchronized(lock):
    """ Synchronization decorator. """

    def wrap(f):
        def newFunction(*args, **kw):
            lock.acquire()
            try:
                return f(*args, **kw)
            finally:
                lock.release()
        return newFunction
    return wrap

if __name__ == '__main__':
    from threading import Thread, Lock
    import time

    myLock = Lock()

    class MyThread(Thread):
        def __init__(self, n):
            Thread.__init__(self)
            self.n = n

        @synchronized(myLock)
        def run(self):
            """ Print out some stuff.

            The method sleeps for a second each iteration.  If another thread
            were running, it would execute then.
            But since the only active threads are all synchronized on the same
            lock, no other thread will run.
            """

            for i in range(5):
                print 'Thread %d: Start %d...' % (self.n, i),
                time.sleep(1)
                print '...stop [%d].' % self.n

    threads = [MyThread(i) for i in range(10)]
    for t in threads:
        t.start()

    for t in threads:
        t.join()

numpy dot vs inner

dot and inner returns the same result for 1D arrays.
However, for 2D arrays (matrices), dot gives the matrix product, whereas inner gives the sum-product over the last axis.

References
inner vs. dot
inner document
dot document

what is sum product?

A = [[x1, x2], [x3, x4]]
B = [[y1, y2], [y3, y4]]

sum product of A and B is given by numpy.inner(A,B) as follows
A's row i is element-wise multiplied by B's row j and the values are added to get the (i,j) element

[[x1*y1+x2*y2, x1*y3+x2*y4], [x3*y1+x4*y2, x3*y3+x4*y4]]

Multiprocessing in Python

There are numerous ways to multiprocess in python. Pools give a very easy way.

Note: If you do not catch keyboard interrups as shown in the code then you will have to wait until all workers have finished even if you send Ctrl+C to the parent process. There are other methods such as map (blocks until each result is received and orders sequentially), apply and apply_async where you can provide a callback function that actually determines what must be done with the result. If the task items are not iterable then you must use apply or apply_async and handle the input in the callback.

from multiprocessing import Pool
from functools import partial

import time
import sys

def doWork(i, x):
    """
    Does the actual work.
    """
    print i
    time.sleep(2)
    return (x * i)
    pass


def multi():
    """
    Does the actual multiprocessing.
    """
    # Create the pool.
    pool = Pool()

    # We must convert the doWork to a function of one argument.
    # This argument will be filled in by an iterator.
    partialWorker = partial(doWork, x=5)

    tasks = range(100)

    p = pool.map_async(partialWorker, tasks)

    try:
        results = p.get(0xFFFF)
    except KeyboardInterrupt:
        print "Received Ctrl-c"
        sys.exit(1)

    pool.close()
    pool.join()          
    pass


if __name__ == "__main__":
    multi()
    

Using lambda functions for sorting in Python

Give a list L = [56, 78, 98, 34] the following one linear will sort the elements in the descending order of their values.
L.sort(lambda x, y: 1 if y > x else -1)

Checking the site-packages

If you want to know where the python site-packages directory is located in your machine do the following.


from distutils.sysconfig import get_python_lib
print(get_python_lib())

pygmentize

Install pygmentize by

sudo easy_install Pygments

quick start (copied from the official Web site)

You can use Pygments from the command line, using the pygmentize script:

$ pygmentize test.py

will highlight the Python file test.py using ANSI escape sequences (a.k.a. terminal colors) and print the result to standard output.

To output HTML, use the -f option:

$ pygmentize -f html -o test.html test.py

to write an HTML-highlighted version of test.py to the file test.html. Note that it will only be a snippet of HTML, if you want a full HTML document, use the "full" option:

$ pygmentize -f html -O full -o test.html test.py

This will produce a full HTML document with included stylesheet.

A style can be selected with -O style=.

If you need a stylesheet for an existing HTML file using Pygments CSS classes, it can be created with:

$ pygmentize -S default -f html > style.css

Python urllib proxies

It is possible to open a url via a proxy server using urllib as shown in the following code.
Here is the documentation.
Here is a list of proxies. To check your IP you can use this site.

import urllib
import re

httpProxy =  "http://85.172.205.91.static.giga-dns.com:3128"
webSite = "http://whatismyipaddress.com/"

if False:
    # no proxy.
    txt = urllib.urlopen(webSite).read()

if True:
    # with proxy.
    proxies = {'http':httpProxy}
    txt = urllib.urlopen(webSite, proxies=proxies).read()
    
IPreg = re.compile('

Aptana Studio

In Apatana studio you can perform block editing. First select some lines of text and then press Command+Option+A. This will enable block highlighting mode. Now use Shift+Arrow Keys to expand or shrink the current selection. When you type in some text, all the highlighted fields will change simultaneously.

Numpy Exceptions

To print all warnings in numpy do the following

import numpy as np
np.seterr(all='print')

This will print all warnings to stderr.

You can also make warnings to exceptions.
An example is shown below. The documentation is here.

import numpy

numpy.seterr(all='raise')

s = 10000.0

try:
    x = numpy.exp(s)
    print x
except:
    print "That is too big!"

Flusing output in Python

When you use "print" statement to output messages to terminal during the execution of a python program, those messages might not get outputted promptly. This is true for time consuming programs that perform a lot of work in between print statements. To promptly flush the text to terminal invoke the python interpreter with "-u" option. For example, if your program is myprogram.py then do the following.

python -u myprogram.py

Alternatively, if you are using sys.stdout.write to print messages to the terminal, then you can do sys.stdout.flush() right after the sys.stdout.write() to do the same. This works even with file I/O.
file.flush() immediately writes the text that is in buffer. This is particularly useful when you are writing log files.

Multiprocessing

In Python GIL (Global Interpreter Lock) prevents from running more than one thread at a time in a process. If you use Jython or IronPython this limitation is not there. But with CPython it is there. Use the multiprocess module (from Python version 2.6 afterwards) to overcome this limitation and have truly multiprocessing capabilities. If you encounter an error during execution and if your processes become zombies then you could either close the terminal (which will kill the zombies as well), or more elegantly use kill -9 on the "parent process id (PPID" of those zombies. To find the process ids use "ps -epw" command. You can also kill a process within the top window by typing "k" and then entering the corresponding process id.

multiprocessing module has similar API as in threading module. It works on Windows, Linux and OS/X.
The following code shows how to create and start several processes.

from multiprocessing import Process
import os

def info(title):
    print title
    print 'module name:', __name__
    print 'parent process:', os.getppid()
    print 'process id:', os.getpid()

def f(name):
    info('function f')
    print 'hello', name
    c = 0
    for i in range(1,100000):
        for j in range(1, 100000):
            if i == j:
                c += 1
    pass
    

    
if __name__ == "__main__":
    info('main line')
    p = Process(target=f, args=('bob',))
    q = Process(target=f, args=('sam',))
    r = Process(target=f, args=('dan',))
    s = Process(target=f, args=('david',))
    q.start()
    p.start()
    r.start()
    s.start()
    p.join()
    q.join()
    r.join()
    s.join()
    pass

Snow Leopard NumPy, SciPy, matplotlib

Installing the above mentioned packages in Snow Leopard can be quite difficult.

An easy solution is to use the script mentioned here
There is a very good description here if you want to do it step by step.

Installing Python 2.6

Install the prerequistes

http://www.talino.org/tutorials/install-python-261-without-trashing-ubuntu/

Download the source from

create a directory
/home/user/myusr/python/2.6

decompress the tar
./configure --prefix=/home/user/myusr/python/2.6

make
make test
make install

Write XML from Python

from xml.sax import saxutils
then use saxutils.escape("raw_string")
will give all exceptions escaped!