Java like thread dumps for the Python process

import threading, sys, traceback

def dumpstacks(signal, frame):
    id2name = dict([(th.ident, th.name) for th in threading.enumerate()])
    code = []
    for threadId, stack in sys._current_frames().items():
        code.append("\n# Thread: %s(%d)" % (id2name.get(threadId,""), threadId))
        for filename, lineno, name, line in traceback.extract_stack(stack):
            code.append('File: "%s", line %d, in %s' % (filename, lineno, name))
            if line:
                code.append("  %s" % (line.strip()))
    print "\n".join(code)

import signal
signal.signal(signal.SIGQUIT, dumpstacks)

List the methods/operations available on an object

Use the dir(object) command to lis the various operations available on the object.

eg.

>>> dir(stringObj)

['__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__getslice__', '__gt__', '__hash__', '__init__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_formatter_field_name_split', '_formatter_parser', 'capitalize', 'center', 'count', 'decode', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'index', 'isalnum', 'isalpha', 'isdigit', 'islower', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']

Start a HTTP server in python

A simple HTTP Server can be started in seconds.

python -m SimpleHTTPServer

The server starts on port 8080 by default which can be changed. I have found this quite handy at times.

For example: To share a complete directory with someone over the Internet, I cd to the directory and start the server. The directory is shared, and I share my IP address. The directory can be viewed and files can be downloaded easily. I can also monitor access requests on the terminal.

original link:
http://www.quora.com/Python-programming-language-1/What-are-some-cool-Python-tricks

TopCoder SRM 567 - Div I Lvl 1

The Square Root Dilemma
-----------------------------------
=> (A*B) is the a perfect square

For a given value of (A), we try to find all possible values of B which make A*B a perfect square.

A = OA * EA,  where EA is the perfect square factor of A and OA is the other factor

So, (A*B) is perfect square when B can be factored as OA*(a perfect square).


def srm567(a,b):
    ctr = 0
    for i in range(1,a+1):
        j = 2
        s = 1
        while (j*j)<= i:
            if i%(j*j) == 0:
                s = j*j
            j += 1

        r = i/s
        #print "i,s:",i,s
        y = 1
        while (y*y*r)<=b:
            ctr+= 1
            y += 1
    return ctr

TopCoder SRM 569 - Div I Lvl 1

We need to be able to find the Operation of each bit, i.e. whether it is OR, AND , XOR.

If you see the truth table for these operations:

A	B	OR	AND	XOR
0	0	0	0	0
0	1	1	0	1
1	0	1	0	1
1	1	1	1	0

You see that the first combination where both bits A/B are '0' all the results are zero, so this combination of bits doesn't help us identify the operation. => If we have bits 0,1,1 available to be inputted at a bit position we can identify the operation.

So, now the problem reduces to finding the no. of plates to be added so that there is a combination of 011 bits available at each position.


 def srm569(plates):
    more_plates_needed = 0
    M = len(plates[0])
    for i in range(0,M):
        bits_needed = ['0','1','1']
        for j in range(0,len(plates)):
            if plates[j][i] in bits_needed:
                bits_needed.remove(plates[j][i])
        more_plates_needed = max(more_plates_needed,len(bits_needed))
    return more_plates_needed


Examples:
=======

srm569(["01010101",
 "10101010"])
Out[34]: 1

srm569(["10010101011",
 "00010101001",
 "00100010111",
 "00101010101",
 "01010111101"])
Out[35]: 1

srm569(["1101001011010",
 "0010000010101",
 "1010101011110",
 "1101010100111",
 "1011111110111"])
Out[36]: 0

Learning python regular expressions

https://developers.google.com/edu/python/regular-expressions

Python Regular Expressions

Regular expressions are a powerful language for matching text patterns. This page gives a basic introduction to regular expressions themselves sufficient for our Python exercises and shows how regular expressions work in Python. The Python "re" module provides regular expression support.

In Python a regular expression search is typically written as:

  match = re.search(pat, str)

The re.search() method takes a regular expression pattern and a string and searches for that pattern within the string. If the search is successful, search() returns a match object or None otherwise. Therefore, the search is usually immediately followed by an if-statement to test if the search succeeded, as shown in the following example which searches for the pattern 'word:' followed by a 3 letter word (details below):

str = 'an example word:cat!!'
match = re.search(r'word:\w\w\w', str)
# If-statement after search() tests if it succeeded
  if match:                      
    print 'found', match.group() ## 'found word:cat'
  else:
    print 'did not find'

The code match = re.search(pat, str) stores the search result in a variable named "match". Then the if-statement tests the match -- if true the search succeeded and match.group() is the matching text (e.g. 'word:cat'). Otherwise if the match is false (None to be more specific), then the search did not succeed, and there is no matching text.

The 'r' at the start of the pattern string designates a python "raw" string which passes through backslashes without change which is very handy for regular expressions (Java needs this feature badly!). I recommend that you always write pattern strings with the 'r' just as a habit.

Basic Patterns

The power of regular expressions is that they can specify patterns, not just fixed characters. Here are the most basic patterns which match single chars:

• a, X, 9, < -- ordinary characters just match themselves exactly. The meta-characters which do not match themselves because they have special meanings are: . ^ $ * + ? { [ ] \ | ( ) (details below)
• . (a period) -- matches any single character except newline '\n'
• \w -- (lowercase w) matches a "word" character: a letter or digit or underbar [a-zA-Z0-9_]. Note that although "word" is the mnemonic for this, it only matches a single word char, not a whole word. \W (upper case W) matches any non-word character.
• \b -- boundary between word and non-word
• \s -- (lowercase s) matches a single whitespace character -- space, newline, return, tab, form [ \n\r\t\f]. \S (upper case S) matches any non-whitespace character.
• \t, \n, \r -- tab, newline, return
• \d -- decimal digit [0-9] (some older regex utilities do not support but \d, but they all support \w and \s)
• ^ = start, $ = end -- match the start or end of the string
• \ -- inhibit the "specialness" of a character. So, for example, use \. to match a period or \\ to match a slash. If you are unsure if a character has special meaning, such as '@', you can put a slash in front of it, \@, to make sure it is treated just as a character.

Basic Examples

Joke: what do you call a pig with three eyes? piiig!

The basic rules of regular expression search for a pattern within a string are:

• The search proceeds through the string from start to end, stopping at the first match found
• All of the pattern must be matched, but not all of the string
• If match = re.search(pat, str) is successful, match is not None and in particular match.group() is the matching text

  ## Search for pattern 'iii' in string 'piiig'.
  ## All of the pattern must match, but it may appear anywhere.
  ## On success, match.group() is matched text.
  match = re.search(r'iii', 'piiig') =>  found, match.group() == "iii"
  match = re.search(r'igs', 'piiig') =>  not found, match == None

  ## . = any char but \n
  match = re.search(r'..g', 'piiig') =>  found, match.group() == "iig"

  ## \d = digit char, \w = word char
  match = re.search(r'\d\d\d', 'p123g') =>  found, match.group() == "123"
  match = re.search(r'\w\w\w', '@@abcd!!') =>  found, match.group() == "abc"

TopCoder SRM 571 - Div I Lvl 1

http://community.topcoder.com/stat?c=problem_statement&pm=12436&rd=15491
The following code prints the first 50 lexicographically sorted values of the records names:


class Test:
def __init__(self):
self.res = []
def f(self,n,x):
if x >n:
return
if len(self.res) < 50:
self.res.append(x)
#print x
else:
return

self.f(n,x*10)
rem = x%10
if rem != 9:
self.f(n,x+1)
def getRes(self,n,x):
self.f(n,x)
print self.res

Sample values printed from the program:
Test().getRes(1010,1)


1
10
100
1000
10000<
1001
10010<
1002
1003
:
1009
101
1010

TopCoder SRM 572 - Div I Lvl 1

http://community.topcoder.com/stat?c=round_overview&er=5&rd=15492


def myfunc(s,k):
    res = 0
    incr = len(s)-k
    for i in range(0,incr):
        # find the first set of alphabets which don't match
        j = i
        t = {}
        ctr = 0
        #print "i:",str(i)
        while j<len(s):
            #print "j:",str(j)
            if s[j] in t:
                t[s[j]] += 1
            else:
                t[s[j]] = 1
            j += incr
            ctr += 1
        mx = max(t.values())
        res += ctr - mx
   
    return res

TopCoder SRM 573 - Div I Lvl 1

Div II Lvl 2 problem: easier version solution
http://community.topcoder.com/stat?c=problem_statement&pm=12471&rd=15493


def myfunc(s):
    our_strength = sum(s[0:3])- min(s[0:3])
    rst = s[3:]
    #print our_strength
    if len(rst) == 0:
        return 1
    # find the rank of our team, start with rank 1 and increment it as you find better ranks
    res = 1
    rst.sort()
    #print rst
    found = True
    while found == True:
        # get the max element
        max = rst[-1]
        rst.remove(max)
        # find the min element which is greater than our_rank
        found = False
        for i in rst:
            if i+max > our_strength:
                rst.remove(i)
                found = True
                res += 1
                break
    return res

http://apps.topcoder.com/wiki/display/tc/SRM+573

Using rsync to analyse hadoop cluster logs at the name node

If you have worked on a hadoop cluster and tried going through the logs on all different cluster nodes you know how painful can it be. This script can be run on the name node and it will copy all the logs for the given hadoop job-id to the current directory of the name node. You will have to change the rsync parameters to suit yourself.

Python script:

import sys
import subprocess
import os

if len(sys.argv) < 2:
        print "Usage: python ",sys.argv[0]," <hadoop-job-id>"
        sys.exit(0)

nodes = ['192.168.112.117', '192.168.156.63', '192.168.152.31', '192.168.112.118', '192.168.156.65', '192.168.156.62' ]

#subprocess.Popen(["ls","-lr"])

for node in nodes:
        subprocess.Popen(["rsync", "-rav", "root@"+node+":/data/hadoop_logs/userlogs/"+sys.argv[1], ".")

#os.system("rsync -rav root@node-0de4bd:/data/hadoop_logs/userlogs/"+sys.argv[1]+" .")

IMP: Rsync requires that you are able to do the password-less access to these nodes from the current node (on which you are running this script.) For help on setting up password-less access check here.

Getting started with Facebook App development on Heroku environment

I am currently working on setting up the work environment for Facebook App development on Heroku server using python as the development language.

The complete tutorial is at: https://devcenter.heroku.com/articles/facebook

On high level the steps involve:

1. Creating a new facebook app and checking the option for Heroku hosting.
2. This creates a new domain on Heroku for your app and installs your app there.
3. Now, you can 'git clone' the code on your local machine, make changes to the code and 'git push heroku master' to push changes back to the world.
4. Need to setup the local environment for testing:
1. This involves setting up the virtualenv for python. http://www.virtualenv.org/en/latest/

virtualenv is a tool to create isolated Python environments. The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python2.7/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.

Or more generally, what if you want to install an application and leave it be? If an application works, any change in its libraries or the versions of those libraries can break the application. Also, what if you can’t install packages into the global site-packages directory? For instance, on a shared host.  

In all these cases, virtualenv can help you. It creates an environment that has its own installation directories, that doesn’t share libraries with other virtualenv environments (and optionally doesn’t access the globally installed libraries either).

-----------------------------
Using Virtualenv:
http://stackoverflow.com/questions/10763440/how-to-install-python3-version-of-package-via-pip

virtualenv -p /usr/bin/python3 py3env
source py3env/bin/activate
pip install package-name

---------------------------------

Now locally test your changes before deploying on the server.

Next steps:

- https://devcenter.heroku.com/articles/python

- To be able to use Django here - https://devcenter.heroku.com/articles/django

- To be able to use a DB backend.

Understanding open source architectures I

https://github.com/foursquare
https://code.google.com/p/foursquared/wiki/Design
https://code.google.com/p/foursquared/source/browse
https://sites.google.com/site/foursquareddev/ux/venueactivity

https://github.com/foursquare/Android-PullToRefresh
https://github.com/foursquare/SlidingMenu

Eclipse:
http://www.aosabook.org/en/eclipse.html

Graphite:
http://www.aosabook.org/en/graphite.html

http://www.tornadoweb.org/

http://blog.reddit.com/2008/06/reddit-goes-open-source.html

https://code.google.com/p/guava-libraries/wiki/GuavaExplained

SPOJ problem ID [4]: Transform the Expression

Following is my first solution to the SPOJ problem 4: http://www.spoj.com/problems/ONP/

Python solution:


import sys
import string

operators = ['+', '-', '*', '/', '^']

n = int(raw_input())

def isoperator(char):
        if char in operators:
                return True
        else:
                return False

def process_line(line):
        stck = []
        output = ""
        for i in range(0,len(line)):
                if line[i].isalpha():
                        output += line[i]
                else:
                        if isoperator(line[i]):
                                while len(stck) >0 and stck[-1] in operators and operators.index(line[i]) < operators.index(stck[-1]): #lower precendence of new operator, so pop the higher precendence from stck
                                        output+=stck.pop()
                                stck.append(line[i])
                        elif line[i] == ')': #is a closing brace
                                while stck[-1] != '(' :
                                        output+=stck.pop()
                                stck.pop()
                        else : # opening brace, and put it to stck
                                stck.append(line[i])
                #print "stck     :" + str(stck)
                #print "output   :" + output
        while len(stck) > 0:
                output+=stck.pop()
        print output

while n > 0:
        line = raw_input()
        n-=1
        process_line(line)

Learning Python and Django

Print python module path


>>> import os
>>> print os.__file__
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.pyc


>>> import django
>>> print django.__file__
/Library/Python/2.7/site-packages/django/__init__.pyc

Creating my first Django project
-------------------------------------------
1. django-admin.py startproject mysite


This creates the following structure:

mysite/
    manage.py
    mysite/
        __init__.py
        settings.py
        urls.py
        wsgi.py

These files are:

• The outer mysite/ directory is just a container for your project. Its name doesn't matter to Django; you can rename it to anything you like.
• manage.py: A command-line utility that lets you interact with this Django project in various ways. You can read all the details about manage.py in django-admin.py and manage.py.
• The inner mysite/ directory is the actual Python package for your project. Its name is the Python package name you'll need to use to import anything inside it (e.g. import mysite.settings).
• mysite/__init__.py: An empty file that tells Python that this directory should be considered a Python package. (Read more about packages in the official Python docs if you're a Python beginner.)
• mysite/settings.py: Settings/configuration for this Django project. Django settings will tell you all about how settings work.
• mysite/urls.py: The URL declarations for this Django project; a "table of contents" of your Django-powered site. You can read more about URLs in URL dispatcher.
• mysite/wsgi.py: An entry-point for WSGI-compatible webservers to serve your project. See How to deploy with WSGI for more details.

The development server

Let's verify this worked. Change into the outer mysite directory, if you haven't already, and run the commandpython manage.py runserver. 

2. Creating an app for polls under this project

python manage.py startapp polls

That'll create a directory polls, which is laid out like this:

polls/
    __init__.py
    models.py
    tests.py
    views.py

This directory structure will house the poll application.

Edit the polls/models.py file.

But first we need to tell our project that the polls app is installed.

Edit the settings.py file again, and change the INSTALLED_APPS setting to include the string 'polls'. 

Now Django knows to include the polls app. Let's run another command:

python manage.py sql polls

The sql command doesn't actually run the SQL in your database - it just prints it to the screen so that you can see what SQL Django thinks is required. If you wanted to, you could copy and paste this SQL into your database prompt. However, as we will see shortly, Django provides an easier way of committing the SQL to the database.

If you're interested, also run the following commands:

• python manage.py validate -- Checks for any errors in the construction of your models.
• python manage.py sqlcustom polls -- Outputs any custom SQL statements (such as table modifications or constraints) that are defined for the application.
• python manage.py sqlclear polls -- Outputs the necessary DROP TABLE statements for this app, according to which tables already exist in your database (if any).
• python manage.py sqlindexes polls -- Outputs the CREATE INDEX statements for this app.
• python manage.py sqlall polls -- A combination of all the SQL from the sql, sqlcustom, and sqlindexes commands.

Looking at the output of those commands can help you understand what's actually happening under the hood.

-

Questions:
-------------
What are the common practices used by python based websites to improve performance ?