Using wget to download an ASP site

You can download an ASPX site, which asks for username/password for log in as follows:

First provide the username/password to the login page and save the cookie file.

wget --mirror -r \
--user-agent="" \
--keep-session-cookies --save-cookies cookies.txt \
--post-data '__VIEWSTATE=%2FwEPDwULLTE3MDc5MjQzOTdkZIP%2Fxc105yfz2jGFj4Nd3tPvrEeNara43fIRI5oAW%2Bwv&__EVENTVALIDATION=%2FwEWBAKisoyCAwLB2tiHDgK1qbSRCwL2k8O9DUQa5owMFDWzFnBoIDusNkznjB65a6zRyNETOEZfBM1o&txtUser=admin&txtPassword=admin&login_btn=Sign+In' \
-E -k -p http://www.xyz.com/Login.aspx

Then you can access other pages in the next step using the above generated cookie files.

wget --mirror -r \
--user-agent="" \
--keep-session-cookies --load-cookies cookies.txt \
-E -k -p http://www.xyz.com/Index.aspx

For details of the options, refer to the WGET manual i.e "man wget" :)

vimdiff tips and tricks - how to copy between two screens

do (diff obtain) and dp (diff put) is what you need. Here is a small list of other helpful commands in this context.

]c               - advance to the next block with differences
[c               - reverse search for the previous block with differences
do (diff obtain) - bring changes from the other file to the current file
dp (diff put)    - send changes from the current file to the other file
zo               - unfold/unhide text
zc               - refold/rehide text
zr               - unfold both files completely

Note: Both do and dp work if you are on a block or just one line under a block.

:diffupdate will re-scan the files for changes

Original source:http://unix.stackexchange.com/questions/52754/whats-the-recommended-way-of-copying-changes-with-vimdiff

Debugging / Tracing bash shell scripts

Source: http://www.linuxcommand.org/wss0100.php

Watching your script run

It is possible to have bash show you what it is doing when you run your script. To do this, add a "-x" to the first line of your script, like this:

#!/bin/bash -x
       

Now, when you run your script, bash will display each line (with substitutions performed) as it executes it. This technique is called tracing. Here is what it looks like:

[me@linuxbox me]$ ./trouble.bash
+ number=1
+ '[' 1 = 1 ']'
+ echo 'Number equals 1'
Number equals 1

Alternately, you can use the set command within your script to turn tracing on and off. Use set -x to turn tracing on and set +x to turn tracing off. For example.:

#!/bin/bash

number=1

set -x
if [ $number = "1" ]; then
    echo "Number equals 1"
else
    echo "Number does not equal 1"
fi
set +x
       

Vim tips and tricks

Cut and paste:

1. Position the cursor where you want to begin cutting.
2. Press v (or upper case V if you want to cut whole lines).
3. Move the cursor to the end of what you want to cut.
4. Press d.
5. Move to where you would like to paste.
6. Press P to paste before the cursor, or p to paste after.

Copy and paste can be performed with the same steps, only pressing y instead of d in step 4.

Using rsync to analyse hadoop cluster logs at the name node

If you have worked on a hadoop cluster and tried going through the logs on all different cluster nodes you know how painful can it be. This script can be run on the name node and it will copy all the logs for the given hadoop job-id to the current directory of the name node. You will have to change the rsync parameters to suit yourself.

Python script:

import sys
import subprocess
import os

if len(sys.argv) < 2:
        print "Usage: python ",sys.argv[0]," <hadoop-job-id>"
        sys.exit(0)

nodes = ['192.168.112.117', '192.168.156.63', '192.168.152.31', '192.168.112.118', '192.168.156.65', '192.168.156.62' ]

#subprocess.Popen(["ls","-lr"])

for node in nodes:
        subprocess.Popen(["rsync", "-rav", "root@"+node+":/data/hadoop_logs/userlogs/"+sys.argv[1], ".")

#os.system("rsync -rav root@node-0de4bd:/data/hadoop_logs/userlogs/"+sys.argv[1]+" .")

IMP: Rsync requires that you are able to do the password-less access to these nodes from the current node (on which you are running this script.) For help on setting up password-less access check here.

Print to dynamic files from AWK cmd

To print the output of AWK to a dynamic file:

head  236_236_raw |  awk -F';' '{  print $7,"\t",$6 > ("_"$2"_"$3".txt")}'

If the number of files open are large, you will get the following error:

awk: * makes too many open files

 input record number 203, file

 source line number 1

To overcome the above problem, you can close the files like this:

head  236_236_raw |  awk -F';' '{  print $7,"\t",$6 >> ("_"$2"_"$3".txt"); close("_"$2"_"$3".txt")}'

The important point to note is that now we changed the redirection operator to ">>" from ">". Makes sense ? :)

Getting started with Facebook App development on Heroku environment

I am currently working on setting up the work environment for Facebook App development on Heroku server using python as the development language.

The complete tutorial is at: https://devcenter.heroku.com/articles/facebook

On high level the steps involve:

1. Creating a new facebook app and checking the option for Heroku hosting.
2. This creates a new domain on Heroku for your app and installs your app there.
3. Now, you can 'git clone' the code on your local machine, make changes to the code and 'git push heroku master' to push changes back to the world.
4. Need to setup the local environment for testing:
1. This involves setting up the virtualenv for python. http://www.virtualenv.org/en/latest/

virtualenv is a tool to create isolated Python environments. The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python2.7/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.

Or more generally, what if you want to install an application and leave it be? If an application works, any change in its libraries or the versions of those libraries can break the application. Also, what if you can’t install packages into the global site-packages directory? For instance, on a shared host.  

In all these cases, virtualenv can help you. It creates an environment that has its own installation directories, that doesn’t share libraries with other virtualenv environments (and optionally doesn’t access the globally installed libraries either).

-----------------------------
Using Virtualenv:
http://stackoverflow.com/questions/10763440/how-to-install-python3-version-of-package-via-pip

virtualenv -p /usr/bin/python3 py3env
source py3env/bin/activate
pip install package-name

---------------------------------

Now locally test your changes before deploying on the server.

Next steps:

- https://devcenter.heroku.com/articles/python

- To be able to use Django here - https://devcenter.heroku.com/articles/django

- To be able to use a DB backend.

Using multi-dimensional associative arrays in AWK

Examples:

cat /tmp/33 | awk -F'[\\^\\[\\]]' '{c[$5,$6]+=1}END{for( i in c) {split(i,sep,SUBSEP); print sep[1],sep[2], c[i] ;}}' 

Reading:

http://www.chemie.fu-berlin.de/chemnet/use/info/gawk/gawk_12.html

How to find the process which has the file handle open ?

A lot of times you need to find which Process (PID) has the handle to a particular file.
In linux doing this easy using the following command:

$ ls -ld1 /proc/*/fd/* | grep <file-name-filter>

Enjoy !!

Using awk to find the sum of a CSV file after group by on a particular column

http://www.theunixschool.com/2012/06/awk-10-examples-to-group-data-in-csv-or.html

I have a CSV file like :


65523 , 100
65522 , 900
65522 , 1800
65522 , 100
65522 , 100
65521 , 500
65521 , 200
65521 , 200

I need to find the sum of the 2nd column based on the grouping by the 1st column, so that the output looks something like:


65523 , 100
65522 , 2900
65521 , 900

SOLUTION:

This can be easily achieved using a single line awk script:

awk -F"," '{a[$1]+=$2;}END{for (i in a)print i, a[i];}' file

Awesome isn't it !! :)

Sorting a CSV file based on a particular column list

http://stackoverflow.com/questions/9471101/sort-csv-file-by-column-priority-using-the-sort-command-unix

sort --field-separator=';' --key=2,1

sort -nr -t',' -k3

To sort based on multiple columns use the syntax:

sort --key=1,1 --key=2,2r --key=3,3 --key=4,4r
sort -k1,1 -k2,2r -k3,3 -k4,4r

as in the following transcript:

pax$ echo '5 3 2 9
3 4 1 7
5 2 3 1
6 1 3 6
1 2 4 5
3 1 2 3
5 2 2 3' | sort --key=1,1 --key=2,2r --key=3,3 --key=4,4r

1 2 4 5
3 4 1 7
3 1 2 3
5 3 2 9
5 2 2 3
5 2 3 1
6 1 3 6

Remember to provide the -n option if you want them treated as proper numbers (variable length), such as:

sort -n -k1,1 -k2,2r -k3,3 -k4,4r




Learning to use Screen command in Unix

http://kb.iu.edu/data/acuy.html

If your local computer crashes, or you are connected via a modem and lose the connection, the processes or login sessions you establish through screen don't go away. You can resume your screen sessions with the following command: screen -r

screen                                ==>    Start a new screen
(Ctrl+A) & C                    ==>    Start a new screen sub-window
(Ctrl+A) & K                    ==>    Kill the current sub-window

(Ctrl+A) & (Shift + ")       ==>    Show the list of screens running on the system

screen -r                            ==>    restore to the old screens
screen -ls                           ==>    list of running screens
(Ctrl+A) & (Shift + A)      ==>   rename the current screen

Using GDB to debug in linux

To debug a process: "./a.out arg1 arg2", run:

$ gdb ./a.out
(gdb) run arg1 arg2


Reading: http://cs.baylor.edu/~donahoo/tools/gdb/tutorial.html

Understanding the I/O redirection in linux

STDIN is FD 0, while STDOUT is FD 1, and STDERR is FD 2.

To redirect the errors to the output stream use this: 2>&1

Here '&' tells that it is a FD(file descriptor) and not an ordinary file named '1'.

There are lots of redirection symbols that you can use, and here are some of them:

< file	means open a file for reading and associate with STDIN.
<< token	Means use the current input stream as STDIN for the program until token is seen. We will ignore this one until we get to scripting.
> file	means open a file for writing and truncate it and associate it with STDOUT.
>> file	means open a file for writing and seek to the end and associate it with STDOUT. This is how you append to a file using a redirect.
n>&m	means redirect FD n to the same places as FD m. Eg, 2>&1 means send STDERR to the same place that STDOUT is going to.

Reading: http://www.linuxsa.org.au/tips/io-redirection.html

Awk - with loops and regex

Awk built-in variables:

FNR         The input record number in the current input file.
NF          The number of fields in the current input record.
NR          The total number of input records seen so far.

// an awk command to match the regex "UST" in the line
cat matrace_20120430234915.dat | awk -F'[,]' '{print FNR; i=1;for(i=1;i<(NF);i++) { print i,$i, /UST/;} }'


// an awk command which finds the column matching the regular expression of date in format "yyyy-mm-dd" and splits the date and adds 1 to the date and then prints the same.
awk -F'[,]' '{print FNR; i=1;for(i=1;i<(NF);i++) { print i,$i; if(match($i,/(....)\-(..)\-(..)/)) { split($i,a,"-"); printf("%d-%d-%d",a[1],a[2],(a[3]+1));};} }'

// awk using sprintf
awk -F'[,]' '{x=sprintf("%s",$1); i=1; for(i=2;i<(NF);i++) {  if(match($i,/(....)\-(..)\-(..)/)) { split($i,a,"-"); x=sprintf("%s,%s-%s-%d",x,a[1],a[2],(a[3]+1));} else { x=sprintf("%s,%s",x,$i); } } x=sprintf("%s,%s",x,$(NF)); print x;}'

Using Strace in linux

The following article explains the usage of strace command for troubleshooting in linux:
http://www.hokstad.com/5-simple-ways-to-troubleshoot-using-strace.html

strace is a tool for tracing system calls and signals. It intercepts and records the system calls made by a running process. strace can print a record of each system call, its arguments, and its return value. You can use strace on programs for which you do not have the source since using strace does not require recompilation. It is often useful in instances where a program freezes or otherwise fails to work and offers few clues as to the problem.


Strace Output: Each line starts with a system call name, is followed by its arguments in parenthesis and then has the return value at the end of the line. Errors (which typically have a return value of -1) have the symbolic error name (such as ENOENT in the first line in the example above) as well as a more informative error string appended.

SCP and SSH without password

To setup the password-less logging from one box 'Source' to another box 'Dest'.
You need to generate the 'Source' user's public/private key pair, and then append the public key of this user to 'Dest' user's ".ssh/authorized_keys" file.
Important thing to note is that destination and source users can be different. However, the ".ssh" folder inside 'Dest' user's home directory needs to have unix permissions '700' for this to work (Because otherwise anyone can write their public key into this folder and gain access to the destination user's account).

The following article explains the setup of password less ssh or scp setup between two systems.
It provides the troubleshooting for common problems as well.

http://homepage.mac.com/kelleherk/iblog/C1901548470/E20061128145420/index.html

Find all files recursively in the current directory and process them one by one

Finding all files in the current directory, and execute a command on each file:


find ./ -mtime +1 -exec rm -vf {} \;


the command executed on each file is in italics in the above line, and '{}' is replaced by the file name during the command execution.

Check the memory used by a unix process

To see the memory used by the process, ps can be used with flags -AH
ps -AH v | grep 3408
where 3408 is the PID of process.

Check the column RSS (resident set size). It is the non-swapped physical memory that a task has used (in kiloBytes). (alias rssize) rsz).

Setting up HTTP proxy in the linux shell

You can set the http_proxy environment variable using the command specific to your shell (e.g. set or export). 

To make this change persistent, add the command to the appropriate profile file for the shell. For example, in bash, add a line like the following to your .bash_profile or .bashrc file:

http_proxy=http://username:password@hostname:port;  

export http_proxy