Thursday 6 November 2014

Using Memory Mapped files in Java

When two processes map the same file in memory, the memory that one process writes is seen by another process, so memory mapped files can be used as an interprocess communication mechanism. We can say that memory-mapped files offer the same interprocess communication services as shared memory with the addition of filesystem persistence. However, as the operating system has to synchronize the file contents with the memory contents, memory-mapped files are not as fast as shared memory.


/*--------------------------------------------------------------
Class MemoryMapWriter - creates mmap file
---------------------------------------------------------------*/

import java.io.File;
import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;

public class MemoryMapWriter {

public static void main(String[] args) throws Exception {
File f = new File("/tmp/mapped.txt");
f.delete();

FileChannel fc = new RandomAccessFile(f, "rw").getChannel();

int start = 0;
long counter = 1;
long HUNDREDK = 100000;
long startT = System.currentTimeMillis();
long noOfMessage = HUNDREDK * 100 * 10;

long bufferSize = 8 * noOfMessage;
MappedByteBuffer mem = fc.map(FileChannel.MapMode.READ_WRITE, 0,
bufferSize);

for (;;) {
if (!mem.hasRemaining()) {
start += mem.position();
mem = fc.map(FileChannel.MapMode.READ_WRITE, start, bufferSize);
}
mem.putLong(counter);
counter++;
if (counter > noOfMessage)
break;
}
long endT = System.currentTimeMillis();
long tot = endT - startT;
System.out.println(String.format("No Of Message %s , Time(ms) %s ",
noOfMessage, tot));

fc.close();
unmap(fc, mem);
}

private static void unmap(FileChannel fc, MappedByteBuffer bb)
throws Exception {
Class<?> fcClass = fc.getClass();
java.lang.reflect.Method unmapMethod = fcClass.getDeclaredMethod(
"unmap", new Class[] { java.nio.MappedByteBuffer.class });
unmapMethod.setAccessible(true);
unmapMethod.invoke(null, new Object[] { bb });
}

}

/*--------------------------------------------------------------
Class MemoryMapReader - reads mmap file
---------------------------------------------------------------*/
package org.g;

import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;

public class MemoryMapReader {

public static void main(String[] args) throws FileNotFoundException,
IOException, InterruptedException {

FileChannel fc = new RandomAccessFile(new File("/tmp/mapped.txt"), "rw")
.getChannel();

// long bufferSize=8*10000;
MappedByteBuffer mem = fc.map(FileChannel.MapMode.READ_ONLY, 0,
fc.size());
long oldSize = fc.size();

long currentPos = 0;
long xx = currentPos;

long startTime = System.currentTimeMillis();
long lastValue = -1;

int exporterId = 8;

mem.position(exporterId * 8);
long res = mem.getLong() - 1;
if (res != exporterId) {
System.out.println("this should not happen !! ");
}

System.out.println("Resetting pos ...");
// reset position again
mem.position(0);

for (;;) {

while (mem.hasRemaining()) {
lastValue = mem.getLong();
currentPos += 8;
}
if (currentPos < oldSize) {

xx = xx + mem.position();
System.out.println("this shuold never occur !!! ");
// mem = fc.map(FileChannel.MapMode.READ_ONLY,xx, bufferSize);
continue;
} else {
long end = System.currentTimeMillis();
long tot = end - startTime;
System.out.println(String.format(
"Last Value Read %s , Time(ms) %s ", lastValue, tot));
break;
}

}

}

}

More Reading:
--------------------

To unmap the memory mapped byte buffer - http://lotusandjava.blogspot.in/2012/02/how-to-unmap-mappedbytebuffer.html
http://stackoverflow.com/questions/8462200/examples-of-forcing-freeing-of-native-memory-direct-bytebuffer-has-allocated-us


Measuring system time with micro second precision in c++

#include <sys/time.h>
#include <stdio.h>
#include <unistd.h>

struct timeval start, end;
long mtime, secs, usecs;

int main(int argc, char * argv[]){

gettimeofday(&start, NULL);

sleep(10);

gettimeofday(&end, NULL);
secs  = end.tv_sec  - start.tv_sec;
usecs = end.tv_usec - start.tv_usec;
mtime = ((secs) *1000* 1000 + usecs) ;
printf("Elapsed time: %ld usecs\n", mtime);
        return 0;
}


OUTPUT:
-------------
> ./a.out
Elapsed time: 10,003,866 usecs

JNI - Calling native functions in Java

This example has been taken from this link -  http://www.science.uva.nl/ict/ossdocs/java/tutorial/native1.1/stepbystep/index.html

Hello world example for native functions :

1.) Compile your Java code using javac.
/*-------------------------------------
Class HelloWorld
--------------------------------------*/
class HelloWorld {
    public native void displayHelloWorld();

    // Declare a native method average() that receives two ints and return a double containing the average
    public native double average(int n1, int n2);

    static {
        //System.loadLibrary("hello");
        // You can specify the full path to the lib file
    System.load("/Users/xyz/Documents/workspace/JNITest/src/libhello.so");
    }
}


/*-------------------------------------
Class JNITest - with main()
--------------------------------------*/
public class JNITest {
public static void main(String[] args) throws InterruptedException {
HelloWorld hw = new HelloWorld();
hw.displayHelloWorld();

Thread.sleep(1000);
System.out.println("Exiting ... "+hw.average(2, 100));
}

}

2.) Generate the c++ header file using the javah command

> javah HelloWorld

3.) Write the c++ native function implementation

#include <jni.h>
#include "HelloWorld.h"
#include <stdio.h>

JNIEXPORT jdouble JNICALL Java_HelloWorld_average(JNIEnv * jenv, jobject jo, jint n1 , jint n2){

return ((jdouble)n1 + n2) / 2.0;
}

JNIEXPORT void JNICALL
Java_HelloWorld_displayHelloWorld(JNIEnv *env, jobject obj)
{
  printf("Hello world!\n");
  return;
}

4.) Compile your c++ code and create a shared library

Note: If you have problems importing <jni.h> on linux box ensure that the include path contains your java's include directories. For example:
g++ -O3 -shared -fPIC mibc.cpp -I /usr/java/default/include/ -I /usr/java/default/include/linux/ -o libhello.so -lrt

5.) Run your java program
Place your shared library in the correct path as specified in the java System.load() call. And run the java program using "java JNITest"

More Reading:
To use different data types for native calls -  https://www3.ntu.edu.sg/home/ehchua/programming/java/JavaNativeInterface.html
JNI performance and overheads -
http://thinkingandcomputing.com/2014/03/30/eliminating-jni-overhead/
http://a-hackers-craic.blogspot.in/2012/03/jni-overheads.html
http://stackoverflow.com/questions/6175209/low-latency-ipc-between-c-and-java
http://stackoverflow.com/questions/7699020/what-makes-jni-calls-slow
http://www.ibm.com/developerworks/java/library/j-jni/
http://normanmaurer.me/blog/2014/01/07/JNI-Performance-Welcome-to-the-dark-side/
http://www.javaworld.com/article/2077554/learn-java/java-tip-54--returning-data-in-reference-arguments-via-jni.html
http://stackoverflow.com/questions/1632367/passing-pointers-between-c-and-java-through-jni

Tuesday 7 October 2014

Find smallest +ve no. missing from the array

Original link:http://www.careercup.com/question?id=12708671

You are given an unsorted array with both positive and negative elements. You have to find the smallest positive number missing from the array in O(n) time using constant extra space. 
Eg: 
Input = {2, 3, 7, 6, 8, -1, -10, 15} 
Output = 1 

Input = { 2, 3, -7, 6, 8, 1, -10, 15 } 
Output = 4

Solution:
1.) Partition the array into the values smaller than zero and +ve integers using the Quicksort partition method. This can be done in O(n) time and in the same array.
2.) Now you the index of element '0' or first +ve number, say this is idxZ.
3.) Traverse starting from idxZ and set the sign bit of the value at position (idxZ + A[idxZ]) to 1. You do this only in case the value of position lies in the bounds of array.
4.) Now starting at 'idxZ' find the first position where the sign bit is UNSET. This gives you the number you are looking for.

Friday 13 June 2014

Read an HDFS file functional way in scala

This example reads an HDFS file in scala in a functional manner. We use Stream class to read data lazily when required.

val path = new Path("/data/abc.csv")
val conf = new Configuration()
val fileSystem = FileSystem.get(conf)
val stream = fileSystem.open(path)

// Important to make this def, bcoz if we make it val the memory might bloat up as it keeps the old
// values in the stream as well
def readLines = Stream.cons(stream.readLine, Stream.continually( stream.readLine))

readLines.takeWhile(_ != null).foreach(line => println(line))

Tuesday 4 March 2014

Learning Scala

Reading : http://twitter.github.io/scala_school/

def f1(x : Int) : Int = x+1

def fact( z: Int) : Int = z* fact(z-1)

Anonymous functions
-------------------
scala> val addOne = (x: Int) => x + 1
addOne: (Int) => Int = <function1>

scala> addOne(1)
res4: Int = 2

def timesTwo(i: Int): Int = {
  println("hello world")
  i * 2
}

Partial application
-------------------

scala> def adder(m: Int, n: Int) = m + n
adder: (m: Int,n: Int)Int

scala> val add2 = adder(2, _:Int)
add2: (Int) => Int = <function1>

scala> add2(3)
res50: Int = 5

Curried functions
-----------------

scala> def multiply(m: Int)(n: Int): Int = m * n
multiply: (m: Int)(n: Int)Int


You can take any function of multiple arguments and curry it.

scala> val curriedAdd = (adder _).curried
curriedAdd: Int => (Int => Int) = <function1>

scala> val addTwo = curriedAdd(2)
addTwo: Int => Int = <function1>

scala> addTwo(4)
res22: Int = 6


Variable length arguments
-------------------------

def capitalizeAll(args: String*) = {
  args.map { arg =>
    arg.capitalize
  }
}

scala> capitalizeAll("rarity", "applejack")
res2: Seq[String] = ArrayBuffer(Rarity, Applejack)


Classes
-------

scala> class Calculator {
     |   val brand: String = "HP"
     |   def add(m: Int, n: Int): Int = m + n
     | }
defined class Calculator

scala> val calc = new Calculator
calc: Calculator = Calculator@e75a11



http://jim-mcbeath.blogspot.in/2009/05/scala-functions-vs-methods.html