Kalanand's September 2014 Log

   August 2014   
September 2014
   October 2014   

September 1st

Books on network security

Here are a few good introductory books on network security to get started.


Download from: https://www.wireshark.org/download.html
User's guide: https://www.wireshark.org/docs/wsug_html_chunked/
Documentation: https://www.wireshark.org/docs/

Useful tips:

September 2nd

Bit coin mining

The original bitcoin paper (by Satoshi Nakamoto): https://bitcoin.org/bitcoin.pdf

Some introductory articles about bitcoin: Bitcoin technicals: Bitcoin mining hardware comparisons: Bitcoin mining software comparisons: Comparison of bitcoin mining pools: Stratum protocol introduction: Stratum protocol documentation: Stratum protocol public code examples:

September 4th

Testing pcap

pcap go 0
add time 360000
save http session file test.dat
save session table test-session.dat
set json  ## if I want to save the standard json files 
To mask a source IP:
set internal addr-mask <src_ip_internal_address>
For example
set internal addr-mask

September 5th

Running GDB with debug options

To start
gdb --args <executable_with_options>
Set a break point
br <file>:<line_number>
Go to next line
n # note: pressing the return key will run the previous command
Jump inside a function call
Print some program variable within the scope
p <variable>

September 8th

Emacs: To display line number on the left side I needed to add the following line in my .emacs config
       ;; Display line number on the left side
       (global-linum-mode t)
I also enabled 'Ido' setting to switch between the buffers and/or files:
       ;; Enable ido mode
       (require 'ido)
       (ido-mode t)
       (setq ido-enable-flex-matching t) ;; enable fuzzy matching
To switch between buffers, press "C-x b". To open a new file, press "C-x C-f". To comment a block of code use "alt;" after selecting the block.

September 10th

Scala learning resources: Spark programming guide:
Running spark:
Let's start the scala shell (First I needed to setup some env variables in .profile to make it work)
Now, let's open one of the hadoop files ("hadoop fs -ls")
val data = sc.textFile("myfile.json")
Check how many elements we have in the data and parse it
val pdata = data.map( x => new Parsed(x) )
Apply filter to the above parsed data
val out = pdata.filter( x=> isInternal(x.src_ip,"blah") && !isInternal(x.dst_ip,"blah" ))
Get an array of the first 10 elements of the filtered data
out.take(10).foreach(x => println(x) )
To filter the TCP port (number 8663)
val tcpdata = pdata.filter( x=> x.dst_port == 8663)
To filter the Stratum port (number 8332)
val strdata = pdata.filter( x=> x.dst_port == 8332)
Sometimes we may need to convert hex characters to ascii for easy reading

September 11th

Copy local file to hadoop hdfs:
hadoop fs -copyFromLocal test.json /user/kalanand/.
Run pyspark on the local cluster:
pyspark --master local
Convert pcapng files to pcap:
tcpdump -r <pcapng_file>  -w <pcap_file>

Some emacs tips!

To comment a block of code, first select the code then type
alt  ;
(both keys together).

To query replace
alt %

September 14th

My first scala project: hello world!

Just the first baby steps, nothing complicated :-)
package example

object hello {
  def main(args: Array[String]) =
  if (args == null) println("hello world!")
  else println("hello world!" + args.toList)      //> main: (args: Array[String])Unit
  main(null)                                      //> hello world!
  main(Array("How",  "are",  "you ?"))            //> hello world!List(How, are, you ?)
And the first worksheet:
package greeter

object first_worksheet {
  println("Welcome to the Scala worksheet")       //> Welcome to the Scala worksheet

 val x = 5                                        //> x  : Int = 5
  def increase(i: Int) = i + 1                    //> increase: (i: Int)Int
  increase(x)                                     //> res0: Int = 6

Eclipse editor tips

To indent a block of code, first select the block then type
Command  Shift  F
(all three keys together). This is sorthand for
editor → Source → Format

in the Eclipse menu.

To save code type
Command  S

September 15th

http server return code meanings

200: OK  # Very frequent 
201: Created
202: Accepted
204: No Content 
205: Reset Content
206: Partial Content

300: Multiple Choices
301: Moved Permanently (redirect)
302: Found (temporary redirect)
303: See Other (temporary redirect)
304: Not Modified  # Very frequent 
305: Use Proxy
307: Temporary Redirect

400: Bad Request
401: Unauthorized
403: Forbidden
404: Not Found
405: Method Not Allowed
406: Not Acceptable
407: Proxy Authentication Required
408: Request Timeout
409: Conflict
410: Gone
415: Unsupported Media Type

500: Internal Server Error
502: Bad Gateway
503: Service Unavailable
504: Gateway Timeout


In the Domain Name System (DNS) hierarchy, a second-level domain (SLD) is a domain that is directly below a top-level domain (TLD).
For example, in example.com, the top-level domain is .com (the "dot" and the word that follows it) and the second-level domain is example.
Second-level domains commonly refer to the organization that registered the domain name with a domain name registrar.
For details see:

A list of valid TLD names (as a text file) can be found at the IANA website

A good description of the internet addressing scheme can be found at

September 18th


Installing on Linux machines
sudo apt-get install python-pip 
sudo pip install couchdb
Often times one stores data notes in such a way that each line is a new JSON. So, in order to parse the information contained in the note one needs to do
import json

data = []
with open('file') as f:
    for line in f:

September 22nd

JIRA and Stash

Both developed by Atlassian. JIRA is the tracker for team planning and task management. Stash is for code tracking, review, and management.
More details at
Although they are two separate products, a JIRA task number can be attached to one or more code branches in Stash, and vice versa.
Atlassian also has a brand new HipChat for group text and video chats.

Colloquy is an IRC chat client for Mac. For details and download, see http://colloquy.info.

September 23rd

Scala week-1: simple algebraic stuff like root finding using Newton's method

package week1

object session {
  1 + 3                                           //> res0: Int(4) = 4
  def abs(x: Double) = if (x < 0) -x else x       //> abs: (x: Double)Double

  def sqrt(x: Double) = {
    def sqrtIter(guess: Double): Double =
      if (isGoodEnough(guess)) guess
      else sqrtIter(improve(guess))

    def isGoodEnough(guess: Double) = abs(guess - x / guess) < 0.001

    def improve(guess: Double) = (guess + x / guess) / 2

  }                                               //> sqrt: (x: Double)Double

  sqrt(2)                                         //> res1: Double = 1.4142156862745097
  sqrt(4)                                         //> res2: Double = 2.0000000929222947
  sqrt(0.001)                                     //> res3: Double = 0.031642015868650786
  sqrt(0.1e-20)                                   //> res4: Double = 9.765625000000002E-4
  sqrt(1.0e20)                                    //> res5: Double = 1.0E10
  sqrt(1.0e50)                                    //> res6: Double = 1.0E25

  def gcd(a: Int, b: Int): Int =
    if (b == 0) a else gcd(b, a % b)              //> gcd: (a: Int, b: Int)Int

  gcd(14, 21)                                     //> res7: Int = 7

  def factorial(n: Int): Int =
    if (n == 0) 1 else n * factorial(n - 1)       //> factorial: (n: Int)Int

  factorial(4)                                    //> res8: Int = 24

September 24th

Git rehash

Clone a new package from source
git clone <source_code_location_with_path.git>
Checkout an existing branch from the source repository
git checkout <some_existing_branch_I_need_to_checkout>
To see which branch I am currently working in
git branch
To fetch branches and/or tags from the source repository along with the objects necessary to complete their histories
git fetch
To incorporates changes from the remote repository into the current branch
git pull
To check modification/commit status of the tracked files only
git status -uno 
To track a new file or to prepare to commit a modified file
git add <file>
To commit (make sure to include JIRA issue number in the comment, otherwise the remote repository will decline push request)
git commit -m <comment>
To push this commit to the remote repository
git push
To create a totally new branch and switch to this new branch in one step
git checkout -b <new_branch_name>
To create a totally new branch while still remaining on the current branch
git branch <new_branch_name>
To merge some other branch ("other_branch") into the current branch ("this_branch")
git fetch
git checkout <that_branch>
git pull
git checkout <this_branch>
git pull
git merge <that_branch> 

September 26th

Scala week-1 continued: map-reduce, inner product, ...

package week1
object exercise {
  def factorial(n: Int): Int = {
    def loop(acc: Int, n: Int): Int =
      if (n == 0) acc
      else loop(acc * n, n - 1)
    loop(1, n)
  }                                               //> factorial: (n: Int)Int
  factorial(4)                                    //> res0: Int = 24

  def product(f: Int => Int)(a: Int, b: Int): Int =
    if (a > b) 1 else f(a) * product(f)(a + 1, b) //> product: (f: Int => Int)(a: Int, b: Int)Int

  product(x => x * x)(3, 4)                       //> res1: Int = 144

  def fact(n: Int) = product(x => x)(1, n)        //> fact: (n: Int)Int

  fact(5)                                         //> res2: Int = 120

  def mapReduce(f: Int => Int, combine: (Int, Int) => Int, zero: Int)(a: Int, b: Int): Int =
    if (a > b) zero else
      combine(f(a), mapReduce(f, combine, zero)(a + 1, b))
                                                  //> mapReduce: (f: Int => Int, combine: (Int, Int) => Int, zero: Int)(a: Int, b:
                                                  //|  Int)Int

  def product1(f: Int => Int)(a: Int, b: Int): Int =
    mapReduce(f, (x, y) => x * y, 1)(a, b)        //> product1: (f: Int => Int)(a: Int, b: Int)Int

  def fact1(n: Int) = product1(x => x)(1, n)      //> fact1: (n: Int)Int

  fact1(5)                                        //> res3: Int = 120


September 30th

We can filter a pcap file and dump the output to a new pcap. The command is:
tcpdump -r <input.pcap> -w <output.pcap> <filter_string>
For example:
tcpdump -r zeroaccess.pcap -w new.pcap "tcp and port 80 and host and (tcp[((tcp[12:1]&0xf0)>>2):4]=0x47455420 or (tcp[((tcp[12:1]&0xf0)>>2):4]=0x48545450 and tcp[((tcp[12:1]&0xf0)>>2)+4:2]=0x2f31))"  

Go to August's log

Last modified: Tue Sep 30 22:15:56 PDT 2014