Wireshark installation in Mac OS Mountain Lion

Wireshark Installer hangs in the screen “Preparing for installation”, it does not throw any error. Wireshark requires x windows, but moutain lion does not ship with X11. You need to install it separately.

http://support.apple.com/kb/HT5293 http://ask.wireshark.org/questions/12140/cant-run-wireshark-in-mac-os-x-mountain-lion

I spent quite some time to figure out this, as there is no post related to this. Hope you find it helpful. Watch the following video for detailed explanation on wireshark installation.

http://www.youtube.com/watch?v=RkJt6JdtnOU

by ramandv Posted on August 12, 2013
Posted in Mac OS | Tagged | Leave a comment

Catch.com to Evernote migration

I have been using catch notes for the past 2 years. When I knew the news about catch notes shutting down “Catch will no longer be available after 30 August 2013.”, I started looking for alternatives. One of the best alternative is evernote. Though I have tried few years ago, but I havent used it effectively.

http://support.catch.com/customer/portal/articles/988949-how-can-i-export-notes-on-the-web-interface-

As per the above URL, exported my notes from catch.com

http://support.catch.com/customer/portal/articles/1243609-importing

Though the exported data contains enex file to import into Evernote. It does not automatically add the pictures and tags.

As I had only few pictures in my notes, I manually added it to the notes in Evernote. But I had around 800 notes, I dont want to tag each and every note manually. So I wrote a simple python script to convert all the #tag to evernote tags.

Check the gist at https://gist.github.com/ramandv/6146111

from evernote.api.client import NoteStore
from evernote.api.client import EvernoteClient

#Get your dev token from the following URL
#https://www.evernote.com/api/DeveloperToken.action

dev_token = "<your api token>"
client = EvernoteClient(token=dev_token, sandbox=False)
noteStore = client.get_note_store()


filter = NoteStore.NoteFilter()

#Find the guid of your catch notebook, by selecting the catch notebook and see the URL
#https://www.evernote.com/Home.action#b=<guid>
filter.notebookGuid = "<catch_notebook_guid>"

spec = NoteStore.NotesMetadataResultSpec()
spec.includeTitle = True
spec.includeTagGuids = True

index = 0
while True:
    noteList = noteStore.findNotesMetadata( dev_token, filter, index, 50, spec)
    for note in noteList.notes:
        index = index + 1
        if (note.tagGuids != None and len(note.tagGuids) > 0):
            print "already tags present, so skipping...."
            print "--------------[" + str(index) + "]"
            continue

        note = noteStore.getNote(dev_token, note.guid, True, True, True, True)
        content = note.content
        print 'Title:', note.title,
        print 'Content:', content
        print "--------------[" + str(index) + "]"
        titlecontent = note.title + " " + content

        #Remove HTML tags
        titlecontent = re.sub('<[^<]+?>', ' ', titlecontent)

        #get tags
        tags = [word for word in titlecontent.split() if word.startswith('#') and len(word) > 1]

        #remove unwanted characters
        tags = [word.strip(" #;.,").lower() for word in tags ]
        tags = [word.split()[0] for word in tags if len(word.split()) > 0 ]
        tags = [word.strip(" #;.,") for word in tags if len(word.strip(" #;.,")) > 0 ]

        #remove duplicate tags
        tags_set = set(tags)
        tags = list(tags_set)

        #Add tags only if the actual note does not contain tags
        if len(tags) > 0 and (note.tagNames == None or len(note.tagNames) == 0):
            print "[" + str(index) + "]###########" + str(tags)
            note.tagNames = tags
            noteStore.updateNote(dev_token, note)

    if noteList.totalNotes <= index:
        break;
    print "FETCHING NEXT PAGE======================="
by ramandv Posted on August 3, 2013
Posted in python | Tagged , | 7 Comments

Django get_or_create IntegrityError due to racing condition

In Django, get_or_create is not a thread safe method. So, it might results in racing condition. There is no standard solution available. More info on this issue in https://code.djangoproject.com/ticket/13906

Possible solutions from stackoverflow answer, and also better explanation about the problem

Basically get_or_create can fail – if you take a look at its source,
the you’ll see that it’s: get, if-problem: save+some_trickery,
if-still-problem: get again, if-still-problem: surrender and raise.

This means that if there are two simultaneous threads (or processes)
running create_or_update_myobj, both trying to get_or_create the same
object, then:

first thread tries to get it – but it doesn’t yet exists, so, the
thread tries to create it, but before the object is created…
…second thread tries to get it – and this obviously fails now,
because of the default AUTOCOMMIT=OFF for MySQLdb database connection,
and REPEATABLE READ serializable level, both threads have frozen their
views of MyObj table. subsequently, first thread creates its object
and returns it gracefully, but… …second thread cannot create
anything as it would violate unique constraint what’s funny,
subsequent get on the second thread doesn’t see the object created in
the first thread, due to the frozen view of MyObj table So, if you
want to safely get_or_create anything, try something like this:

 @transaction.commit_on_success
 def my_get_or_create(...):
     try:
         obj = MyObj.objects.create(...)
     except IntegrityError:
         transaction.commit()
         obj = MyObj.objects.get(...)
     return obj

Transaction.commit_on_success helps to get rid of frozen view of MyObj by creating new transaction.

by ramandv Posted on August 2, 2013
Posted in Django | Leave a comment

Basic CSS Notes

I have programmed in CSS too little. In this post, I will keep updating the things that I learned

by ramandv Posted on July 25, 2013
Posted in CSS | Leave a comment

SPDY Protocol – Simple tutorial

I was trying to understand about SPDY protocol. I couldnt find simple tutorial about “what is SPDY protocol?” and “Why SPDY”. So thought of writing one.

As the name implies, it makes the webpages to load faster.

Why SPDY protocol?

  • Many resources per page:

    For any webpage, The avg number of requests per page and the content size that needs to be downloaded has increased drastically over the past decade. It results in high latency.

  • Problems with <=HTTP1.1 protocol

    • HTTP does not have multiplexing
    • HTTP pipelining is susceptible to head of line blocking (http://en.wikipedia.org/wiki/Head-of-line_blocking) at the HTTP transaction level
    • verbose headers
    • \r\n in each line of header

Sample HTTP Header

GET /blog/ HTTP/1.1
Host: ramandv.com
Connection: keep-alive
Cache-Control: no-cache
Pragma: no-cache
Accept: text/html,application/xhtml+xml,application/xml;
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36
Referer: http://ramandv.com/blog/wp-admin/post-new.php
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8

  • TCP Slow start :

    TCP is designed to probe the network to figure out the available capacity. So, to use full capacity the connection should exists for longer time. But currently, the HTTP connections dies after downloading a image or js or html files. So, before the TCP starts using full capacity, the TCP connection dies.

Current workaround to sove HTTP problem:

Avoiding multiple HTTP requests by

  • Concatenating js, css files
  • Image spriting
  • Resource inlining

Increase the number of parallel requests to download faster

What is SPDY protocol?

SPDY is an application-layer protocol for transporting content over the web, designed specifically for minimal latency.

Features of SPDY :

[from Chromium SPDY white paper]

Basic features

Multiplexed streams

SPDY allows for unlimited concurrent streams over a single TCP connection. Because requests are interleaved on a single channel, the efficiency of TCP is much higher: fewer network connections need to be made, and fewer, but more densely packed, packets are issued.

Request prioritization

Although unlimited parallel streams solve the serialization problem, they introduce another one: if bandwidth on the channel is constrained, the client may block requests for fear of clogging the channel. To overcome this problem, SPDY implements request priorities: the client can request as many items as it wants from the server, and assign a priority to each request. This prevents the network channel from being congested with non-critical resources when a high priority request is pending.

HTTP header compression

SPDY compresses request and response HTTP headers, resulting in fewer packets and fewer bytes transmitted.

Advanced features

In addition, SPDY provides an advanced feature, server-initiated streams. Server-initiated streams can be used to deliver content to the client without the client needing to ask for it. This option is configurable by the web developer in two ways:

Server push

SPDY experiments with an option for servers to push data to clients via the X-Associated-Content header. This header informs the client that the server is pushing a resource to the client before the client has asked for it. For initial-page downloads (e.g. the first time a user visits a site), this can vastly enhance the user experience.

Server hint

Rather than automatically pushing resources to the client, the server uses the X-Subresources header to suggest to the client that it should ask for specific resources, in cases where the server knows in advance of the client that those resources will be needed. However, the server will still wait for the client request before sending the content. Over slow links, this option can reduce the time it takes for a client to discover it needs a resource by hundreds of milliseconds, and may be better for non-initial page loads.

BTW, SPDY v2 has been adopted for HTTP2.0

TCP, TLS, SPDY and HTTP relation

Currently, SPDY protocol is being supported in port 443. It could have been implemented in other ports, but because of the risk of that port being firewalled and transparent proxy issues, port 443 is being used.

During the TLS handshake (specifically ClientHello and ServerHello), TLS extension Next Protocol Negotiation (NPN) is used for the server to advertise a list of protocols it supports. As of now www.google.com is sending spdy/3 spdy/2 http/1.1. The client chooses which to use.

If a spdy protocol is selected, TLS transports SPDY. If http/1.1 was chosen, TLS transports HTTP, as it does for regular https sites. If a client does not advertise that is supports NPN the server should assume an http/1.1 connection.

TLS TCP SPDY protocols relation

Can I use the SPDY protocol?

Many web servers supports SPDY

How to know the SPDY protocol support?

Use the following addons, to know whether a site supports SPDY or not.

  • Chrome SPDY indicator
  • Firefox indicator
  • Opera indicator

In addition to this, Chrome helps to view the SPDY sessions in the chrome browser, just type the following URL

chrome://net-internals#spdy

Chrome net internals spdy protocol

by ramandv Posted on July 6, 2013
Posted in performance | Tagged | Leave a comment

Angularjs Unit testing – angular.mock.module and angular.mock.inject

Read the following post to why karma to get overview about unit testing with AngularJS

By default, ng module will be loaded during unit testing. But our modules should be loaded explicitly with angular.mock.module.

The angular.mock.inject function wraps the function into an injectable function. Basically, it will inject all the necessary parameters.

For example, you want to test moduleA, but it depends on other modules also, then first, you need to load them as follows

angular.mock.module('moduleA', 'moduleB', 'moduleC');

Then, You can inject the services from these modules as follows

angular.mock.inject(function(moduleAservice){
    moduleAservice.methodA();
});

If you want to mock the service, there are 3 different ways. Check the following post

http://stackoverflow.com/questions/13592259/mocking-my-own-service-in-a-unit-test

Note: angular.mock.inject and angular.mock.module is only available
for jasmine and mocha testing framework

by ramandv Posted on July 5, 2013
Posted in AngularJS | Tagged | Leave a comment

Angularjs Unit testing – Why Karma

There are somany tutorial available on how to write unit test. So you have writtern one simple unit test, but want to know the connection of jasmine, karma and angularjs.

Basically, to test a single angular controller/service etc, you need to make it as page. And include all the dependencies like jasmine, angular and your source files.

This service/controller will be called in your tests. So, include those tests in that page. Now, this tests should be called, which will be performed by jasmine specrunner.

Sample jasmine specrunner.html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
  "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
  <title>Jasmine Spec Runner</title>

  <link rel="shortcut icon" type="image/png" href="lib/jasmine-1.3.1/jasmine_favicon.png">
  <link rel="stylesheet" type="text/css" href="lib/jasmine-1.3.1/jasmine.css">
  <script type="text/javascript" src="lib/jasmine-1.3.1/jasmine.js"></script>
  <script type="text/javascript" src="lib/jasmine-1.3.1/jasmine-html.js"></script>

  <!-- include vendor libs... -->
  <script type="text/javascript" src="vendor/angular/angular.js"></script>
  <script type="text/javascript" src="vendor/angular/angular-resource.js"></script>
  <script type="text/javascript" src="vendor/angular/angular-cookies.js"></script>

  <!-- include source files here... -->
  <script type="text/javascript" src="src/Player.js"></script>
  <script type="text/javascript" src="src/Song.js"></script>

  <!-- include spec files here... -->
  <script type="text/javascript" src="spec/SpecHelper.js"></script>
  <script type="text/javascript" src="spec/PlayerSpec.js"></script>

  <script type="text/javascript">
    (function() {
      var jasmineEnv = jasmine.getEnv();
      jasmineEnv.updateInterval = 1000;

      var htmlReporter = new jasmine.HtmlReporter();

      jasmineEnv.addReporter(htmlReporter);

      jasmineEnv.specFilter = function(spec) {
        return htmlReporter.specFilter(spec);
      };

      var currentWindowOnload = window.onload;

      window.onload = function() {
        if (currentWindowOnload) {
          currentWindowOnload();
        }
        execJasmine();
      };

      function execJasmine() {
        jasmineEnv.execute();
      }

    })();
  </script>

</head>

<body>
</body>
</html>

But, if you use karma, creation of this page, launching browser will be taken care by editing a simple config file. It provides many additional features also.

Quote from https://github.com/karma-runner/karma

When should I use Karma?

You want to test code in real browsers.
You want to test code in multiple browsers (desktop, mobile, tablets, etc.).
You want to execute your tests locally during development.
You want to execute your tests on a continuous integration server.
You want to execute your tests on every save.
You love your terminal.
You dont want your (testing) life to suck.
You want to use Istanbul to automagically generate coverage reports.
You want to use RequireJS for your source files.
by ramandv Posted on July 5, 2013
Posted in AngularJS | Tagged , | Leave a comment

Angularjs – Console log Errors

TypeError: Converting circular structure to JSON

If you encounter this “TypeError: Converting circular structure to JSON”, check whether you are trying to $watch any object which has circular reference. It could be the culprit.

Uncaught Error: No module:

possible reasons:

Syntax : angular.module(name[, requires], configFn);

  • Check whether “angular.module” function is being called without optional square brackets. Basically, if the “requires” parameter is not being passed, angular will try to find the module. If the “requries” param passed, it will create the module.

  • you might face the problem when executing the unit test. If you split the angular module source across different files [ie, in only one file, ‘angular.module’ is called with ‘requires’ parameter. in other files, the module is being retrieved and updated]. In this scenario, karma.conf.js might include the js files like

‘app/js/**/*.js’

. You might get the above error, if the order of files [moduleA-part1.js, moduleA-part2.js] is changed,

Uncaught Error: 10 $digest() iterations reached. Aborting! Watchers fired in the last 5 iterations

possible reasons:

  • Main reason is that the function inside $digest cycle returns different value each time. For example,

    • the function inside $digest cycle, might return different object each time,
    • or you might try to randomise the return value of the function which is in $digest cycle.

  • If $watch is giving this problem, you can use the third paramter in $watch function. It makes it to compare object for equality rather than for reference. [angular doc of $watch]

by ramandv Posted on June 16, 2013
Posted in AngularJS | Leave a comment

Installing Django1.5-Haystack-2.0.0-beta with Solr-4.2.1

Install Haystack with his dependency pysolr

pysolr==3.0.6
django-haystack==2.0.0-beta

Install the solr-4.2.1 by following the steps

https://django-haystack.readthedocs.org/en/latest/installingsearchengines.html#solr

Follow the tutorial to configure haystack

http://django-haystack.readthedocs.org/en/latest/tutorial.html

Configuring Solr-4.2.1

  • Use the command build_solr_schema to generate the schema.xml file

  • Copy the schema.xml file to the following directory

    `solr-4.2.1/example/solr/collection1/conf/

    NOTE: The conf directory changed in this version

  • Add the following line in the section in the schema.xml

....
<field name="_version_" type="long" indexed="true" stored ="true"/>
....

Otherwise, the following exception will be thrown

Caused by: org.apache.solr.common.SolrException: _version_field must exist in schema, 
using indexed="true" stored="true" and multiValued="false" (_version_ does not exist)

  • Update the stopwordsen.txt path in the schema.xml to “lang/stopwordsen.txt”

    Otherwise, the following exception will be thrown

java.io.IOException: Can't find resource 'stopwords_en.txt' in classpath 
or 'solr/collection1/conf/', cwd=solr/solr-4.2.1/example`
by ramandv Posted on June 16, 2013
Posted in Django | Tagged , | Leave a comment

Iterm2 AppleScript: Opening Iterm2 with multiple named tabs

I always open multiple tab to launch server, mysql shell, ipython shell etc. Every time, setting the virtualenv etc in the shell and start server etc is a pain, which resulted in the following iterm2 applescript.

The following iterm2 applescript helps to launch the multiple tabs in iterm2 . And you can name the tabs. In addition to that, You can launch the tab with some start commands too.

tell application "iTerm"
    activate
    set startCmdList to {"renametab 'tab1' && ls -l", "renametab 'tab2' && ls"} as list
        -- probably you can add default commands to be executed
    set defaultcmd to "cd ~ &&"
    -- Create a new terminal window...
    set myterm to (make new terminal)

    tell myterm
        set number of columns to 150
        set number of rows to 30

        repeat with cmd in startCmdList
            launch session "Default"
            tell current session
                write text defaultcmd & cmd
                --set name to cmd
            end tell
        end repeat
    end tell

end tell

iterm2 “renametab” alias:

Add the following snippet in your ~/.bash_profile file. It add the function to rename your tab.

function renametab () {
    echo -ne "\033]0;"$@"\007"
}

If the renaming tab does not work, check whether your .bash_profile modifies the PS1 variable. You might need to comment out the following snippet.

#case "$TERM" in
#xterm*|rxvt*)
#    PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"
#    ;;
#*)
#    ;;
#esac

Installing the iterm2 applescript as an application

  • Open “AppleScript Editor” (/Applications/Utilities)
  • In a new document, paste the AppleScript code from the above snippet
  • Change the icon by opening bundle contents, and replacing with new applet.icns
  • Export the script to /Applications/Utilities, use the file format “Application” (do not select “run only” if you want to be able to change the script later on)
by ramandv Posted on June 16, 2013
Posted in Mac OS | Tagged | Leave a comment