Profiling Node.js applications on OSX

Tue, 02 Sep 2014 19:30:34 +0530

Note: ustack helpers don't work at all on OSX. So the output you get from using dtrace on OSX would be of little help only. Use SmartOS or Illumos based OS to enjoy the advantage of ustack helpers and USDT probes. More about why it doesn't work on OSX is explained nicely here

The happy flow:

Make sure all dependencies are installed.

g++ --version #preferably latest
make --version #preferably latest
python --version #Should be 2.6 or 2.7 (as required by node)

Download the Source Code of the latest version of node from http://nodejs.org/download/.
Extract the contents

tar xvzf node-v0.10.31.tar.gz #replace the filename if necessary

This is the most important part. You have to configure node build to be 32-bit and to use dtrace.

./configure --with-dtrace --dest-cpu=ia32

Build node with the configured options and install.

make
sudo make install

Go have a cup of coffee while make is taking place.
Verify node and npm installation

node --version
npm --version

Install stackvis - our flame graph generation tool. Use sudo if required.

npm install -g stackvis

Install your app dependencies. Insert the Nodejs dtrace probes and start your node application

npm install
export DTRACE_DOF_INIT_DEBUG=1
  node app.js > console.log 2> error.log & #you can start it however you like

Capture the profile using Dtrace. Dtrace requires additional privileges.

sudo dtrace -n 'profile-97/execname == "node" && arg1/{
  @[jstack(100, 8000)] = count(); } tick-60s { exit(0); }' > stacks.out

Run stackvis on the captured stacks.out to generate the flame graph.

stackvis dtrace < stacks.out > stacks.html

Open stacks.html from your favourite browser.

Gotchas:

If you start your node application with forever or pm2, it might not appear in the pstree as node. So you might have to change - execname == "node" to pid == 12345 or execname = "pm2" or something similar.
To get SVG, set the output type as flamegraph-svg.

stackvis dtrace flamegraph-svg < stacks.out > stacks.svg

I usually run my web app with NODE_ENV=production, 4 processes on a 4 core machine using cluster, and get profiles under the following conditions - 1. no load, 2. avg load - ( ab -n50 -c3 ), 3. high load ( ab -n200 -c10 ).
Found something ? Comment on this post, and I'll add them here.

Dockerfile for golang applications

Fri, 17 Jan 2014 15:09:58 +0530

A few headers

We will assume that we are deploying our applications on Ubuntu.

    FROM ubuntu:12.04
    MAINTAINER My Name <me@domain.tld>

Install dependencies

Add Mercurial Repository (if necessary)

# Mercurial
RUN echo 'deb http://ppa.launchpad.net/mercurial-ppa/releases/ubuntu precise main' > /etc/apt/sources.list.d/mercurial.list
RUN echo 'deb-src http://ppa.launchpad.net/mercurial-ppa/releases/ubuntu precise main' >> /etc/apt/sources.list.d/mercurial.list
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 323293EE

Install dependencies

RUN apt-get update
RUN apt-get install -y curl git bzr mercurial

Curl is required to download Go tar ball.
git, bazaar and mercurial are used by go get to download your application dependencies.

Install Go

RUN curl -s https://go.googlecode.com/files/go1.2.linux-amd64.tar.gz | tar -v -C /usr/local/ -xz

Configure Environment for Go

ENV PATH /usr/local/go/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin
ENV GOPATH /go
ENV GOROOT /usr/local/go

Hack - To cache go get

RUN go get github.com/myname/my-golang-project

This is called immediately before adding the project files to the repo and essentially before performing a go get from inside the repo's directory within the container. Docker's ADD doesn't support caching until 0.7.3. Adding the source files to the container and then performing a go get doesn't utilize the cache. It downloads all the dependencies once again and is time consuming.

Add source files to the container

WORKDIR /go/src/github.com/myname/my-golang-project
ADD . /go/src/github.com/myname/my-golang-project

Fetch new dependencies if any

RUN go get

Install application

RUN go build

If your app exposes a port, expose it

EXPOSE 8080

Set entrypoint

ENTRYPOINT ./my-golang-project -option value args

When you set the entrypoint in docker, the entire container behaves as if it was that single application/executable. You can daemonize this with docker run -d .

A simpler method

I have done the first few parts and pushed it to index.docker.io. You can create your image with that as the base image. And this is how you do it.

FROM boopathi/go:latest
MAINTAINER myname <emailaddress>
WORKDIR /go/src/github.com/myname/my-golang-repo
RUN go get github.com/myname/my-golang-repo
ADD . /go/src/github.com/myname/my-golang-repo
RUN go get
RUN go build
EXPOSE 8080
ENTRYPOINT ./my-golang-project -option value args

Links

Go dev environment docker image: https://index.docker.io/u/boopathi/go/

A fancy timeout in bash

Wed, 15 Jan 2014 19:02:34 +0530

Timing-out-and-moving-on is a programming paradigm generally followed in many multi-process or multi-threaded programs. In a scripting environment like bash, the solution is to do this in a hacky way. Of course coreutils now provides a tool called "timeout" which does exactly what we need - Run the command and timer concurrently. Whenever something exits, the other one gets killed.

Bash is all about finding amazing ways to do things. Here is one of them.

sh -ic "{ sleep $T1; echo one >&3; \
  kill 0; } | { sleep $T2; echo two >&3; \
  kill 0; }" 3>&1 2>/dev/null

Internals:

Going from inside out.

In bash { some command; } represents a block of code or inline group ( http://tldp.org/LDP/abs/html/special-chars.html#CODEBLOCKREF ). This construct creates an anonymous function and all the processes invoked come under the same process group.
The structure of the invoked script is { } | { }.
Each inline group has a kill 0; . The man page of kill says that kill 0 signals all processes in the current process group.
When you do kill 0; all that happens is the shell will start a new job to execute that kill command, so kill will only kill itself.
All the processes invoked inside { } will be under the same process group, so kill 0 would work there.
Note the output redirection >&3 within each of the inline groups. Here, we are writing the output to a different file descriptor identified by 3. 1 and 2 are already defined as STDOUT and STDERR.
This file descriptor 3 is used to communicate with the outside world. Since there is a pipe between the two inline groups, the output of first group cannot be captured - It goes as input to the second inline group. You have to either write it to a file or a stream represented by a file descriptor.
Just for clarity, I've used >&3 in both the inline groups. Capturing the output in fd3 outside this setup.
sh -ic "" creates a sub-shell and runs the command that's passed as argument. The -i flag is to run interactively.
Within this sub-shell, we have our two inline groups sending necessary outputs to fd3. So outside this sub-shell (running interactively ) we can capture fd3 and redirect it however required
3>&1 fd3 is redirected to STDOUT of the session from which the sub-shell is invoked.

Putting everything together: (the short version)

Create a new sub-shell to run interactively. Within this sub-shell, create an inline group that outputs to fd3. Pipe this inline group to another inline group which also outputs to fd3. In the session which invoked the sub-shell, redirect fd3 to STDOUT.

Why does this work ?

All that we are doing is pipe two inline groups. One of the inline groups contain our main program which shouldn't run longer than $TIMEOUT . The other inline group would sleep for $TIMEOUT - both followed by a kill 0; statement. Whichever exits first is going to call the kill command and all the processes in that process group will be killed. This includes the sub-shell and all the processes within the sub-shell.

Example:

Capture bandwidth usage per process using nethogs for 1 hour.

sh -ic "{ /usr/sbin/nethogs -t >&3; \
  kill 0; } | { sleep 3600; \
  kill 0; }" 3>&1 2>/dev/null

More about this is discussed here - http://boopathi.in/blog/capturing-per-process-bandwidth-usage-using-nethogs.

Capturing per process bandwidth usage using nethogs

Wed, 15 Jan 2014 13:05:01 +0530

On some of our servers at directi, we capture the bandwidth per process using nethogs on an interface over a period of time. The official description of nethogs from its author is as follows.

http://nethogs.sourceforge.net/

NetHogs is a small 'net top' tool. Instead of breaking the traffic down per protocol or per subnet, like most tools do, it groups bandwidth by process. NetHogs does not rely on a special kernel module to be loaded. If there's suddenly a lot of network traffic, you can fire up NetHogs and immediately see which PID is causing this. This makes it easy to indentify programs that have gone wild and are suddenly taking up your bandwidth.Since NetHogs heavily relies on /proc, it currently runs on Linux only.

The -t parameter : Trace mode

This is the bug-hunt mode. You enable it and collect the output stream over a period of time. From this log you calculate the necessary results. Nethogs refreshes every second and gives the output in kbps.

How ?

Start nethogs to listen on some interface and grab the output to some file.

    /usr/sbin/nethogs -t eth1 &> output

Now you might want to run this in the background. And the solution is going to be simple.

    nohup /usr/sbin/nethogs -t eth1 &> output &

But remember that, that script is going to run forever unless you manually kill it. Now you have to follow the time-out-and-move-on paradigm. There are many ways to do it in bash, but the simplest and most efficient way is this:

    sh -ic "{ /usr/sbin/nethogs -t eth1 &> output; \
    kill 0; } | { sleep $TIMEOUT; \
    kill 0; }" 3>&1 2>/dev/null

The output:

The output of the nethogs tracemode will be something like this.

Refreshing:
/usr/sbin/exim/689041/47 0.0167969 0.289844
10.10.10.10:80-11.11.11.11:58485/0/0 0.0105469 0.194141
pop3/703049/2926 2.46855 0.140625
/usr/local/apache/bin/httpd/700895/99 4.44023 0.0949219
/usr/local/apache/bin/httpd/669759/99 1.69883 0.0832031
/usr/local/apache/bin/httpd/678264/99 1.71016 0.0738281
/usr/local/apache/bin/httpd/673811/99 2.99844 0.0730469
/usr/local/apache/bin/httpd/578292/99 0.588281 0.0621094
unknown TCP/0/0 0.855469 0.0503906
/usr/local/apache/bin/httpd/701327/99 3.65273 0.046875
pure-ftpd (PRIV)/680421/0 0.0574219 0.0460938
imap-login/88707/97 0.0183594 0.0271484
sshd: boopathi@pts/7/656215/32076 0.0730469 0.0222656
12.12.12.12:80-13.13.13.13:58271/0/0 0 0.0144531
/usr/local/apache/bin/httpd/677948/99 0.291797 0.0117188
Unknown connection: 60.60.60.60:80-66.66.66.66:36813
Unknown connection: 100.200.105.205:80-65.56.56.56:42293

And the fields (for the lines that are not either a warning or an error) are

Process / Process id / User id    sent kbps    recv kbps

Parser

Source code: https://github.com/boopathi/nethogs-parser

Automating

Let's take into consideration the output stream of nethogs - specifically number of bytes per second and assume that the file size will not be too high for logs fetched over a particular time period. Let's fix this as 1hour. So, for 1hour nethogs is going to listen on some interface and for the next hour a new process will be invoked. The simplest way to do is to write the script that will run for an hour and call it in an hourly cron.

monitor.sh:

    #!/bin/bash


    TIMEOUT=3600
    IFACE=eth0
    TIMESTAMP=`date +%b_%d_%Y_%H_%M_%S`
    OUTPUT=/var/log/nethogs/$TIMESTAMP.log


    sh -ic "{ /usr/sbin/nethogs -t $IFACE &>$OUTPUT; \
    kill 0; } | { sleep $TIMEOUT; \
    kill 0; }" 3>&1 2>/dev/null

cron: /etc/cron.d/nethogs.cron

0 5,6,7,10,11,20,21 * * *    root    /path/to/monitor.sh

The Graphite story @ Directi

Wed, 27 Mar 2013 16:14:31 +0530

TL;DR

At Directi, we use graphite as one of our principal monitoring tools. Initially, when Graphite set its foot into the company's monitoring system, the setup was a cinch where one server was dedicated to receive metrics data, render the graphs and serve it to the requesting users. The problem with this setup was pretty much like a talking turkey, that it hit the bottleneck when the amount of metrics sent to this server became significantly high. This'd result in carbon-relay and carbon-cache instances on the system consuming a lot of CPU and Memory leaving not much resources available for the webapp and WSGI Daemon to use. This indubitably resulted in performance issues with rendering the graphs. It literally took a minute to render each graph. And when we perform some functions(provided by graphite) over the graph data, it'd almost take a minute and a half or more to render the graph. I'd have a coffee-break in the time my dashboard loads.

The primary bottleneck that we were facing was the disk I/O - as faced by every damn graphite beginner out there in the dev-ops team. So we decided to use SSDs on the server. But, it wouldn't be a cheap solution to replace all rotational drives with SSDs, because, we'd require a hell a lot of storage to store that much amount of whisper data. But we didn't want to sacrifice the power and advantage of SSDs. Flashcache came to the rescue. Flashcache is a block-cache for Linux. It helps extending Linux block-cache with a SSD drive. It is cheaper than buying half a TB of RAM and 5 TB of SSD drives. After a few tests and try-outs with flashcache, we decided to use write-back caching mode. In write-back mode, all writes go to the cachedev (SSDs) initially, and based on few policies, they are written to the disk later. All disk reads are cached. It's determined and documented to be the fastest mode to write and is pretty unsafe. We really didn't care much about the safety of the data received, because, even if we lose a few metric points, it's not that big a deal to be significantly important.

Once flashcache was up and doing its work properly, we started to enjoy the bounty of the performance improvements with respect to IOPS. During this time, metrics were sent from 100s of servers, and the maximum number of metrics per minute reached a value of 300k. A single carbon-relay instance was able to handle this much load i.e. it held the incoming metrics in its queue and relayed it to the 3 carbon-cache instances on the same system. At this time, relay was not our bottleneck. Meanwhile, when planning to experiment and expand this setup to serve greater extents, we faced this NFC (abbreviated as No F**king Clue) issue with flashcache - Kernel Panic on system reboot OR halt. Every time we rebooted graphite server, during halt scripts' execution, KERNEL panic occurs. NFC!!!. It's not that easy to understand the Kernel Panic logs the first time you see it. It's exactly the same thing as reading a bunch of scattered items from a pool of hexadecimal values. Investigating further, we developed experience on "How to" and "How not to" use flashcache, but none of these solved our Kernel panic issue. Finally, there came a moment, where we had to dig through Linux's halt process and confirmed that flashcache was the culprit. When the halt process was happening, removing the Logical Volume using dmsetup wouldn't happen automatically. The halt script would just try to unmount whatever is on its way to completing the shutdown process. So we plugged in a SysVInit script as a hook that is invoked by the halt script when it sees a flag for flashcache in subsys. This init script is responsible for loading and removing the LV. While removing we need to consider whether or not to sync the data between the cache and the backend disk. This is determined by the sysctl flag "fastremove". However, the default behaviour on a remove is to clean the dirty cache blocks to disk and the remove will not return until all blocks are cleaned OR in simple terms - Setting the fastremove option to 0, flushes the cached data to the backend disk and setting it to 1, the cache would still have uncommitted dirty blocks. A little more on tweaking flashcache could be found in my other blog post - http://boopathi.in/blog/flashcache-blah-blah.

With this setup, the graphite server with a dozen cores CPU, and a dozen gigs of Memory was receiving and replying fairly well for sometime until we hit the next bottleneck with carbon-relay process. At an average, the graphite server received 300k metrics, and the sender script used plain TCP connection for every line of the metric. We didn't require aggregation for most of the metrics, since preserving the individual values mattered. The problem with the Carbon agents was that the number of file descriptors got exhausted and they were not able to open any more files to write to. This resulted in an increase in the queue size of relay. And this hit the limit as set in carbon.conf under relay section. I remember the queue size to be around 5000 data points. At this point, relay started throwing errors onto its console. The error said - "Could not accept new connection (EMFILE)". EMFILE refers to the shortage of file descriptors available to that particular process, here, carbon-cache. So, we increased the ulimit from 1024 to 4096 for the session that initiated the carbon-daemons. But this wasn't sufficient. So, we increased it to 10240, and increased the queue size to 10000. The relay started accepting connections, and everything was back to normal. Not so fast buddy? To where? A stable graphite setup ? Huh. You know that's insanity. By the time we realized that we had to handle a larger load, relay started trembling again. The root cause for this was a bug in the metric collector scripts. The collector scripts works in a way that it doesn't send to graphite immediately. We do some caching on every client system, where we write all the collected metrics to a temporary device, and a cron reads the cached metrics and flushes it to the graphite server. When this cron is not able to establish the TCP connection, the metrics remain in the cache file - cluttering. When the bug was fixed, the amount of cached metrics emerging from 100s of servers was more than twice the average value (around 800k). At this point, relay had no other choice but to stop accepting metrics. A quick fix that we discovered was to restart relay when the number of metrics it accepts goes below a critical value. After some analysis, 4hrs seemed to be a fair value. So we kept restarting relay every 4 hours, a graph of which would explain the failure of the process gradually.

In this way, we reached to some arbitrary solution, where we tweaked the collector scripts to send lesser number of metrics. At this point, it's quite obvious that graphite server was hitting the bottleneck every 4 hours. The average amount of metrics received is around 300k, but the graph shows 750k metrics everytime carbon-relay starts accepting all the connections made to it. The reason for this to happen is the caching mechanism we follow on the client systems. So all those metrics which failed to get into graphite would still be there on the clients. The next cron would try to push it, but graphite just wouldn't accept any more connections. So when relay is restarted, graphite starts fresh, and memory is literally empty where it can accept all the connections made to it. That's why we could see a spike to 750k data points every time relay starts fresh.

Moving on the the next step in optimizing, we decided to put relay on a separate system, so that carbon-cache would have enough system resources to write to whisper files, while, on the other hand, relay would also have enough system resources to maintain a larger queue size as well as accept a larger number of connections. So we cloned relay to a new system graphite02 which was an exact clone (in system configuration) of graphite01. We initiated relay on graphite02, and it seemed to have solved the problem. All the cached metrics were accepted by the graphite02 relay continuously without any interrupts and they were relayed to carbon-cache instances on graphite01. A graph of that found below would explain that the area covered by the red lines was the initial load that graphite01 should have bared to be stable.

An obvious analogy corresponding to the area mapped and area unmapped between two spikes can be found to correlate with the area mapped by the red region. And after all the cached metrics are sent, the rate of metrics reception becomes normal to a value of 200k per min. So, as mentioned earlier, all that graphite needed to handle was the initial spike (cached data points).

Here comes the situation, where we feel that we aren't using the system resources completely. We had two servers dedicated to monitoring, each had a dozen cores CPU and a dozen gigs of RAM, and SSDs and flashcache. So the next thing to try is to make this setup a cluster and load balance the carbon-agents and the webapp. Initial thoughts was to make it a High Availability cluster using corosync. But, we wanted to do things simpler than that. Graphite, by itself, provides a mechanism to load balance all the services and the webapp. They call it "The Federated Graphite Setup". We read through some of the blog posts and launchpad answers and started tweaking carbon.conf and local_settings.py in graphite webapp. The graphite cluster now contained two servers, each running a relay, three carbon-cache instances, and the graphite webapp. Flashcache was setup on graphite02 too in a similar manner as present in graphite01. The relays are configured to feed its queue to all the 6 carbon-cache instances on the two systems. The issue is to determine which relay would serve the need, but the aim is to use both the relays. First tryout was that one relay(graphite01) received the metrics and relayed them to the relay on the other system(graphite02) and this load balanced between the 6 carbon-caches. The relay on graphite01 was configured to occupy more memory - bigger queue, and relay on graphite02 (second stage) was configured to occupy a lesser memory. This setup wasn't that appreciable. So we decided to load balance before the data points reach relay. HAProxy solved our issue. We configured HAProxy to load balance between graphite01 relay and graphite02 relay, each of which would further load balance between 6 carbon-caches.

Our next issue was with syncing the old metrics to the new system. We hit one more bottleneck where syncing between two production systems that is live becomes impractical. The root cause for this problem is the load balancing technique used. We used roundrobin in HAProxy and consistent-hashing in graphite relay. To understand consistent-hashing, consider the 6 carbon-cache instances to be the slots and the data-points to be the keys K. So, on an average only K/6 keys need to be remapped. This splits the relay's incoming load almost equally to the 6 carbon-caches. On seeing a data-point, carbon would know where it'd get relayed to. (perks of consistent-hashing). Some of the metrics got relayed to graphite02 creating new files on graphite02 while leaving the files on graphite01 untouched. The old data which was already present on graphite01 should be merged with the new incoming metrics which didn't happen automatically. Also graphite has this sick mechanism that if a whisper file is present in the same system the webapp is running, it wouldn't query the other systems for the data. So the only way to solve this issue is to copy the old data to the new system, apply the consistent hash manually upon all the files, and unlink them if they don't belong there. But we hit the bottleneck right at the first step - syncing data between the two servers. Started rsyncing the data, and estimated the time to complete. The value that appeared really shook my head - 10 days, which is impractical to be done on a live production system. So we decided to keep only few important metrics, rsync them and clean the files based on consistent hashing applied by carbon. The syncing was done, and some of the important metrics were retained, and the performance of the cluster was amazing that it could handle 4x load. We also increased the number of carbon-cache instances to 6 on each system. I would like to share an infographic of our setup.

_ _

I know it's a very long post. One of the conclusions that we arrived at one point was to completely chuck graphite as the monitoring system, and use OpenTSDB. But it doesn't provide enough functions like graphite does. And graphite is a really good project that I just can't let it go. Now I'm trying out ceres database to use it in the place of whisper. Will blog about the same some time later ;). Thank you for reading till here.

Writing your own JavaScript build system

Wed, 13 Feb 2013 18:53:31 +0530

I'm assuming that Google Closure compiler is one of the best JS compilers, and ll be using their API " closure-compiler.appspot.com" in this tutorial. Also, I'm going to assume that you must be writing coffee-script and you have coffee-script compiler pre-installed on your system. Don't get demotivated or threatened by seeing the words "build system". We are actually making a script that automates the final step in our development process, and makes the workflow a little swifter.

The gist of this is, we write coffee-script, use coffee-script compiler to compile our code to plain JavaScript, we feed this JS code to Google Closure Compiler along with the link to libraries that our code depends on, we get the output from the Google Closure Compiler API, and we export that to a file in our repository.

1. Coffee-Script to JavaScript

From various resources and a little of personal experience, the conclusion is - the subprocess module in Python suits the need fairly well.

    command = ['/usr/bin/coffee','-cp','./path/to/file.coffee']
    process = subprocess.Popen(command, stdout=subprocess.PIPE)
    js_code_1 = process.communicate()[0]

The code is pretty much simple and is self explanatory. For those who are not familiar with Python, the gist of the above code is this

    execute the command '/usr/bin/coffee -cp ./path/to/file.coffee'
    grab the output to some variable instead of throwing it to the system STDOUT

2. Grab plain JavaScript

This can be viewed in two ways. One is that you don't write coffee-script. The other is, you write some coffee-script and then you'd want to add some plain JavaScript too. Once again, if you're not familiar with Python, grab the file contents to some variable. Python users, this might be one of the best ways to grab file contents into a variable.

    with open('./path/to/script.js') as javascript_file:
        js_code_2 = javascript_file.read()

3. Collecting dependencies

This section is just blah. You've to fetch the URLs of the libraries that you use. It doesn't matter if that is not a minified version. Google Closure Compiler would handle that for you. If you don't have a permalink to your library, clone that to your system and follow the previous step to grab it's contents.

4. Sending request to Google Closure Compiler API

Send a HTTP - POST request to 'closure-compiler.appspot.com/compile' with the content-type "application/x-www-form-urlencoded". Grab the response to some variable. This would represent the minified/compiled JS code. Now you can throw this to some file.

The following POST variables are important while requesting the API.

code_url : the permalink to your library code
jscode : the contents - plain JavaScript code. ( jscode1 and jscode_2 from the previous sections)
compilationlevel : 'SIMPLEOPTIMIZATIONS' . Refer the API for more options.
output_format : 'text'
outputinfo : 'compiledcode'

Download and Use:

Have uploaded a sample working Python code. https://gist.github.com/boopathi/4944562

To get the script,

    wget https://gist.github.com/boopathi/4944562/raw/3bd83077ef3081eaedc5d74712a09fc0981e8f1d/jscompile.py && chmod +x jscompile.py

Prototypes can't talk to local variables in the constructor

Wed, 13 Feb 2013 16:10:22 +0530

You can create local variables via a closure but then you'd have problems with inheritance.

    var counter = function () {
      var count = 0;
      this.increment = function () { count++; }
      this.get = function() { return count; }
    }
    //now
    counter.prototype.decrement = function () {
       count--; //will be undefined :)
    }

Well, this might not seem weird unless you compare it with some other programming language where Object Orientation is proper and true.

    class counter {
    private: int count;
    public:
      void increment() { count++; }
      void decrement();
    }
    void counter::decrement { count--; } //count is not undefined.

JavaScript is not a queer creature. There is no actual OOP in the language. The scope is what that helps us in simulating Object Orientation. Well, to make that work with JavaScript, you could use something like,

    var counter = function () {
      this.count = 0;
      this.increment = function () { count++; }
      this.get = function() { return count; }
    }
    //now
    counter.prototype.decrement = function () {
       this.count--; // yes, you have to use `this` in javascript a lot!
    }

    // using the "class"
    c = new counter();
    c.decrement();
    console.log(c.count); // -1

Keep in mind that, certain aspects won't be enforced by the language itself, but instead provided as a syntactic sugar. The bottom line is that, when you write some code that plugs into a browser, you need not worry about protecting variables from the user. He can anyway peep in and do nasty things if he wishes. This might be a concern in environments like NodeJS.

Flashcache - Blah blah

Mon, 04 Feb 2013 22:13:21 +0530

We were recently testing Flashcache on one of our servers at Directi. From various resources over the web, and from a little of my personal experience, I'm writing this post about flashcache.

Flashcache, for those who haven't seen before, by definition, is a block-cache for Linux. That just means - it's a method for extending Linux block-cache with SSDs, by which, we save a few bucks avoiding half a TB of RAM just for caching. The performance is determined by various factors. In an abstraction, I'd tell these factors are important -

the size of the SSDs (best performance is obtained if the size of your SSDs is larger than the active-set of blocks. And, if you get perfect caching, the performance will be similar to running your entire system on SSDs) ,
the estimate of reads and writes,
flush frequency (when the writes are set to flush immediately - in case of file-system ops and db ops, caching is briefly done and performance will not be affected by the presence of flashcache).

Some internals:

CPU cache - The Linux-block cache works in a manner that it caches accessed blocks and NOT files. So long as you're not giving your program / some KVM direct access to these block devices, the Linux block cache will be in play. However, if you provide direct access, the answer to will this work? would become less clear.

Quoting from flashcache docs ( flashcache - modes of caching ) , It provides 3 modes of caching: writeback, writethrough, and writearound. What happens when your SSD fails/disappears depends on your caching mode. Follow the above mentioned link for more info.

During the initial stages of testing, we were handling a large number of writes per second - which resulted in periodic saturation of primary storage. We decided on using write-back caching mode (as the writes were very high) and the performance increase was significant. As our reads got higher, the performance went down, and now, I'm scratching my head to find a way to get around this.

Unhelpful JavaScript story: Are you new to JavaScript ?

Fri, 23 Nov 2012 17:01:18 +0530

Well, here goes a sophomoric short life story. A programmer who started developing his skills with C, C++ or any other imperative language, arriving at JavaScript could think that he has nothing to learn except the syntax and a few syntactic sugars the language'd provide. Well, like every movie, there has to be a not so ambiguous twist, and here we go - It's definitely going to result in vain if that thought is his perseverance. In a contradiction, he might think this is a broad effort (quote: learning the language), but his goals are not to simply regurgitate the things he learnt, but to help boil down to what's important to get him on his way. Coming to the next part of his JavaScript life, he would understand that, it's merely a language which leaves out some of the basic structures and components that are so commonly used/needed. To reduce these boiler-plates and abstract browser inconsistencies, and to increase the level of tools/API, he would choose to use libraries which satisfy few/most of the requirements. He gets caught up in excitement and understands how easy development could be. And, it wouldn't be anytime sooner, that he would understand, these libraries, some of them, don't play well together. They overlap in functionality and features.

Like there comes a time in every developer's life where we arrive at some hypothesis of writing our own code for a library, he scrawls some code implementing some features. Before we continue, I would like to analyse his mistakes. May be he thought that his work'd be superior as he is eliminating most of the unnecessary stuff the library provides (or) may be he didn't do enough research on what's likely to be the most necessary thing in the application (or) may be there's just this one small thing that isn't quite right in the library, so he wants to rewrite it from scratch. The truth is, he has done it, I've done it, and you likely have too. The next phase of his life would be the annoying part. He realizes, but slowly, that he is doing nothing but what is so called as "re-inventing the wheel" and whatever functionality he writes is already there under development for a significant amount of time, and the community that develops these things must have experienced something similar. He finds it funny that he wasn't the only one who fell into this ludicrous life.

Not a great scheme for an ending. I know. But some of you would've got the memo. :).

The advice section:

The long-evo idea about libraries is that these are developed in some remote part of the web and the community that develops these things is not available to approach by normal people. The idea, as certain as Heisenberg could be, is nothing but drivel.

One thing that most of us don't understand immediately is that, Learning JavaScript is not a one-time event. It'll happen over a long period of time, but (with a little/more stress on this point) it's important that you actually make it a priority to LEARN the language. Often people learn libraries, and not the language, a mistake which will certainly bite you in the end.

Blah blah stuff

Don't get me wrong about writing your own libraries. It's completely possible ;) .

Threaded SSH

Thu, 22 Nov 2012 00:28:02 +0530

Kindly note, this doesn't mean implementation of threading in SSH.

Problem statement

When we have multiple servers to manage, we configure it in such a way that they are easily manageable. To meet this requirement, as a first thought, we make similarities in few of the configurations, packages and repositories used in/by those servers. This is mostly applicable in cases where/whenever High Availability, Sharing of resources and Horizontal Scaling comes into picture. Once everything is set, we might run into a situation, where we would want to run a script on all those servers and get the output and error to one place for reviewing. Now, one way to do this is, while setting up those servers, we could use a tool, called MCollective, that pertains for these kinds of operations. The quick and dirty way is to ssh into every server, write the script, create a cron job that invokes this script, and throw the output to some static page that can be fetched over HTTP. Now, being a system administrator, we would NOT want to do that. We would end up writing a tiny script in bash, which sends the command over ssh, executes it, and standard output is redirected to the system(callee). We have to find a better way (and a Simple way) to do this.

Sequential SSH

As discussed above, whenever we would like to execute something over SSH, we write something like this.

    for i in {1..50}; do
        ssh boopathi@server-$1@myhost.com "top -b -n1 | head -n5";
    done;

The code, sequentially, after getting authorized, executes the command within quotes (the top 5 processes in that system), and echoes the output to the callee. This takes lots of time where the callee waits until the previous instance has exited.

Threaded SSH (Solution)

Threading seems to solve the issue of waiting for every instance to be completed. So, one of the possible SIMPLE solutions, would be to invoke the ssh function in separate threads. As, each of the thread spawned is further going to invoke a sub-process that runs SSH, and also, as there is nothing to be shared between these instances, we can safely write a SIMPLE module that would facilitate this feature. So, now, all connections are instantiated in parallel or whenever a new thread can be spawned. The output is thrown to STDOUT of the callee whenever a thread exits. And it obviously becomes much faster on a multi-core system.

Code (Python)

You could as well follow this gist, where the revisions are maintained.

https://gist.github.com/4046754

Code:

    #!/usr/bin/python

    # This doesn't mean threading in ssh. What it means is, every thread
    # that this program spawns, creates a new subprocess to perform the
    # system "ssh" command.

    # Sample Usages
    # =============
    #
    # 1.
    # for i in server1 server2 server3 server4;
    # do echo username@$i;
    # done | ./tssh "echo 'Good night :P'; rm -rf /; "
    #
    # 2.
    # cat connection_list | ./tssh "ifconfig lo"
    #
    # 3.
    # cat server_list | while read server;
    # do echo "username@$server";
    # done | ./tssh "top -b -n1 | head -n10" --noheader

    import sys,os,threading,shlex,subprocess


    basecommand = "ssh -o StrictHostKeyChecking=no"
    header = "echo \"%s says,\"; echo '=========================================='; "

    nohead=len(sys.argv)>2 and sys.argv[2]=='--noheader'

    class Runner(threading.Thread):
        def __init__ (self, server, command):
            super(Runner,self). __init__ ()
            self.command = shlex.split(basecommand)
            self.command += [server]
            self.command += [(header % server, "")[nohead] + str(command) +";"]
        def run(self):
            sout = subprocess.Popen(self.command, stdout=subprocess.PIPE).stdout
            for i in sout:
                print i.strip()

    if __name__ == ' __main__':
        while True:
            try:
                server=raw_input()
            except EOFError:
                break
            t=Runner(server=server, command=len(sys.argv)>1 and sys.argv[1] or "echo ")
            t.start()

Internals

The code reads from STDIN (Standard Input) - the list of servers, till EOF (End of File). Once this collection is made, it spawns a thread(for every server), which constructs a sub-process that runs the SSH command. The --noheader flag is just to prevent pretty print option. Use it if you've to parse the output or use it on another process.

Installation

No big deal. Save the source. Give execute permissions. Those who are struggling to install, use this script to download and install.

    curl https://raw.github.com/gist/4046754/af7b26f723c9da37a9e34102be93285ed92be756/tssh.py > /usr/bin/tssh && chmod +x /usr/bin/tssh