My new address: http://freethreads.wordpress.com/

This blog has been moved to: http://freethreads.wordpress.com/

NDMP: An open protocol to backup enterprise data

NDMP is an open protocol that enables backup/ restore of data in a heterogeneous environment. A real world case: You have two file servers(filer) from NetApp and EMC. And you want to backup your data without bothering about which filer you'd use.

Here NDMP comes into play. If both the filers are running an NDMP server, you can seamlessly carry out backup and restore. You just concentrate on your backup operation and leave out the details of :
- How data is stored ( you need to know data format though)
- What operating system is running on the filer
- Add/ remove a filer with no problem

Extensive information: www.ndmp.org/

VMWare player vs VirtualBox: which is better?

I have a Windows XP host machine with guest Ubuntu OS. I started with the latest VMWare player and benefits I noticed:

1. Hassle-free installation of Ubuntu
2. Seamless integration between guest and host
     o) You can copy-paste files across host and guest!
     o) Clipboard is shared bi-directionally

3. Very good performance: applications, network, and devices (DVD)
4. Display scales well on bigger screen with VMWare tools.
5. Alas! you can have at-max one processor simulated

VirtualBox

1. Easy Ubuntu installation
2. Most horrible and pathetic clipboard sharing. Contrary to claimed, it provided one-way clipboard from guest to host.
3. XP clipboard stopped working and never worked till I stopped VirtualBox.
4. Supports VMDK files but it's crappy, buggy and leaves VMDK in an un-usable state. I could never run my VMDK with VMWare later.
5. Can simulate up to 4 cores.
6. Supports shared folder between host and guest. I think it's a generation behind what VMWare provides.
7. Very slow performance: application, system start-up/ shutdown or stand-by. This is the biggest letdown.

My verdict: VirtualBox has miles to go and I am using VMWare still :)

How to Upgrade Ubuntu guest in VmWare?

o) Get the ALTERNATE Ubuntu image from UBUNTU website. (I prefer using Torrent to get this file)
A DESKTOP ISO will _NOT_ work!
o) Now, it's a blatant lie that you would not need an Internet connection. You got to have it; if not, go and do something else.
o) Okay, once you got you alternate ISO image ready, do the following:

• sudo mkdir -p /media/cdrom
  sudo mount -o loop ~/Desktop/ubuntu-10.10-alternate-i386.iso /media/cdrom

• gksu "sh /media/cdrom/cdromupgrade"

o) Yup! that's pretty much you need to do.

Corrupted VMWare VMDK file? How to fix?

o) Create a new VMWare machine of the same guest OS.
o) Assign the corrupted VMDK file as additional HDD to the new machine. VMWare machine config allows you to add an existing HDD (VMDK file) to a machine.
o) Browse your corrupted virtual HDD. Enjoy!

Understanding quirks of C: Structures

Targeted audience

=============

C programmers

System information

==============

Linux ubuntu 2.6.32-25-generic #44-Ubuntu SMP i686 GNU/Linux

gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5)

Structure are the most popular user-defined data type in C. It allows an user to create new data type by packing different data types and use them with a single name. In this article we will discuss fundamental operations provided by C structures and their internal behavior. We'd use following structure for understanding the concepts.

----CODE----

struct test{

  char c;

  int a;

  char d;

};

----CODE----

A variable of type “structure test” would have following memory layout in the process stack. We are assuming that compiler padding is done for a 4-byte boundary.

 Structure layout in memory

Let's assign values to this structure variable.

----CODE----

struct test var = {'a', 10, 'b'};

struct test *ptr;

ptr = &var;

printf(“%c”, var.d);

printf(“%c”, *(ptr->d));

----CODE----

How do the member access expression “var.d” and “ptr->d” work?

“var.i”: This expression is converted by the compiler into two components:

          a) Base address of symbol 'var' and,

          b) Offset of symbol 'd' in the structure memory layout

Base address of the structure variable is “0x0000” in our example. Next, compiler finds the distance of 'd' from the base. The distance is 8 bytes (1+3+4).

Similarly, “ptr->d” is also factored in mentioned manner, and address of member 'd' is returned.

How to calculate a member's byte offset in a given structure

In a trivial manner, we can calculate offset of member 'd' by getting the address of first element of the structure i.e. 'c' and then finding the difference with address of 'd'.

----CODE----

offset_of_d = (&var.d - &var.c);

----CODE----

The difference of these two addresses would fetch us the offset of 'd'. “&(var.c)” fetches us the base address of the structure.

The statement &(var.d) would be treated as = (Base address of variable 'var' + distance of 'd' from the base).

Efficient method to find the offset of a structure member

If we could set the base address to zero in above equation, we would directly get the offset of 'd' i.e. 8.

We can accomplish this as follows:

----CODE----

#define POINTER ((struct test*)(0))

int main()

{

  unsigned int x;

  x = &(POINTER->d);

  printf("\nOffset of d = %d\n", x);

}

----CODE----

The macro “POINTER” expands to “&(((struct test*) (0))->d)”.

We are typecasting '0' as an address, pointing to type of data 'struct test'. Now when you do “&(((struct test*)(0))->d)”, it is fetching the address of 'd' as:

                      (Base address + offset of d in the structure).

Since base address is set to zero, we are left with the offset of 'd'. We are tricking the compiler by giving base address as zero. This trick will work irrespective of compiler's padding scheme.

Both the discussed approaches lack portability though, and hence may suffer with incorrect output on different hardware. Another portable way to accomplish the same is with the help of “offsetof” macro, defined in header file stddef.h. The macro 'offsetof' accepts two arguments:

– The structure definition, so you don't need to create a structure variable

– The member element, for which the offset is to be calculated.

This macro too takes care of structure padding of elements, performed by the compiler.

------CODE------

#include "stddef.h"

int main()

{

  unsigned int x = offsetof(struct test, d);

  printf("\nOffset of d = %d\n", x);

}

------CODE--------

How “sizeof” behave in a quaint manner, where syntactically you find a NULL pointer dereference?

Consider the following code:

----CODE-----

struct test{

  char c;

  int a;

  char d;

};

int main()

{

  // Define a dangling pointer of type struct test

  struct test *p;

  // Getting the size of the structure with an uninitialized pointer

  printf("%u", sizeof(*p));

  printf("%u", sizeof(p));

  // Try it with one of implicit types like 'char'

  char *i= 'a';

  printf("%u", sizeof(*i)); // Displays 1, as the size needed to store a character

}

----CODE----

If you dereference a pointer inside “sizeof” operator, it fetches the size of data-type, the pointer is pointing to, i.e. 12 bytes for “struct test”.

 It is because sizeof operator works on “data type”, not data. And interestingly, it works for both internal types and language defined types. So, when you dereference a pointer inside sizeof; you are asking for the number of bytes pointee would need in the memory.

----CODE----

char *c;

// It would fetch you result as 1 byte.

sizeof(*c);

----CODE----

Remember that it is illegal to dereference a dangling pointer otherwise. In the mentioned code, you do not dereference the address carried by the pointer, and hence never get a classic segmentation fault error.

Hope you enjoyed these interesting facts of C. Please share your comments and suggestions.

----References----

[1] http://en.wikipedia.org/wiki/Offsetof

[2] http://en.wikipedia.org/wiki/Struct_(C_programming_language)

How to debug with GDB: Philosophy

Debugging is an art.

In general, debugging is the most admired tool in the developer's arsenal. My points in this article are related to GNU GDB debugger, for Linux operating system.

What is your first reaction when you see a core-dump?

Let's debug it!

Hold-on. Wait, and first ruminate over the problem. I'd suggest performing a few rituals before treading on to debugging would be helpful in saving time and effort.

o) Try to reproduce the problem. Ensure problem is consistent.

o) Check for the logs emitted by the application. If not available, see if you could get the logs.

o) Note down the application configuration, including the input, system configuration and every tiny detail that makes sense to you.

o) Okay, check the stack-frames with "backtrace".

o) You may inquire about frame locals and dig-in parent frames as well.

o) Also, it's useful to have a look at the code and a 2-3 level dry-run of code to reach at the point of failure. So, say you have a stack-trace as:
foo()
bar()
pop()
top() --> dumps core
hop()

Check how did we reach till top from say bar(). Look out for any ancillary function call in meanwhile and report its side-effect.

Remember, debugging is more than digging-deep; it's understanding the code tree and it's branches. Always approach to the problem on a high level, think what might have caused this failure.

Then, just jump in.

Publication in Linux For You, January, 2011: Python threading and its Caveats

My friend and I had got the article "Python threading and its Caveats" published in prestigious Linux for You magazine. The article is part of Jan, 2011 edition.

What a great treat of new year!

Best C coding practices: Saving many hours of efforts

C code has potential to be very compact, often at a price of losing comprehensibility and maintainability.

Let's discuss a few tips that might put smile on your face in a distress time.

o) Always save return value from a function, if available

if(!myfun()) {...}

This is very poor coding style. If myfun() can return one out of hundreds of non-zero error codes, it makes sense to save the return value.

if( (ret = myfun()) != 0 )

This value would help you in
o) zero down the point of return
o) debugging as "ret" is now part of the caller's frame and hence available for examination.

o) An extra log message would cause less harm than being savvy and avoiding it. It is tantamount to have enough log messages embedded in your code at critical points. Trust me, it makes your life so easy. Logging in itself is a vast topic and, I plan to take that separately.

o) Comments should be added generously. Explain what a file, function is doing. Explain the algorithm, input/output parameters of a function and return types.
If your function allocates dynamic memory, who bears the burden of freeing that memory, be explicit and mention that.

o) Keep a function "static" unless you need it in other file of the project. This saves linkers effort.

goto vs return statement: C programming

It is a classic conflict of opinion when comes to choosing between goto and return.

I feel that goto is treated unfairly, biased, and unjustly. Why people are so paranoid about avoiding goto, is beyond my comprehension.

Let me explain my point with following example:

void foo()
{
if{...}
if{...}
if{...}
if{...}
if{...}
if{...}
if{...}
if{...}
}

Now if I populate the body of "if" with "return"; it is perfectly okay. It creates problems in later point of time.

Now, say you allocate a resource in foo(), where would you free it?

Going with "return" makes you free the resource in all "if" conditions that follow the place where you have allocated the reosurce.

Now, solve this problem with goto. All "if" should goto a single label, which would be placed at the end of the foo().

So, the clean-up work can now be placed under this label. This is much better than "return" statements:

- Code is more exensible. You may add more "if" conditions, without any fear of resource leaks.

- Avoids code duplication

- Even if the current function do not need "goto", I propose to use "goto" for future extensibility.

The best virtualization software: VirtualBox v/s VMPlayer

I was an avid user of VirtualBox and was pretty happy with it. It gave me freedom to experiment with different OS in a rather safe, transparent, and convenient way. So I used to keep Linux as host and Windows as guest. Performance was good and bugs were non-existent.

Then I found VMPlayer. It surprised me a lot with following reasons:

a) It supported ISO images for Ubuntu, my fav Linux disto.

b) Set up was easier/comparable to VirtualBox.

c) Unity mode: What a feature! A seemless integration of guest OS with host application. You don't have to click the mouse, you never lose window focus. I can now browse through guest OS with Alt+Tab.

d) Very stable, good performance, and I am a happy customer.

How tail -f work?

"tail -f" is a special command in a way that it polls the specified file for any change and prints the new stuff on the fly. It is very helpful in observing logs and any event based data.

Ever wondered how tail achieves this?

"tail" opens the given file and obtains the file-descriptor. It opens it with xfreopen() -> freopen() -> fopen() call. It does its first round of fstat() on the file as well.

Once it has got the fd, it loops infinitely and do the following:

It does fstat() of the file and observes the mtime value. If the mtime value is changes from the last time..it dumps the data. To print the latest data, it lseek() the file to the last reported file size.

Source: http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/tail.c

How touchpad of a laptop work?

It was a curiosity to know the internal details of one of the most used 6x6 cm2 part of a laptop. My curiosity increased with diffrent behavior of the touchpad with different objects. For example, it works well with fingers but a pen, pin, or paper are not entertained.

So how does it work?

It is based on concept of capacitive difference on two parallel plates. Each of these plates have a grid of conductors. When we put our finger on the surface, it creates a charge difference on this surface and the difference of capacitance is sensed by the hardware.

This difference is mapped to mouse motion on our screen.

MeeGo: Marriage of Nokia and Intel

Nokia and Intel are merging their Linux based intiatives: Meamo and Moblin = Meego

* Open source platform to serve mobiles, notebook, TV, and tablets.
* Hosted under Linux Foundation
* Support x86 and ARM
* Visit http://meego.com for more information

But why would people use this new OS? What's wrong with Andriod? How is it better than Symbion, Android, Windows Mobile or BADA?

We already have Linux based mobile OS, so it'd be hard for this new guy to find a decent place.

Uninterrupted Linux session : screen command

Have you ever faced losing connections to a remote machine(e.g. from a putty) and you happened to be in middle of a script that took ten hours to complete. So what would you do??

Restart the script after re-connecting.

Not anymore...

Linux screen solves this problem with providing a terminal that runs on server and just exported to your putty/Terminal client. In simple terms, you are running your putty on remote machine and watching the output on your local client.

Now, if you client goes down, just chill!

Screen is running your script on remote machine.

To start a screen session:
- Login to remote machine with as you may wish.
- Run $screen
- Do you stuff

Power cut and, no net connection and your client is down.

Once power is back, just re-login to remote machine.
and issue:
$screen -r

It'll list all screen sessions running on the remote machine. Get attached to one of the session with:

$session -r

Now,

Expect Pexpect: Python module to help you automate interactive tests

Many a times we encounter writing a code that requires user input to proceed. Testing and validating such an application can become tedious and frustating. Python implements an elegant solution to cut down the effort needed to "talk to application".

This is called pexpect module, borrowed from Tcl "expect". Pexpect is a tool for controlling and automoating programs. It simple "fools" the application with an user input. It runs the program and monitors the output. When output matches a given pattern, it respond to application mimicking the human intervention.

Pexpect can be extensively used in testing and automation. I found it particularly useful for interacting with application like ssh, ftp, passwd, telnet etc.

It comes packaged with standard Ubuntu 9.10. The version of Python used by me is:

$python --version
Python 2.6.4rc2

Half century of technical know-how sharing

It's 50th post on this blog. I am happy with my contribution. Learning is a continuous process and I am an active participant. Today's post will talk of something that I had never imagined to do in my development career: Testing.

Testing of the product is an interesting job, especially when you are a new user. Past two weeks, I am testing the product I'd be developing.

I'm sharing my experience here:

- Don't test just for sake of doing it; Understand the ecosystem in which your product would be working.
- Take generous logs of your testing.
- Keep an eye on what you can improve: Be it a process or a feature.
- Try to automate processes if possible.
- Try feeding weird input to your product.
- Testing is learning. You become a very good user of your product.

Bash Command fo you

Once again, I am sharing a few good commands with you.

a) Run command in a subshell
$(cd /tmp && ls)

# You will see the contents of /tmp but will stay in PWD only.

b) Reusing cscope database
$cscope -d

c) If you default shell is csh, and you want to move to bash as soon as you login
- Create a file .cshrc in your home directory.
- Edit and add "bash"
- Save and exit.

d) vi can help you see a split screen with two or more files. Try :vsplit.

Ubuntu 9.10 on Windows XP: Virtual Box

My office laptop has Windows XP installed and I need Windows for sake of Outlook and Communicator. So I started to look out for alternative for having Linux on my laptop. There were three options suggested by different people:

- Get Ubuntu installed and run Windows XP as a guest on VirtualBox.
Now my workplace's IT guys do not install Windows on a VM, so they asked me to create an image of the Windows installed on my laptop. I searched for software to accomplish this and tried a few in vain. I tried a software called WinDD to copy Windows image to my portable HDD and since I did not checked thoroughly where it was copying, I got my portable HDD corrupted. It caused me a lot of pain, my important docs and many pics just got shredded.
Well that was it. I dropped this option.

- Get the Ubuntu on a pen-drive(PD) and make your PD boot-able.
It had a problem that Ubuntu installation was not persistent. Many of my setting could not be saved. Though I got a way to create a persistent image, still it can't save a few things like system time. Verdict: Dropped

- Get the Ubuntu running as a guest on host Windows.
After installing the guest additions, I am very happy with the screen resolution. I got few glitches with network setting and shared folders help. I got over them and will post the details in the next post.

So I am a happy man now, seeing my Ubuntu running on XP. All happy and gay!

PS: I am a fan of VirtualBox and so far have not used any other VM software.

Torrent: Demystified

What is a torrent?

A torrent is like a broker between a customer and a seller. It works in a peer-to-peer setup that means a true democracy; all are equal, no master-no slave.

So you need to download a classic movie, which may be on a computer(s), located anywhere across the globe.

How would you know where is it?

Torrent comes to help you here. You google for a torrent of this movie. Download the torrent and voila!.

Torrent has three primary components:

a) Seeders: People that have a copy of the movie you want
b) Leechers: That's you!
c) Torrent file : It has information about seeders and leechers.

While you are downloading your movie, torrent share the downloaded part with anyone else interested in downloading the same movie. So effectively every leecher becomes a seeder. Now you are even! Hail Democracy!!