Profiling tool - How to use OProfile ?

OProfile can collect event information for the whole system in the background with very little overhead.
OProfile mainly consists of the opcontrol shell script and the oprofiled daemon.

The opcontrol script is used for configuring, starting, and stopping a profiling session, which are then recorded into sample files by oprofiled. And opreport is used to analyze these results.

Profiling setup parameters that you specify using opcontrol are cached in /root/.oprofile/daemonrc. Subsequent runs of opcontrol --start will continue to use these cached values until you override them with new values.

Creating profile Report:

Setup:
If you don't want to profile kernal we need to exclude it from the oprofile.
to do this we need to set --no-vimlinux to tell OProfile you don't have a vmlinux file

opcontrol --no-vmlinux

By default profile sample data is written to /var/lib/oprofile/samples. we can change this path using session-dir flag.

opcontrol --no-vmlinux --session-dir=/root/tmpsession

Run:
Now we need to start the daemon(oprofiled) which will collect the profile data.

opcontrol --start

Run your program. if it is an infinite process let it run in background and profile for as long as a time as possible.

Collect:
To collect that session profile data you can dump it using the command

opcontrol --dump

This ensures that any profile data collected by the oprofiled daemon has been flusehd to disk.(default: /var/lib/oprofile/samples) 

we can even save the session to the different files using --save flag
opcontrol --save <session_dir_name1>
opcontrol --save <session_dir_name2>

Stop:
When you want to stop profiling, you can do so with :

opcontrol --stop

But this will not kill the daemon this will give SIGHUP to oprofiled to stop collecting profile data.

to kill the daemon do this

opcontrol --shutdown

Analyzing profile data:

opreport is the tool that can be used to analyze the collected profile data.

opreport --long-filenames will show the whole summary of the system.

to analyze particular process during the specific time or session use

opreport -lg <process path> session:<saved session> 

opreport -lg /opt/bin/my_test_pgm session:saved_session_dir

-g : Show source file and line for each symbol

-l : List per-symbol information instead of a binary image summary

Comparing two profiles:

Often, we'd like to compare two profiles. when analyzing the performance of an application, we'd like to make code changes and examine the effect of the change. This is supported in opreport by giving a profile specification that identifies two different profiles.

opreport <shared-spec> { <first-profile> } { <second-profile> }

opreport  /opt/bin/my_test_pgn  { ./saved_session_1 } {} #empty brasses refers to current session.
or
opreport  /opt/bin/my_test_pgn  { ./saved_session_1 } { ./current_session }

 

This will create a %diff report.

Read More

Unix command line tricks

1. Ever tried creating pdf files in your unix machine ??

man -t ascii | ps2pds - > ascii_man_page.pdf

-t flag formats the output and ps2pds converts post script to pdf.
hyphen (-) after ps2pdf command indicates it to read input from stdin and write output to stdout.


2. Want to terminate the script after first failure ?

'set -e' thats it. it does everything.

$> cat setexit.sh

#/bin/bash
set -e
echo "line 1"
cat tmp.x.12.12
echo "line 2"


$> ./setexit.sh
line 1
cat: tmp.x.12.12: No such file or directory #line 2 is not echoed

3. Simple way to create huge file

yes, we have 'yes' in unix to do that. yes will echo what ever you give as argument.

#press ctr+c after a while
$yes "some random text..." > bigfile.txt

'yes' has another interesting advantage. just imagine to answer yes to the every question by bash.

$rm file.txt
rm: remove regular file 'file.txt' ?

#you probably give 'y' to remove it. what if you want to remove 100 such files?

yes | rm *.sh
  

though this can be achieved just by ignoring alias ( \rm *.sh ) this is just an example to demonstrate the usage.


4. Creating dirty random number

we have $RANDOM shell variable to generate a random number may be max of 5 characters. If you wish to create a big random number use 'mcookie'. This command generates a "magic cookie," a 128-bit pseudorandom hexadecimal number, normally used as an authorization signature by the X server. This also available for use in a script as a quick and dirty random number.

$mcookie
a2cae14a060720bca183d05f165a5a8e


5. Get directory name or file name from path and file extension

#extract dir name and file name
$> basename /dir1/dir2/file.txt      
file.txt  
$> dirname  /dir1/dir2/file.txt 
/dir1/dir2

#Extract extension 
$> path="/dir1/dir2/file.txt"
$> ext=${path##*.}
$> echo $ext
txt

6. Simple 'cut'

var="one two three"
set -- $var
a=${1};   #one 
b=${2};   #two
c=${3};   #three

Just a basic example to avoid invoking forked process (cut/awk)


7. Base conversion

'bc' can convert from any base to any.

Ex: decimal to hexadecimal conversion
echo 'obase=16; ibase=10; 64' | bc            # (ibase=input base, obase=output base)

This will convert 64 from base 10 to base 16.

Read More

Programmable Completion : Autocomplete the arguments of your program

Auto completion is one of the best features in bash. How good if it completes argument for your program as well !!

The Bash complete and compgen builtins make it possible for tab completion to recognize partial parameters and options to commands. In a very simple case, we can use complete from the command-line to specify a short list of acceptable parameters.

with complete [option] [command] you define the completion for some command.

A basic example to illustrate auto completion:

lets think we have a script called 'run.sh' which will take 4 arguments --all, --help, --html, --version.
as you type run.sh -[TAB] it should display all the above four option.
to make it work lets write a small function will will be loaded by complete when ever TAB is pressed.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

# file: load_opt _load_opt () { local cur COMPREPLY=() cur=${COMP_WORDS[COMP_CWORD]} if [[ ${cur} == -* ]] then COMPREPLY=($(compgen -W "--all --help --html --version" -- $cur)) return 0 fi } complete -F _load_opt run.sh

save the file as load_opt.  and source it before running run.sh. then it shows the magic.

$  source load_opt

$  run.sh -[TAB][TAB]
--all --help --html --version

$ run.sh --h[TAB][TAB]
--help --html

How it works ?

Once the load_opt script is sourced, bash understands that run.sh arguments will be completed bye load_opt function which is specified by complete command at the end of load_opt file.
so when ever TAB is pressed  _load_opt function will be called.

This script has some special shell variables.

COMP_WORDS

An array variable consisting of the individual words in the current command line.

COMP_CWORD

An index into ${COMP_WORDS} of the word containing the current cursor position.

COMPREPLY

An array variable from which Bash reads the possible completions generated by a shell function invoked by the programmable completion facility

a local 'cur' variable is used to point current completion word.

so if your option starts with - it will match -* and set the value of COMPREPLY to the output of compgen [ check here ] command. compgen will return the matching words of 'cur'
and those set of words are assigned to COMPLREPLY array.
bash will now print those word to console.


But it is not a good idea of sourcing the loader before running the script.
so create this load_opt file under /etc/bash_completion.d/ directory. all the scripts which are under  /etc/bash_completion.d/ will be loaded by bash in the start up.
After adding the file under that directory you can run bash then run the script or you can source that file alone and run the script.


Note:
above example work if you run your script run.sh[TAB] not ./run.sh[TAB]. if you want it to work with ./ as well add 'complete -F _load_opt ./run.sh' also at the end of load_opt file.

Read More

compgen - List All Unix Commands !!

compgen is bash built-in command and it will show all available commands, aliases, and functions.
and it is used to get the specific set of words from a word list ( ?? )

To list all the commands available :

$ compgen -c
alert
c
d
egrep
fgrep
..
..
..
bzexe
espdiff
sol
gnome-mines
gnome-sudoku

To list all the bash shell aliases available :

$ compgen -a
alert
c
d
egrep
fgrep
grep
l
la

To list all the bash build-ins :

$ compgen -b
.
:
[
alias
bg
bind
break
builtin
caller
cd

To list all the bash keywords :

$ compgen -k
if
then
else
elif
fi
case
esac
for
select
while
until
do
done
in
function
time
{
}
!
[[
]]
coproc

To list all the bash functions :

dinesh@ubuntu:~$ compgen -A function
__colormgrcomp
__expand_tilde_by_ref
__get_cword_at_cursor_by_ref
__grub_dir
__grub_get_last_option
..
..
_zeitgeist_daemon
command_not_found_handle
dequote
quote
quote_readline

-A (action) is very useful option. The action may be one of the following to generate list of possible completions.
alias, arrayvar, binding, builtin, command, directory, enabled, disabled, export, file, function, group, hostname, job, keyword, running, service, setopt, signal, user, variable.


To get word[s] from word-list :

dinesh@ubuntu:$ compgen -W "aa ab ac ba bb"
aa
ab
ac
ba
bb
dinesh@ubuntu:$ compgen -W "aa ab ac ba bb" -- b
ba
bb

is it really useful ??☺

Read More

Difference between array[@] and array[*] in shell

It is important for quoting and spacing that the ${array[@]}
format (rather than ${array[*]}) is used, and also that double quotes are put around the whole construct. 

items=(pen "permanent marker" pencil "temporary marker")

 
for item in ${items[@]}
do
  echo "Item is : $item"
done

Item is : pen
Item is : permanent
Item is : marker
Item is : pencil
Item is : temporary
Item is : marker

for item in ${items[*]}
do
  echo "Item is : $item"
done

Item is : pen
Item is : permanent
Item is : marker
Item is : pencil
Item is : temporary
Item is : marker


items[@] and items[*] produced the same output and which is not expected. "permanent" and "marker" are treated as two different words in both the cases though it is a single word. lets try now with the quotes around the array..

items=(pen "permanent marker" pencil "temporary marker")

for item in "${items[@]}"
do
  echo "Item is : $item"
done

Item is : pen
Item is : permanent marker
Item is : pencil
Item is : temporary marker
 
 YES this what we expected.... but items[*] will not behave this way.

for item in "${items[*]}"
do
  echo "Item is : $item"
done

Item is : pen permanent marker pencil temporary marker


the * is not suitable either with or without quotes. Without quotes, it does the same as the @ symbol. With quotes, the whole array is boiled down into a single string.

Read More

A Bit Close Look At Vi - More On 'registers'

Note - read Marks & Registers before reading this post.

There are some other ways of copy or delete text in vi.

Working with paragraphs:

If you are in paragraph text you can use  }  and  {  to move  cursor to the beginning or ending of the paragraph respectively.  so if you want to move a paragraph you can use  {  and  d} . 
and   p  to paste the content.  if you are already in beginning or ending of a paragraph you can use   d}  and  d{  respectively to cut the paragraph.
If you want to delete 3 continuous paragraphs you can use  d3} (From current to next 3 paragraphs will be cut) 
And if you want to move multiple paragraphs from multiple locations to other locations we can make use of registers ( Marks & Registers ).

Ex:
I am at para 1: hit   "ad3}  to cut 3 paragraphs  to the register 'a'
I am at para 9: hit   "by}  to copy next paragraph to the register 'b' 
At EOF: hit  "ap  and  "bp   to paste the paragraphs.

Copy/Cut by Search:

Search using  /  or  ?  is most notable technique in vi. But if you want to copy the content from current line to the next line containing a specific word ??.
for example one can use  y/foo  to copy the text from current line to next line containing the word 'foo'. similarly 'd' can be used to cut ( d/foo).
y?foo to copy from the current line to the most recent (previous) line containing 'bar'.

Similarly as above we can use registers to copy/cut these content.
Thus if you use "ay/foo then you are yanking a copy of the text from here to the next line containing "foo" into the 'a' register. and can use   "ap  to paste the content.

Read More

Vi - Marks and Registers

Marks:

vi has 26 "marks". A mark is set to any cursor location using the m command. 
Each mark is designated by a letter. 
Thusma sets the 'a' mark to the current location, andms sets the 's' mark. 
 You can move to the line containing a mark using the ' (single quote) command. 
 Thus 'a moves to the beginning of the line containing the 'a' mark. 
 You can move to the precise location of any mark using the ` (backquote) command. 
 Thus `s will move directly to the exact location of the 's' mark.

copy/cut/paste :

One can use d,y,p to delete,yank,paste respectively. 
step 1 : set the mark by using m
step 2 : y'a to copy from current position to line where mark a is set.
             y`a to copy from current position to exact location of mark a.
             [ 'd' can be used similarly ]
step 3 : use p to paste the content.

To access all currently defined markers hit :makrs

some other interesting features are there in vi where we can use marks.

# for example if you want to sort a set of lines in vi what you do ?? There is a simple solution for it.
place two marks on two lines. and then you you hit :'a,'b!sort ( ! will execute shell commands on vi)

# if you want to redirect a part of file to another file
place two marks on two lines and then :'a,'b w temp_file. this will copy the content from mark a to b to the new temp_file. if file already exist use :'a,'b w>>temp_file to append.
  
Note: a-z can be used to mark only in same file. use A-Z to mark across the files.. So now copy a line from one file to other file is much more easier !! 

Registers:

vi has 26 "registers". we can use these registers by prefixing  " (double quote)Each register is designated by a letter.
Thus"a starts the register. now its up to you to use this register to copy the content to it or cut the content to it or paste the content from it.

This is more useful when you want to copy lines from multiple locations and paste it in different places.

copy/cut/paste :
Ex:

I am at line 2: "ayy will copy the current line to register to a
I go to line 30:  "bdd will cut the current line to register b.
I am at the end of file : "ap  and "bp  will paste line 2 and line 30 at end of file.

"ap pastes a copy of the 'a' register's contents into the text after the cursor and "aP pastes a copy from 'a' to before the current line.

capital letters can be used to append the content to a register. 
 "Ayy will append the current line to the register A.

To access all currently defined registers hit :reg

We can do some interesting things with registers.
for example if you want to execute a command which is at current line in vi what you do ?? There is a simple solution for it.
Copy the line you want to execute to a register -  "ayy 
Execute the content of register using '@'  -  :@a 

Read More

Shell Special variables and TEST flags

Here are some shell special variables:

Special Shell Variables:

Variable	Meaning
$0	Filename of script
$1	Positional parameter #1
$2 - $9	Positional parameters #2 - #9
${10}	Positional parameter #10
$#	Number of positional parameters
"$*"	All the positional parameters (as a single word) *
"$@"	All the positional parameters (as separate strings)
${#*}	Number of positional parameters
${#@}	Number of positional parameters
$?	Return value
$$	Process ID (PID) of script
$-	Flags passed to script (using set)
$_	Last argument of previous command
$!	Process ID (PID) of last job run in background

TEST Operators: Files

Operator	Tests Whether	-----	Operator	Tests Whether
-e	File exists		-s	File is not zero size
-f	File is a regular file
-d	File is a directory		-r	File has read permission
-h	File is a symbolic link		-w	File has write permission
-L	File is a symbolic link		-x	File has execute permission
-b	File is a block device
-c	File is a character device		-g	sgid flag set
-p	File is a pipe		-u	suid flag set
-S	File is a socket		-k	"sticky bit" set
-t	File is associated with a terminal
-N	File modified since it was last read		F1 -nt F2	File F1 is newer than F2 *
-O	You own the file		F1 -ot F2	File F1 is older than F2 *
-G	Group id of file same as yours		F1 -ef F2	Files F1 and F2 are hard links to the same file *
!	NOT (inverts sense of above tests)

* Binary operator (requires two operands).

from: http://www.tldp.org/LDP/abs/html/refcards.html

Read More

Parameters/String Operations on shell (Bash) without using any third party tools

Parameter Substitution and Expansion:

Expression	Meaning
${var}	Value of var (same as $var)
${var-DEFAULT}	If var not set, evaluate expression as $DEFAULT *
${var:-DEFAULT}	If var not set or is empty, evaluate expression as $DEFAULT *
${var=DEFAULT}	If var not set, evaluate expression as $DEFAULT * (value is assigned to var)
${var:=DEFAULT}	If var not set, or is empty, evaluate expression as $DEFAULT * (value is assigned to var)
${var+OTHER}	If var set, evaluate expression as $OTHER, otherwise as null string
${var:+OTHER}	If var set, evaluate expression as $OTHER, otherwise as null string
${var?ERR_MSG}	If var not set, print $ERR_MSG and abort script with an exit status of 1.*
${var:?ERR_MSG}	If var not set, or is empty, print $ERR_MSG and abort script with an exit status of 1.*
${!varprefix*}	Matches all previously declared variables beginning with varprefix
${!varprefix@}	Matches all previously declared variables beginning with varprefix

* If var is set, evaluate the expression as $var with no side-effects.

String Operations:

Expression	Meaning
${#string}	Length of $string
${string:position}	Extract substring from $string at $position
${string:position:length}	Extract $length characters substring from $string at $position [zero-indexed, first character is at position 0]
${string#substring}	Strip shortest match of $substring from front of $string
${string##substring}	Strip longest match of $substring from front of $string
${string%substring}	Strip shortest match of $substring from back of $string
${string%%substring}	Strip longest match of $substring from back of $string
${string/substring/replacement}	Replace first match of $substring with $replacement
${string//substring/replacement}	Replace all matches of $substring with $replacement
${string/#substring/replacement}	If $substring matches front end of $string, substitute $replacement for $substring
${string/%substring/replacement}	If $substring matches back end of $string, substitute $replacement for $substring
expr match "$string" '$substring'	Length of matching $substring* at beginning of $string
expr "$string" : '$substring'	Length of matching $substring* at beginning of $string
expr index "$string" $substring	Numerical position in $string of first character in $substring* that matches [0 if no match, first character counts as position 1]
expr substr $string $position $length	Extract $length characters from $string starting at $position [0 if no match, first character counts as position 1]
expr match "$string" '\($substring\)'	Extract $substring*, searching from beginning of $string
expr "$string" : '\($substring\)'	Extract $substring* , searching from beginning of $string
expr match "$string" '.*\($substring\)'	Extract $substring*, searching from end of $string
expr "$string" : '.*\($substring\)'	Extract $substring*, searching from end of $string

* Where $substring is a Regular Expression.

Refer http://itsjustsosimple.blogspot.in/2013/02/all-about-variablesparameters-in-shell.htmlfor few examples.

Read More

Reading environment variables in python

The os module contains an interface to operating system-specific functions. this module can be used to access environment variables. 

We can go with os.environ to get the value of environment variable.

import os
print os.environ['HOME']

but there is a catch, this method will raise KeyError variable does not exist 

>>> print os.environ['HOME_NOT_EXIST']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/UserDict.py", line 23, in __getitem__
    raise KeyError(key)
KeyError: 'HOME_NOT_EXIST'




So it is better to go with os.getenv. this return None if key/environment variable does not exist. and if we require we can return default values too.




print os.getenv('KEY')    #returns None if KEY doesn't exist

print os.getenv('KEY', 0) #will return 0 if KEY doesn't exist 

Read More