Python - Thread synchronization - using Thread Events

We may often need to signal the thread to do some special task. Simple solution is to create events.

Definition from python docs:
threading.Event()
    A factory function that returns a new event object. An event manages a flag that can be set to true with the set() method and reset to false with the clear() method. The wait() method blocks until the flag is true 

the event represents an internal flag, and threads can wait for the flag to be set, or set or clear the flag themselves. The methods provided are very much self explanatory: set() clear() isSet() wait(timeout). 

This is a simple mechanism. A thread signals an event and the other thread(s) wait for it.

In the following example we are crating an event and a thread,  and passing that event to the thread function. that thread function is waiting on that event. if user press ctrl+c then it is caught in the main program by keyboardInterrup and we are setting the stop event. This even cause the while in the thread function to break and exits the thread.

#!/usr/bin/python
import threading,time

def collectstats(event):
    while not event.is_set():
          event.wait(1)
    print "event set.. thread exiting.."

stop_event = threading.Event()
th = threading.Thread(target=collectstats, args=(stop_event,))
th.start()

while True:
    try:
        time.sleep(100)
    except KeyboardInterrupt as e:
        stop_event.set()
        th.join()
        break;

print "in main: exiting.."

if we run the program, here is the output

# ./even.py 

^Cevent set.. thread exiting..
in main: exiting..

Any number of threads can wait on same event and can set same event. once the even is set it will notify all the threads ( equivalent to notifyall() method )

Read More

Python - Signal handling and identifying stack frame

Signals are identified by integers and are defined in the operating system C headers. Python exposes the signals appropriate for the platform as symbols in the 'signal' module. 

the signal module in python is used to install your own signal handlers.  When the interpreter sees a signal, the signal handler associated with that signal is executed as soon as possible.

signal handler is a call back function, the arguments to the signal handler are the signal number and the stack frame from the point in your program that was interrupted by the signal. 

First basic example:

import signal,time

def signal_handler(signum, stack):
    print 'Received:', signum

signal.signal(signal.SIGHUP, signal_handler)
signal.signal(signal.SIGINT, signal_handler)

signal.signal(15, signal_handler)
while True:
    print 'Waiting...try kill using signal 1(SIGHUP) or 2(SIGINT)'
    time.sleep(3)

Linux Standard signals:
from signal man page:
       Signal     Value     Action   Comment
       -------------------------------------------------------------------------
       SIGHUP        1       Term    Hangup detected on controlling terminal or death of controlling process
       SIGINT        2        Term    Interrupt from keyboard
       SIGQUIT       3      Core    Quit from keyboard
       SIGILL        4        Core    Illegal Instruction
       SIGABRT       6      Core    Abort signal from abort(3)
       SIGFPE        8       Core    Floating point exception
       SIGKILL       9       Term    Kill signal
       SIGSEGV      11     Core    Invalid memory reference
       SIGPIPE      13      Term    Broken pipe: write to pipe with no readers
       SIGALRM      14       Term    Timer signal from alarm(2)
       SIGTERM      15       Term    Termination signal
       SIGUSR1   30,10,16    Term    User-defined signal 1
       SIGUSR2   31,12,17    Term    User-defined signal 2
       SIGCHLD   20,17,18    Ign     Child stopped or terminated
       SIGCONT   19,18,25    Cont    Continue if stopped
       SIGSTOP   17,19,23    Stop    Stop process
       SIGTSTP   18,20,24    Stop    Stop typed at tty
       SIGTTIN   21,21,26    Stop    tty input for background process
       SIGTTOU   22,22,27    Stop    tty output for background process

The signals SIGKILL and SIGSTOP cannot be caught, blocked, or ignored. 

How to trap all the signals (except few..!!) ??
here is the example: 

#!/usr/bin/python
import signal

def sighandler(signum, frame):
 print "Caough signal :", signum

for x in dir(signal):
  if x.startswith("SIG"):
     try:
        signum = getattr(signal,x)
        signal.signal(signum,sighandler)
     except:
        print "Skipping %s"%x
   
while True:
      pass

Printing stack frames :

The frame argument is the stack frame also known as execution frame. It point to the frame that was interrupted by the signal. to print the stack trace you need to use 'traceback' module. 
this is very useful in multi-threaded program because any thread might be interrupted by a signals and signal is only received by the main program. 

#!/usr/bin/python
import signal,traceback

def sighandler(signum, frame):
 print "Caough signal :", signum
 traceback.print_stack(frame)

for sig in dir(signal):
  if sig.startswith("SIG"):
     try:
        signum = getattr(signal,sig)
        signal.signal(signum,sighandler)
     except:
        print "Skipping %s"%sig
   
while True:
      pass

Ignoring signals:

To ignore a signal, register SIG_IGN as the handler (no call back required). This will ignore the ctr+c signal raised from terminal.

#!/usr/bin/python
import signal

signal.signal(signal.SIGINT,signal.SIG_IGN)

while True:
 pass

Read More

Profiling tool - How to use OProfile ?

OProfile can collect event information for the whole system in the background with very little overhead.
OProfile mainly consists of the opcontrol shell script and the oprofiled daemon.

The opcontrol script is used for configuring, starting, and stopping a profiling session, which are then recorded into sample files by oprofiled. And opreport is used to analyze these results.

Profiling setup parameters that you specify using opcontrol are cached in /root/.oprofile/daemonrc. Subsequent runs of opcontrol --start will continue to use these cached values until you override them with new values.

Creating profile Report:

Setup:
If you don't want to profile kernal we need to exclude it from the oprofile.
to do this we need to set --no-vimlinux to tell OProfile you don't have a vmlinux file

opcontrol --no-vmlinux

By default profile sample data is written to /var/lib/oprofile/samples. we can change this path using session-dir flag.

opcontrol --no-vmlinux --session-dir=/root/tmpsession

Run:
Now we need to start the daemon(oprofiled) which will collect the profile data.

opcontrol --start

Run your program. if it is an infinite process let it run in background and profile for as long as a time as possible.

Collect:
To collect that session profile data you can dump it using the command

opcontrol --dump

This ensures that any profile data collected by the oprofiled daemon has been flusehd to disk.(default: /var/lib/oprofile/samples) 

we can even save the session to the different files using --save flag
opcontrol --save <session_dir_name1>
opcontrol --save <session_dir_name2>

Stop:
When you want to stop profiling, you can do so with :

opcontrol --stop

But this will not kill the daemon this will give SIGHUP to oprofiled to stop collecting profile data.

to kill the daemon do this

opcontrol --shutdown

Analyzing profile data:

opreport is the tool that can be used to analyze the collected profile data.

opreport --long-filenames will show the whole summary of the system.

to analyze particular process during the specific time or session use

opreport -lg <process path> session:<saved session> 

opreport -lg /opt/bin/my_test_pgm session:saved_session_dir

-g : Show source file and line for each symbol

-l : List per-symbol information instead of a binary image summary

Comparing two profiles:

Often, we'd like to compare two profiles. when analyzing the performance of an application, we'd like to make code changes and examine the effect of the change. This is supported in opreport by giving a profile specification that identifies two different profiles.

opreport <shared-spec> { <first-profile> } { <second-profile> }

opreport  /opt/bin/my_test_pgn  { ./saved_session_1 } {} #empty brasses refers to current session.
or
opreport  /opt/bin/my_test_pgn  { ./saved_session_1 } { ./current_session }

 

This will create a %diff report.

Read More

Unix command line tricks

1. Ever tried creating pdf files in your unix machine ??

man -t ascii | ps2pds - > ascii_man_page.pdf

-t flag formats the output and ps2pds converts post script to pdf.
hyphen (-) after ps2pdf command indicates it to read input from stdin and write output to stdout.


2. Want to terminate the script after first failure ?

'set -e' thats it. it does everything.

$> cat setexit.sh

#/bin/bash
set -e
echo "line 1"
cat tmp.x.12.12
echo "line 2"


$> ./setexit.sh
line 1
cat: tmp.x.12.12: No such file or directory #line 2 is not echoed

3. Simple way to create huge file

yes, we have 'yes' in unix to do that. yes will echo what ever you give as argument.

#press ctr+c after a while
$yes "some random text..." > bigfile.txt

'yes' has another interesting advantage. just imagine to answer yes to the every question by bash.

$rm file.txt
rm: remove regular file 'file.txt' ?

#you probably give 'y' to remove it. what if you want to remove 100 such files?

yes | rm *.sh
  

though this can be achieved just by ignoring alias ( \rm *.sh ) this is just an example to demonstrate the usage.


4. Creating dirty random number

we have $RANDOM shell variable to generate a random number may be max of 5 characters. If you wish to create a big random number use 'mcookie'. This command generates a "magic cookie," a 128-bit pseudorandom hexadecimal number, normally used as an authorization signature by the X server. This also available for use in a script as a quick and dirty random number.

$mcookie
a2cae14a060720bca183d05f165a5a8e


5. Get directory name or file name from path and file extension

#extract dir name and file name
$> basename /dir1/dir2/file.txt      
file.txt  
$> dirname  /dir1/dir2/file.txt 
/dir1/dir2

#Extract extension 
$> path="/dir1/dir2/file.txt"
$> ext=${path##*.}
$> echo $ext
txt

6. Simple 'cut'

var="one two three"
set -- $var
a=${1};   #one 
b=${2};   #two
c=${3};   #three

Just a basic example to avoid invoking forked process (cut/awk)


7. Base conversion

'bc' can convert from any base to any.

Ex: decimal to hexadecimal conversion
echo 'obase=16; ibase=10; 64' | bc            # (ibase=input base, obase=output base)

This will convert 64 from base 10 to base 16.

Read More

Programmable Completion : Autocomplete the arguments of your program

Auto completion is one of the best features in bash. How good if it completes argument for your program as well !!

The Bash complete and compgen builtins make it possible for tab completion to recognize partial parameters and options to commands. In a very simple case, we can use complete from the command-line to specify a short list of acceptable parameters.

with complete [option] [command] you define the completion for some command.

A basic example to illustrate auto completion:

lets think we have a script called 'run.sh' which will take 4 arguments --all, --help, --html, --version.
as you type run.sh -[TAB] it should display all the above four option.
to make it work lets write a small function will will be loaded by complete when ever TAB is pressed.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

# file: load_opt _load_opt () { local cur COMPREPLY=() cur=${COMP_WORDS[COMP_CWORD]} if [[ ${cur} == -* ]] then COMPREPLY=($(compgen -W "--all --help --html --version" -- $cur)) return 0 fi } complete -F _load_opt run.sh

save the file as load_opt.  and source it before running run.sh. then it shows the magic.

$  source load_opt

$  run.sh -[TAB][TAB]
--all --help --html --version

$ run.sh --h[TAB][TAB]
--help --html

How it works ?

Once the load_opt script is sourced, bash understands that run.sh arguments will be completed bye load_opt function which is specified by complete command at the end of load_opt file.
so when ever TAB is pressed  _load_opt function will be called.

This script has some special shell variables.

COMP_WORDS

An array variable consisting of the individual words in the current command line.

COMP_CWORD

An index into ${COMP_WORDS} of the word containing the current cursor position.

COMPREPLY

An array variable from which Bash reads the possible completions generated by a shell function invoked by the programmable completion facility

a local 'cur' variable is used to point current completion word.

so if your option starts with - it will match -* and set the value of COMPREPLY to the output of compgen [ check here ] command. compgen will return the matching words of 'cur'
and those set of words are assigned to COMPLREPLY array.
bash will now print those word to console.


But it is not a good idea of sourcing the loader before running the script.
so create this load_opt file under /etc/bash_completion.d/ directory. all the scripts which are under  /etc/bash_completion.d/ will be loaded by bash in the start up.
After adding the file under that directory you can run bash then run the script or you can source that file alone and run the script.


Note:
above example work if you run your script run.sh[TAB] not ./run.sh[TAB]. if you want it to work with ./ as well add 'complete -F _load_opt ./run.sh' also at the end of load_opt file.

Read More

compgen - List All Unix Commands !!

compgen is bash built-in command and it will show all available commands, aliases, and functions.
and it is used to get the specific set of words from a word list ( ?? )

To list all the commands available :

$ compgen -c
alert
c
d
egrep
fgrep
..
..
..
bzexe
espdiff
sol
gnome-mines
gnome-sudoku

To list all the bash shell aliases available :

$ compgen -a
alert
c
d
egrep
fgrep
grep
l
la

To list all the bash build-ins :

$ compgen -b
.
:
[
alias
bg
bind
break
builtin
caller
cd

To list all the bash keywords :

$ compgen -k
if
then
else
elif
fi
case
esac
for
select
while
until
do
done
in
function
time
{
}
!
[[
]]
coproc

To list all the bash functions :

dinesh@ubuntu:~$ compgen -A function
__colormgrcomp
__expand_tilde_by_ref
__get_cword_at_cursor_by_ref
__grub_dir
__grub_get_last_option
..
..
_zeitgeist_daemon
command_not_found_handle
dequote
quote
quote_readline

-A (action) is very useful option. The action may be one of the following to generate list of possible completions.
alias, arrayvar, binding, builtin, command, directory, enabled, disabled, export, file, function, group, hostname, job, keyword, running, service, setopt, signal, user, variable.


To get word[s] from word-list :

dinesh@ubuntu:$ compgen -W "aa ab ac ba bb"
aa
ab
ac
ba
bb
dinesh@ubuntu:$ compgen -W "aa ab ac ba bb" -- b
ba
bb

is it really useful ??☺

Read More

Difference between array[@] and array[*] in shell

It is important for quoting and spacing that the ${array[@]}
format (rather than ${array[*]}) is used, and also that double quotes are put around the whole construct. 

items=(pen "permanent marker" pencil "temporary marker")

 
for item in ${items[@]}
do
  echo "Item is : $item"
done

Item is : pen
Item is : permanent
Item is : marker
Item is : pencil
Item is : temporary
Item is : marker

for item in ${items[*]}
do
  echo "Item is : $item"
done

Item is : pen
Item is : permanent
Item is : marker
Item is : pencil
Item is : temporary
Item is : marker


items[@] and items[*] produced the same output and which is not expected. "permanent" and "marker" are treated as two different words in both the cases though it is a single word. lets try now with the quotes around the array..

items=(pen "permanent marker" pencil "temporary marker")

for item in "${items[@]}"
do
  echo "Item is : $item"
done

Item is : pen
Item is : permanent marker
Item is : pencil
Item is : temporary marker
 
 YES this what we expected.... but items[*] will not behave this way.

for item in "${items[*]}"
do
  echo "Item is : $item"
done

Item is : pen permanent marker pencil temporary marker


the * is not suitable either with or without quotes. Without quotes, it does the same as the @ symbol. With quotes, the whole array is boiled down into a single string.

Read More