Shell Special variables and TEST flags

Here are some shell special variables:

Special Shell Variables:

Variable	Meaning
$0	Filename of script
$1	Positional parameter #1
$2 - $9	Positional parameters #2 - #9
${10}	Positional parameter #10
$#	Number of positional parameters
"$*"	All the positional parameters (as a single word) *
"$@"	All the positional parameters (as separate strings)
${#*}	Number of positional parameters
${#@}	Number of positional parameters
$?	Return value
$$	Process ID (PID) of script
$-	Flags passed to script (using set)
$_	Last argument of previous command
$!	Process ID (PID) of last job run in background

TEST Operators: Files

Operator	Tests Whether	-----	Operator	Tests Whether
-e	File exists		-s	File is not zero size
-f	File is a regular file
-d	File is a directory		-r	File has read permission
-h	File is a symbolic link		-w	File has write permission
-L	File is a symbolic link		-x	File has execute permission
-b	File is a block device
-c	File is a character device		-g	sgid flag set
-p	File is a pipe		-u	suid flag set
-S	File is a socket		-k	"sticky bit" set
-t	File is associated with a terminal
-N	File modified since it was last read		F1 -nt F2	File F1 is newer than F2 *
-O	You own the file		F1 -ot F2	File F1 is older than F2 *
-G	Group id of file same as yours		F1 -ef F2	Files F1 and F2 are hard links to the same file *
!	NOT (inverts sense of above tests)

* Binary operator (requires two operands).

from: http://www.tldp.org/LDP/abs/html/refcards.html

Read More

Parameters/String Operations on shell (Bash) without using any third party tools

Parameter Substitution and Expansion:

Expression	Meaning
${var}	Value of var (same as $var)
${var-DEFAULT}	If var not set, evaluate expression as $DEFAULT *
${var:-DEFAULT}	If var not set or is empty, evaluate expression as $DEFAULT *
${var=DEFAULT}	If var not set, evaluate expression as $DEFAULT * (value is assigned to var)
${var:=DEFAULT}	If var not set, or is empty, evaluate expression as $DEFAULT * (value is assigned to var)
${var+OTHER}	If var set, evaluate expression as $OTHER, otherwise as null string
${var:+OTHER}	If var set, evaluate expression as $OTHER, otherwise as null string
${var?ERR_MSG}	If var not set, print $ERR_MSG and abort script with an exit status of 1.*
${var:?ERR_MSG}	If var not set, or is empty, print $ERR_MSG and abort script with an exit status of 1.*
${!varprefix*}	Matches all previously declared variables beginning with varprefix
${!varprefix@}	Matches all previously declared variables beginning with varprefix

* If var is set, evaluate the expression as $var with no side-effects.

String Operations:

Expression	Meaning
${#string}	Length of $string
${string:position}	Extract substring from $string at $position
${string:position:length}	Extract $length characters substring from $string at $position [zero-indexed, first character is at position 0]
${string#substring}	Strip shortest match of $substring from front of $string
${string##substring}	Strip longest match of $substring from front of $string
${string%substring}	Strip shortest match of $substring from back of $string
${string%%substring}	Strip longest match of $substring from back of $string
${string/substring/replacement}	Replace first match of $substring with $replacement
${string//substring/replacement}	Replace all matches of $substring with $replacement
${string/#substring/replacement}	If $substring matches front end of $string, substitute $replacement for $substring
${string/%substring/replacement}	If $substring matches back end of $string, substitute $replacement for $substring
expr match "$string" '$substring'	Length of matching $substring* at beginning of $string
expr "$string" : '$substring'	Length of matching $substring* at beginning of $string
expr index "$string" $substring	Numerical position in $string of first character in $substring* that matches [0 if no match, first character counts as position 1]
expr substr $string $position $length	Extract $length characters from $string starting at $position [0 if no match, first character counts as position 1]
expr match "$string" '\($substring\)'	Extract $substring*, searching from beginning of $string
expr "$string" : '\($substring\)'	Extract $substring* , searching from beginning of $string
expr match "$string" '.*\($substring\)'	Extract $substring*, searching from end of $string
expr "$string" : '.*\($substring\)'	Extract $substring*, searching from end of $string

* Where $substring is a Regular Expression.

Refer http://itsjustsosimple.blogspot.in/2013/02/all-about-variablesparameters-in-shell.htmlfor few examples.

Read More

Reading environment variables in python

The os module contains an interface to operating system-specific functions. this module can be used to access environment variables. 

We can go with os.environ to get the value of environment variable.

import os
print os.environ['HOME']

but there is a catch, this method will raise KeyError variable does not exist 

>>> print os.environ['HOME_NOT_EXIST']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/UserDict.py", line 23, in __getitem__
    raise KeyError(key)
KeyError: 'HOME_NOT_EXIST'




So it is better to go with os.getenv. this return None if key/environment variable does not exist. and if we require we can return default values too.




print os.getenv('KEY')    #returns None if KEY doesn't exist

print os.getenv('KEY', 0) #will return 0 if KEY doesn't exist 

Read More

All about variables/parameters in shell

Stripping Variable Strings By length:

Syntax: ${parameter:starting_index:number_of_chars}

${variable:3}      # foobar becomes bar
${variable:3:2}   # foobar becomes ba
${variable: -4}   # foobar becomes obar

Upper and lower case :

y="this IS small"
echo "${y^^}"
THIS IS SMALL

y="THIS is BIG"
echo "${y,,}"
 this is big

Removing a Part:

name=${var#removepat}
This will remove 'removepatt' from variable(begining) var and assign to name

name=${var%removepat}
can use % to remove the pattern at the end. useful when one wish to remove trailing / from path names

##   Deletes longest match of $removepat from front of $var.
%%  Deletes longest match of $removepat from back of $var.


Here is the example which will tokenize positional parameters based on =

tmp=${1}
parameter=${tmp%%=*}     # Extract name.
value=${tmp##*=}               # Extract value.

if $1 is a=1, then
parameter=a     #strip from back till = ; so =* will be removed    
value=1            # strip from front till = ; so *= will be removed

Substring Replacement:

${string/substring/replacement}

Replace first match of $substring with $replacement.${string//substring/replacement}Replace all matches of $substring with $replacement.

string=abcABC123ABCabc
echo ${string/abc/xyz}                  # xyzABC123ABCabc
echo ${string//abc/xyz}                 # xyzABC123ABCxyz

Following will simply delete the pattern from string.

string=abcABC123ABCabc
echo ${string/abc/}            # ABC123ABCabc
echo ${string//abc/}           # ABC123ABC

Get Length:

To calculate length of any variable in shell, # can be used.

a="ABCD"
${#a}
4

Calculate number of elements in array:

$ os=("linux" "solaris" "aix")
$ echo {#os[*]}
3
$ echo {#os[@]}
3

@ or * can be used to access all elaments of an array.

These native methods will be faster since we are not invoking any third party tools.

Note: without '(  )' result can't be treated as array.

AWK/CUT implementation (simplified) :

dinesh@ubuntu:~$ a="abc 123 qqq"
dinesh@ubuntu:~$ set -- $a
dinesh@ubuntu:~$ echo $1
abc
dinesh@ubuntu:~$ echo $2
123
dinesh@ubuntu:~$ echo $3
qqq
dinesh@ubuntu:~$

Read More

dbx: Error checking initialization failed ??

How to Run:
                           dbx <binaryname>

Enabling access checking :   in dbx prompt enter :  check -access
Enabling memory leak checking: in dbx prompt enter: check -memuse
Enable all checks :  in dbx prompt enter : check -all

once this is done use run command with appropriate  command line arguments to run the program.
if there are any leaks are access errors present in the binary, an error file (binary.err) will be generated and all errors will be redirected to that file.

Some times if access/memory checks are enabled, system may give errors because of some libraries used by the binary.

Ex:

(dbx) check -access
access checking - ON
(dbx) run 1
Running: a.out 1
(process id 17010)
Reading rtcapihook.so
Reading rtcaudit.so
Reading libmapmalloc.so.1
Reading libc_psr.so.1
Reading rtcboot.so
Reading librtc.so
RTC: Enabling Error Checking...
dbx: internal warning: rtc: ld/st instruction uses %r6 at 0xd55c2388 in `/bghux018/bgh26882/AMS80/FSS_MDS/contrib/lib/libnnz10.so`SHATransform_SOL
dbx: system error: cannot recover; Access checking disabled
dbx: Error checking initialization failed.  All error checking disabled.

(dbx)

here because of libnnz10.so, access check is disabled by dbx. 
to tell dbx not use that library use 'rtc skip' command. and enable the access check and run the process.

(dbx) rtc skip libnnz10.so

RTC: please turn on access checking first

(dbx) check -access

access checking - ON

(dbx) rtc skip libnnz10.so

(dbx) run 1

Running: a.out 1

(process id 17046)

RTC: Enabling Error Checking...

RTC: Using UltraSparc trap mechanism

RTC: See `help rtc showmap' and `help rtc limitations' for details.

RTC: Running program...

When access checking is turned on, RTC detects and reports the following kinds of errors:

        baf     # Bad free
        duf     # Duplicate free
        maf    # Misaligned free
        mar    # Misaligned read
        maw   # Misaligned write
        oom    # Out of memory
        rua     # Read from unallocated memory
        rui      # Read from uninitialized memory
        wro    # Write to read-only memory
        wua    # Write to unallocated memory

With leaks checking, RTC will report the following kinds of errors:


        aib     # Possible memory leak - only pointer points in the middle of the block
        air     # Possible memory leak - pointer to the block exists only in register
        mel   # Memory leak - no pointers to the block

Memory errors can be suppressed. The following command suppress read from uninitialized (rui) in all functions in a.out

(dbx) suppress rui in a.out

RTC instruments memory access assembly instructions for access checking. You can exclude load objects, object files and functions from being instrumented. The following command

(dbx) rtc skippatch a.out -f main

excludes the function main from being instrumented.

Ref : https://blogs.oracle.com/janitor/entry/runtime_memory_checking

bcheck reference : http://docs.oracle.com/cd/E19205-01/820-4180/man1/bcheck.1.html

Read More

Extract particular lines from a file using 'sed'

1. Extract Using Line Numbers :

1 2 3

sed -n '10'p <filename> #Prints 10th line of file sed -n '10,20'p <filename> #Print lines 10 to 20 sed -n '10,$'p <filename> #Print lines 10 to end

$ sed -n '10'p file is equivalent to  
$ sed '10!d' file and results
Line 10

The default output behavior is to print every line of the input file stream.The explicit 10p command just tells it to print the tenth line
So in the first example since sed is called with -n flag, it suppress the default output and prints 10th line
in the  second example, (note -n is not used) sed will delete (d) the line if it is not (!) line number 10

Similarly printing lines from 2 to 4 :

1 2	sed -n '2,4'p file sed '2,4!d' file

Line 2
Line 3
Line 4


2. Extract Using Pattern :

Some time we may need to print the lines which matches the pattern. that also can be done using sed

1	sed -n '/main/p' check.cpp

will print the lines which matches the pattern main in check.cpp file.
grep can be used as an alternative

similarly instead of 'p' (print), 'd' (delete) can be used to delete the line matching the pattern from the file.

1	sed '/main/d' check.cpp > newfile.cpp

3. Extract non empty lines :

Following will remove the empty lines from the file. and redirect remaining to new file.

1 2	sed '/./!d' check.cpp > newfile.cpp sed '/^$/d' check.cpp > newfile.cpp

Read More

Useful aliases in Shell

Like 'ddd' , we can see the code while debugging with gdb too ... 
Set the '-tui' flag for gdb to enable this feature. and '-quiet' will suppress copyright messages. 
      
  alias gdb='gdb -tui -quiet'

Read as many as possible without scrolling

alias tail='tail -n $((${LINES}-2))'

Reading hex data
what most people want from od

alias od='od -Ad -tx1 -v'

Create a directory and go into it

md () { mkdir -p "$1" && cd "$1"; }

Write these functions in bashrc file,common in other languages, available in the shell

ord() { printf "0x%x\n" "'$1"; }

chr() { printf $(printf '\\%03o\\n' "$1"); }

Ex Usage:

dinesh@ubuntu:~$ chr 65
A
dinesh@ubuntu:~$ ord a
0x61
dinesh@ubuntu:~$ ord 1
0x31
dinesh@ubuntu:~$ 

Read More