Secure way of deleting files in Unix

As I mentioned in the previous post there are simple techniques that can recover your deleted file (by using simple 'grep' or 'strings' utilities).
Even some data recovery tools does the same thing. So if you want to delete some data on the disk without being worried about the retrieval, then
you should probably over write the disk which has your file content.

shred utility available in Linux does the same thing.
shred actually overwrite the specified file repeatedly, in order to make it harder for even very expensive hardware probing to recover the data.
By default shred overwrites the file 25 times with the junk data.

you can chose to remove the file after over writing using -u (unlink)
There are multiple options with shred to explore.

$ shred -u filetobedeleted.txt

Just to see how it works, let say my script is writing some data to the file 'testlog.log' repeatedly after every 1 min.
I am tailing the file in one terminal. And in other terminal I did execute shred.
 

$ sh writetodisk.sh &

$ tail -f testlog.log
aaa
bbb
ccc
ddd

$ shred testlog.log

Now observe the terminal one

$ tail -f testlog.log
aaa
bbb
ccc
ddd
\{XÁÀà_ç æIƒòDÊ5žq­Æ 8<TÝõ ¬ S õŸt\Ê1’ïNÐ , éM‚?$Väé@. l"®ÎþÌÕæ ‡Ù+Ž’ bªO"× #f©ÎçN‰/h÷¡çÊhÇöŸz!*ÀA?RAo%æ} ÛZ½PSàpû7Íû3U_ ’e^u÷züê¾Ú6󚶄Ë[Fœ;½êê±î÷
                                                           Cý ]¤¥ˆi                                                                                  ÕÎ8ƒ:SÎq3®B h€'Q“ãªF¹X‘Q'†GÁ–oõ»hï eþ:½U4Úy_£È‘”f}"J_ŠÒ‡±Ê0íÕwº }rºŸoÇpÜ Wá‚À°xfeÒ?ÕC·         ‰JðhJë ™ÀQêM]ÞÑÅ,A {9b ÑùÇ@©}ÅŠ½°Ò¡øÜK-òõ ªLoLƒü
GýÑeÈ#WsG`Þ¼µÅ"–> T/~ [ºÝ ¸ýŒ<C8îzD±š¨š J
#Lwk{lû´köAٍ^0ê(9¿Ó Xnš¼¼ýc+7×Ãó ‡@ ;¥
                                        ŽBýˆÔ
                                              ÀF ⍠?’‰´q’+iQ‰ Y¸¯`± {·;²&%6ÈÄLYdù½­ š¼ÑÖi…ö±É* ÝÜ(Y2Ðc FÔ]þŠ ˜° ˜ƒTãðõ,l‚šl„bÜ8Å òU='µ YR™&iõqmôT ¤¿)“G[¡9îÎD $ˆÉšDÒ–„xFÀjKNs„)½3̆^¹°w

you can see that the shred filled the contents of the file with garbage data.

Read More

Change the screen brightness using bash script - xrandr

In my Ubuntu 14.0 (Dell Inspiron), brightness controls were not working at all. So I started searching for other alternatives to sets the brightness.

finally I found this xrandr.
it is great tool to set the brightness and even to rotate the screen.

--brightness flag can be used to set the brightness (best values: 0 - 1)

--brightness flag cab be used to rotate the screen (possible options : normal, left, right) 

Usage:
          ./brightness_control.sh +      # Increase the brightness by 0.05
          ./brightness_control.sh  -       # Decrease the brightness by 0.05
          ./brightness_control.sh 1       # Set brightness to 1
 
 

Read More

Unix command line tricks

1. Ever tried creating pdf files in your unix machine ??

man -t ascii | ps2pds - > ascii_man_page.pdf

-t flag formats the output and ps2pds converts post script to pdf.
hyphen (-) after ps2pdf command indicates it to read input from stdin and write output to stdout.


2. Want to terminate the script after first failure ?

'set -e' thats it. it does everything.

$> cat setexit.sh

#/bin/bash
set -e
echo "line 1"
cat tmp.x.12.12
echo "line 2"


$> ./setexit.sh
line 1
cat: tmp.x.12.12: No such file or directory #line 2 is not echoed

3. Simple way to create huge file

yes, we have 'yes' in unix to do that. yes will echo what ever you give as argument.

#press ctr+c after a while
$yes "some random text..." > bigfile.txt

'yes' has another interesting advantage. just imagine to answer yes to the every question by bash.

$rm file.txt
rm: remove regular file 'file.txt' ?

#you probably give 'y' to remove it. what if you want to remove 100 such files?

yes | rm *.sh
  

though this can be achieved just by ignoring alias ( \rm *.sh ) this is just an example to demonstrate the usage.


4. Creating dirty random number

we have $RANDOM shell variable to generate a random number may be max of 5 characters. If you wish to create a big random number use 'mcookie'. This command generates a "magic cookie," a 128-bit pseudorandom hexadecimal number, normally used as an authorization signature by the X server. This also available for use in a script as a quick and dirty random number.

$mcookie
a2cae14a060720bca183d05f165a5a8e


5. Get directory name or file name from path and file extension

#extract dir name and file name
$> basename /dir1/dir2/file.txt      
file.txt  
$> dirname  /dir1/dir2/file.txt 
/dir1/dir2

#Extract extension 
$> path="/dir1/dir2/file.txt"
$> ext=${path##*.}
$> echo $ext
txt

6. Simple 'cut'

var="one two three"
set -- $var
a=${1};   #one 
b=${2};   #two
c=${3};   #three

Just a basic example to avoid invoking forked process (cut/awk)


7. Base conversion

'bc' can convert from any base to any.

Ex: decimal to hexadecimal conversion
echo 'obase=16; ibase=10; 64' | bc            # (ibase=input base, obase=output base)

This will convert 64 from base 10 to base 16.

Read More

Difference between array[@] and array[*] in shell

It is important for quoting and spacing that the ${array[@]}
format (rather than ${array[*]}) is used, and also that double quotes are put around the whole construct. 

items=(pen "permanent marker" pencil "temporary marker")

 
for item in ${items[@]}
do
  echo "Item is : $item"
done

Item is : pen
Item is : permanent
Item is : marker
Item is : pencil
Item is : temporary
Item is : marker

for item in ${items[*]}
do
  echo "Item is : $item"
done

Item is : pen
Item is : permanent
Item is : marker
Item is : pencil
Item is : temporary
Item is : marker


items[@] and items[*] produced the same output and which is not expected. "permanent" and "marker" are treated as two different words in both the cases though it is a single word. lets try now with the quotes around the array..

items=(pen "permanent marker" pencil "temporary marker")

for item in "${items[@]}"
do
  echo "Item is : $item"
done

Item is : pen
Item is : permanent marker
Item is : pencil
Item is : temporary marker
 
 YES this what we expected.... but items[*] will not behave this way.

for item in "${items[*]}"
do
  echo "Item is : $item"
done

Item is : pen permanent marker pencil temporary marker


the * is not suitable either with or without quotes. Without quotes, it does the same as the @ symbol. With quotes, the whole array is boiled down into a single string.

Read More

All about variables/parameters in shell

Stripping Variable Strings By length:

Syntax: ${parameter:starting_index:number_of_chars}

${variable:3}      # foobar becomes bar
${variable:3:2}   # foobar becomes ba
${variable: -4}   # foobar becomes obar

Upper and lower case :

y="this IS small"
echo "${y^^}"
THIS IS SMALL

y="THIS is BIG"
echo "${y,,}"
 this is big

Removing a Part:

name=${var#removepat}
This will remove 'removepatt' from variable(begining) var and assign to name

name=${var%removepat}
can use % to remove the pattern at the end. useful when one wish to remove trailing / from path names

##   Deletes longest match of $removepat from front of $var.
%%  Deletes longest match of $removepat from back of $var.


Here is the example which will tokenize positional parameters based on =

tmp=${1}
parameter=${tmp%%=*}     # Extract name.
value=${tmp##*=}               # Extract value.

if $1 is a=1, then
parameter=a     #strip from back till = ; so =* will be removed    
value=1            # strip from front till = ; so *= will be removed

Substring Replacement:

${string/substring/replacement}

Replace first match of $substring with $replacement.${string//substring/replacement}Replace all matches of $substring with $replacement.

string=abcABC123ABCabc
echo ${string/abc/xyz}                  # xyzABC123ABCabc
echo ${string//abc/xyz}                 # xyzABC123ABCxyz

Following will simply delete the pattern from string.

string=abcABC123ABCabc
echo ${string/abc/}            # ABC123ABCabc
echo ${string//abc/}           # ABC123ABC

Get Length:

To calculate length of any variable in shell, # can be used.

a="ABCD"
${#a}
4

Calculate number of elements in array:

$ os=("linux" "solaris" "aix")
$ echo {#os[*]}
3
$ echo {#os[@]}
3

@ or * can be used to access all elaments of an array.

These native methods will be faster since we are not invoking any third party tools.

Note: without '(  )' result can't be treated as array.

AWK/CUT implementation (simplified) :

dinesh@ubuntu:~$ a="abc 123 qqq"
dinesh@ubuntu:~$ set -- $a
dinesh@ubuntu:~$ echo $1
abc
dinesh@ubuntu:~$ echo $2
123
dinesh@ubuntu:~$ echo $3
qqq
dinesh@ubuntu:~$

Read More

Useful aliases in Shell

Like 'ddd' , we can see the code while debugging with gdb too ... 
Set the '-tui' flag for gdb to enable this feature. and '-quiet' will suppress copyright messages. 
      
  alias gdb='gdb -tui -quiet'

Read as many as possible without scrolling

alias tail='tail -n $((${LINES}-2))'

Reading hex data
what most people want from od

alias od='od -Ad -tx1 -v'

Create a directory and go into it

md () { mkdir -p "$1" && cd "$1"; }

Write these functions in bashrc file,common in other languages, available in the shell

ord() { printf "0x%x\n" "'$1"; }

chr() { printf $(printf '\\%03o\\n' "$1"); }

Ex Usage:

dinesh@ubuntu:~$ chr 65
A
dinesh@ubuntu:~$ ord a
0x61
dinesh@ubuntu:~$ ord 1
0x31
dinesh@ubuntu:~$ 

Read More

Timeout - timed wait in shell scripting

TMOUT is the shell built in variable to set the timeout .

TMOUT is used in three different ways: by the read builtin command, by the select builtin, and by the interactive bash shell. If it is unset, or equal to zero, then it is ignored. If it has any positive value, then these three commands which make use of it will timeout after $TMOUT seconds.

Example:
$> cat timeout.sh

1 2 3 4 5 6 7 8 9 10

#!/bin/bash TMOUT=5 read -p "Enter password : " pass if [ $? -eq 0 ] then echo "Password is $pass" else echo "Timed out...." fi

same effect can be achieved using read with -t option.
read -t 5 -p "Enter password : " pass

Read More

PIPESTATUS and its Alternative

$> cat /etc/hosts | grep 000.000
$> $?
1


$> cat /etc/hosts | grep 000.000 | uniq
$> $?
0

Why is it returning success (0) though it fails during grep ??
the return code of a pipeline will be that of the return status of the rightmost command .
How to resolve this ??
use inbuilt PIPESTATUS variable.

PIPESTATUS is a array variable which contain the exit status of each command in piped commands. 

$> cat /etc/hosts | grep 000.000 | uniq
$> echo ${PIPESTATUS[*]}
0 1 0

here $PIPESTATUS[0] contains the exit status of cat /etc/hosts command
$PIPESTATUS[1] contains the exit status of grep 000.000
$PIPESTATUS[2] contains the exit status of uniq


The other way around for this is setting pipefail.

$> set -o pipefail
$> cat /etc/hosts | grep 000.000 | uniq
$> $?
1

Read More

Auto Correct Directory Names / Shell Options

"shopt" can be used to correct typos in cd command.
you can use "cdspell" which will auto correct the directory name you have entered.

Usage:   shopt -s cdspell  
( -s : set
 -u : unset)

$ cd /usr/local/sben     will results in bash: cd: /usr/local/sbEN: No such file or directory
but once you have enabled cdspell option,

$ cd /usr/local/sben    
/usr/local/sbin
$pwd
 /usr/local/sbin

some other useful options are,
histappend : this will append to the history file, instead of overwriting it
checkwinsize: check the window size after each command and, if necessary, update the values of LINES and COLUMNS.

lot of other options are available with shopt.
to list all these options used shopt -p

Or you can refer the man page  http://ss64.com/bash/shopt.html

Read More

`Tar`- An Ultimate Archive Utility

How to use tar

tar  [options] [name of tar file to be created] [list of files and directories to be included]

1. Creating an archive using tar
    
      $ tar -cvf target.tar file1 file2 dir1 
      $ tar -cvzf target.tar.gz dirname/             [ creates gzipped tar archive file ]
      $ tar -cvjf target.tar.bz2 dirname/           [ creates bzipped tar archive file ]

    Note: this -cvf options alone does not provide any compressions

2 Extracting an archive
 
      $ tar -xvf target.tar
      $ tar -xvzf target.tar.gz                            [ extracts gzipped tar archive file ]
      $ tar -xvjf target.tar.bz2                          [ extracts bzipped tar archive file ]

3. Listing an archive

      $ tar -tvf target.tar
      $ tar -tvzf target.tar.gz                             [ lists gzipped tar archive file ]
      $ tar -tvjf target.tar.bz2                           [ lists bzipped tar archive file ]

4. Extract a file/directory

    $ tar -xvf target.tar /mydir/myfile
    $ tar -xvzf target.tar.gz /mydir/myfile
    $ tar -xvjf target.tar.bz2 /mydir/myfile 

    if you want to extract a group of files from the archive use -wildcard option
    $ tar -xvf target.tar --wildcards '*.cc'
 
5. Adding a file or directory to an existing archive
 
    You can add additional files to an existing tar archive by specifying -u or -r options.
    use -u option if u do not want duplicates in the tar archived file. -r option will add the file to the existed archive if it exist also.

    $ tar -rvf target.tar newfile     or $ tar -uvf target.tar newfile

    Note: this can't be done for gz or bz2 archives.
 

Read More

'cat' can be harmful..

'cat' can be harmful at times. some people might have lost the data because of improper usage of cat command like cat > filename.txt, or it can do over write some existing files data without prompting the user.
to get rid out of these situations

1. open the terminal

2. execute " set -o noclobber "

that's it. now next time when cat tries to over write the existing file it will prompt the user.

Read More