Secure way of deleting files in Unix

As I mentioned in the previous post there are simple techniques that can recover your deleted file (by using simple 'grep' or 'strings' utilities).
Even some data recovery tools does the same thing. So if you want to delete some data on the disk without being worried about the retrieval, then
you should probably over write the disk which has your file content.

shred utility available in Linux does the same thing.
shred actually overwrite the specified file repeatedly, in order to make it harder for even very expensive hardware probing to recover the data.
By default shred overwrites the file 25 times with the junk data.

you can chose to remove the file after over writing using -u (unlink)
There are multiple options with shred to explore.

$ shred -u filetobedeleted.txt

Just to see how it works, let say my script is writing some data to the file 'testlog.log' repeatedly after every 1 min.
I am tailing the file in one terminal. And in other terminal I did execute shred.
 

$ sh writetodisk.sh &

$ tail -f testlog.log
aaa
bbb
ccc
ddd

$ shred testlog.log

Now observe the terminal one

$ tail -f testlog.log
aaa
bbb
ccc
ddd
\{XÁÀà_ç æIƒòDÊ5žq­Æ 8<TÝõ ¬ S õŸt\Ê1’ïNÐ , éM‚?$Väé@. l"®ÎþÌÕæ ‡Ù+Ž’ bªO"× #f©ÎçN‰/h÷¡çÊhÇöŸz!*ÀA?RAo%æ} ÛZ½PSàpû7Íû3U_ ’e^u÷züê¾Ú6󚶄Ë[Fœ;½êê±î÷
                                                           Cý ]¤¥ˆi                                                                                  ÕÎ8ƒ:SÎq3®B h€'Q“ãªF¹X‘Q'†GÁ–oõ»hï eþ:½U4Úy_£È‘”f}"J_ŠÒ‡±Ê0íÕwº }rºŸoÇpÜ Wá‚À°xfeÒ?ÕC·         ‰JðhJë ™ÀQêM]ÞÑÅ,A {9b ÑùÇ@©}ÅŠ½°Ò¡øÜK-òõ ªLoLƒü
GýÑeÈ#WsG`Þ¼µÅ"–> T/~ [ºÝ ¸ýŒ<C8îzD±š¨š J
#Lwk{lû´köAٍ^0ê(9¿Ó Xnš¼¼ýc+7×Ãó ‡@ ;¥
                                        ŽBýˆÔ
                                              ÀF ⍠?’‰´q’+iQ‰ Y¸¯`± {·;²&%6ÈÄLYdù½­ š¼ÑÖi…ö±É* ÝÜ(Y2Ðc FÔ]þŠ ˜° ˜ƒTãðõ,l‚šl„bÜ8Å òU='µ YR™&iõqmôT ¤¿)“G[¡9îÎD $ˆÉšDÒ–„xFÀjKNs„)½3̆^¹°w

you can see that the shred filled the contents of the file with garbage data.

Read More

How to recover a file that was removed using 'rm' command ?

In Unix like file systems, the system uses 'hard links' to point to piece of data that you write to the disk.
So when you create a file, you also create its first hard link. You can create multiple hard links using 'ln' command.
When you "delete" a file using rm command, normally you are only deleting the hard link.

If all hard links to a particular file are deleted, then the system removes only the reference to the data and indicate that the blocks as free. But it won't actually delete the file.

If your deleted file is opened by any running process then it means you still have one link left to your file !!.
check if any process who works on your file using lsof command

$ lsof | grep "myfile.txt"
COMMAND    PID     USER   FD      TYPE    DEVICE   SIZE     NODE    NAME
pgm-name   7099    root   25r    REG     254,0    349      16080   /tmp/myfile.txt

Using the process and file descriptor you can try copying the file

$ cp /proc/7099/fd/25 /mydir/restore.txt

If lsof didn't list your file then you could try to locate your data reading directly from the device.
But this works only if blocks containing your files haven't been claimed for something else.

To make sure no one else over writes that free blocks, immediately remount the file system with read-only and then search for your file.

$ mount -o ro,remount /dev/sda1
$ grep -a -C100 "unique string" /dev/sda1 > file.txt

Replace /dev/sda1 with the device that the file was on and replace 'string' with the unique string in your file.
This does is it searches for the string on the device and then returns 100 lines of context and puts it in file.txt.
If you need more lines returned just adjust -C options as appropriate. Alternatively you can use -A, -B options with grep to print lines before and after the matched string.

You might get a bunch of extra garbage date, mostly some binary data but you can get your data back.
If you don't want this binary data then you can apply 'string' on the device and grep for the unique string

$ strings /dev/sda1 | grep -C100 "unique string" > file.txt

Read More

Calculating time differences in python

Recently I had a situation where I need to calculate the timer efficiency.
I have a C++ timer that calls my function after every 5 sec. My function does some critical operation and logs one statement to syslog.
When I observed the logs I found that there is delay in my function execution, slowly the timer started drifting in result and the next function calls are getting delayed !!

So I wanted to calculate how many times the timer has delayed in a day. I grepped for the particular log in my syslog and redirected to a file.

file format is like this:
Jan 29 06:34:24
Jan 29 06:34:29
Jan 29 06:34:34
Jan 29 06:34:39
..
..
Now I should compare the lines in the file and log if the time difference is greater than 5 sec.
compare line 1 with line 2
line 2 with line 3, then line 3 with line 4 and so on...



Here the bad thing is f.readlines() ... It will load whole file in to list and tried to read 2 lines at a time.
If anybody reads this post :P and if you know any better working solution please share. :)

Read More

Change the screen brightness using bash script - xrandr

In my Ubuntu 14.0 (Dell Inspiron), brightness controls were not working at all. So I started searching for other alternatives to sets the brightness.

finally I found this xrandr.
it is great tool to set the brightness and even to rotate the screen.

--brightness flag can be used to set the brightness (best values: 0 - 1)

--brightness flag cab be used to rotate the screen (possible options : normal, left, right) 

Usage:
          ./brightness_control.sh +      # Increase the brightness by 0.05
          ./brightness_control.sh  -       # Decrease the brightness by 0.05
          ./brightness_control.sh 1       # Set brightness to 1
 
 

Read More

Undestanding Generators in Python

Before we understand generator we must know Iterator.

Iterator:
Iterator is any object which understands the concept of a for-loop.
Basically this means that it has a next method, when it is called it returns the next item in the sequence.
Everything you can use in "for" is an iterable: lists, strings, files...etc

>>> [x*x for x in range(3)]
[0, 1, 4]

>>> mylist = [x*x for x in range(3)]
>>> for i in mylist:
    print i


0
1
4

Generator:
Generators are iterators, but you can only iterate over them once.
It's because they do not store all the values in memory, they generate the values on the fly.

If we try to print the generator we will get the object.

>>> (x*x for x in range(3))
<generator object <genexpr> at 0x000000000268C480>

>>> mygenerator = (x*x for x in range(3))
>>> for i in mygenerator:
 print i

0
1
4

Generators can be iterated only once. If we try to iterate over them again we get nothing its because when for loop is running it actually calls next element in the tuple and loop will break when there are no more elements. Once the loop breaks generator is destroyed (object still exist but no more elements to return). So if we try to use the same generator again to print the elements we get nothing.

>>> mygenerator = (x*x for x in range(3))
>>> for i in mygenerator:
 print i

0
1
4

>>> for i in mygenerator:
 print i

In short,  if round parentheses are used, then a generator iterator is created.
If rectangular parentheses are used, then we get a list.

Read More

Encrypt files using Vim

Vim comes with default encryption mechanism refereed as VimCrypt in the documentation. It is capable of writing files encrypted and read them back.

Creating new encrypted file:

vim -x <file>

This will create new file if it is not exist or open an existing file and prompt for password. This password is used as key to encrypt and decrypt the file.

Encrypt the current working file:

:X

You can also encrypt the current working file /open files using :X  option. This will also prompt for password. The typed password is stored in the 'key' option which is used to encrypt the file when it is written.

Disable the encryption:

To disable the encryption reset the 'key' option to empty value. To do this you first need to open the file with proper password.

:set key=

Read More

Automate your Vim - The power of autocmd

1. Vim templates

I often used to write sample C++ codes to test the functionality. But it is pain to add default code for every test code that I write. Vim has a brilliant solution for it. that is templates.

Create some template files in any of the directory you wish, here I created ~/.vim/templates/skeleton.cpp that contains default c++ code i.e #include, using namespace std, main,..etc.
Then add the below code to your ~/.vimrc  file. Now try to open a new cpp file and see the magic.


Here we are telling vim that, if I try to open a new cpp file, read the content from ~/.vim/templates/skeleton.cpp file and write to the new file that is opened. Similarly you can have templates for all type of files.

This means, what ever you write in template will be loaded by default for every file you open newly using vim.

2. Auto compile C++ code on saving

We can even compile the code immediately after saving the file without typing any single command.  Add the below code to your vimrc file and try to save the test cpp file. It will create a executable with the name of the file.

1 2	"Auto compile on save autocmd BufWritePost,FileWritePost *.cpp !g++ -Wall %:p -o %:r

In vim :p expands to full file name path and :r expands to root of the file name (file extension will be removed)

But this is bad ! every time you try to save your file it tries to compile. So there is alternative, vim has quit event. i.e VimLeave.

1 2	"auto compile on quit - silent: do not report errors autocmd VimLeave *.cpp !silent g++ -Wall %:p -o %:r

VimLeave event indicates vim to execute the command specified after ! before exiting the vim after writing viminfo file.

3. Abbreviations for faster coding

Abbreviations can save typing when you want to type same text multiple times thorough the document. and abbreviations can be defined to auto correct the typos.
Add the below code to your vimrc file and try to edit the cpp file. when every you type iff this will be auto corrected to if ()  and moves the cursor in between the parenthesis.


abbrev(ab) is used to create abbreviations, this works on normal mode. iabbrev is used to work with abbreviations on insert mode. and <left> is used to move the cursor one character left.

4. Auto comments

Similarly to automate commenting add the below code to vimrc.


Vim will detect the file type using FileType event. And we are mapping the movement ,c to comment lines.
But this will comment only current line.

For commenting a block of text, go to the first line you want to comment, press Ctrl + V, and select until the last line. Second, press I// + Esc (then give it a second), and it will insert a // character on all selected lines.

ctrl+v
drop down using arrow key(select only first character of the each line)
shif+i
shift+//
esc

Un-commenting a block of text is almost the same: Put your cursor on the first # character, press CtrlV and go down until the last commented line and press x, that will delete all the # characters vertically.

Learning from http://learnvimscriptthehardway.stevelosh.com/

Read More