Python - Selecting a random item from a list or tuple or dict

Say you have a list with 'n' number of items and you wish to get some random item from that list, How do you do it ?

Python has inbuilt 'random' package to do this.

To get one random item from the list:

>>> import random
>>> items = ['a', 'b', 'c', 'd', 'e', 'f', 'g']
>>> print random.choice(items)
'c'

There’s an equally simple way to select n items from the sequence:

>>> import random
>>> items = ['a', 'b', 'c', 'd', 'e', 'f', 'g']
>>> print random.sample(items, 2)
['e', 'a']

Read More

ldapmodify failed with error "dn: attribute type undefined"

I was facing this strange error "dn: attribute type undefined" while I am trying to delete attributes from the LDAP. My ldif file has two attributes to be deleted.

cat /tmp/modifyattr.ldif

dn: fsFgId=Child2, fsFgId=Child1, fsClsId=Root
changetype: modify
delete: fsListeningHost

dn: fsFgId=Child2, fsFgId=Child1, fsClsId=Root
changetype: modify
delete: fsPort

When ldapmodify command is executed with the above ldif file it says "Undefined attribute type".

modifying entry "fsFgId=Child2, fsFgId=Child1, fsClsId=Root"
modify complete
ldapmodify: Undefined attribute type (17)
additional info: dn: attribute type undefined

I googled for this error, every where I found suggestion to add extra blank line before dn. It didn't work for me. After a lot of research and experiments I found that I was adding extra space along with the blank line !!! Because I was creating this file using bash script, I didnt notice that extra space.

Finally it worked like charm.

Read More

Secure way of deleting files in Unix

As I mentioned in the previous post there are simple techniques that can recover your deleted file (by using simple 'grep' or 'strings' utilities).
Even some data recovery tools does the same thing. So if you want to delete some data on the disk without being worried about the retrieval, then
you should probably over write the disk which has your file content.

shred utility available in Linux does the same thing.
shred actually overwrite the specified file repeatedly, in order to make it harder for even very expensive hardware probing to recover the data.
By default shred overwrites the file 25 times with the junk data.

you can chose to remove the file after over writing using -u (unlink)
There are multiple options with shred to explore.

$ shred -u filetobedeleted.txt

Just to see how it works, let say my script is writing some data to the file 'testlog.log' repeatedly after every 1 min.
I am tailing the file in one terminal. And in other terminal I did execute shred.
 

$ sh writetodisk.sh &

$ tail -f testlog.log
aaa
bbb
ccc
ddd

$ shred testlog.log

Now observe the terminal one

$ tail -f testlog.log
aaa
bbb
ccc
ddd
\{XÁÀà_ç æIƒòDÊ5žq­Æ 8<TÝõ ¬ S õŸt\Ê1’ïNÐ , éM‚?$Väé@. l"®ÎþÌÕæ ‡Ù+Ž’ bªO"× #f©ÎçN‰/h÷¡çÊhÇöŸz!*ÀA?RAo%æ} ÛZ½PSàpû7Íû3U_ ’e^u÷züê¾Ú6󚶄Ë[Fœ;½êê±î÷
                                                           Cý ]¤¥ˆi                                                                                  ÕÎ8ƒ:SÎq3®B h€'Q“ãªF¹X‘Q'†GÁ–oõ»hï eþ:½U4Úy_£È‘”f}"J_ŠÒ‡±Ê0íÕwº }rºŸoÇpÜ Wá‚À°xfeÒ?ÕC·         ‰JðhJë ™ÀQêM]ÞÑÅ,A {9b ÑùÇ@©}ÅŠ½°Ò¡øÜK-òõ ªLoLƒü
GýÑeÈ#WsG`Þ¼µÅ"–> T/~ [ºÝ ¸ýŒ<C8îzD±š¨š J
#Lwk{lû´köAٍ^0ê(9¿Ó Xnš¼¼ýc+7×Ãó ‡@ ;¥
                                        ŽBýˆÔ
                                              ÀF ⍠?’‰´q’+iQ‰ Y¸¯`± {·;²&%6ÈÄLYdù½­ š¼ÑÖi…ö±É* ÝÜ(Y2Ðc FÔ]þŠ ˜° ˜ƒTãðõ,l‚šl„bÜ8Å òU='µ YR™&iõqmôT ¤¿)“G[¡9îÎD $ˆÉšDÒ–„xFÀjKNs„)½3̆^¹°w

you can see that the shred filled the contents of the file with garbage data.

Read More

How to recover a file that was removed using 'rm' command ?

In Unix like file systems, the system uses 'hard links' to point to piece of data that you write to the disk.
So when you create a file, you also create its first hard link. You can create multiple hard links using 'ln' command.
When you "delete" a file using rm command, normally you are only deleting the hard link.

If all hard links to a particular file are deleted, then the system removes only the reference to the data and indicate that the blocks as free. But it won't actually delete the file.

If your deleted file is opened by any running process then it means you still have one link left to your file !!.
check if any process who works on your file using lsof command

$ lsof | grep "myfile.txt"
COMMAND    PID     USER   FD      TYPE    DEVICE   SIZE     NODE    NAME
pgm-name   7099    root   25r    REG     254,0    349      16080   /tmp/myfile.txt

Using the process and file descriptor you can try copying the file

$ cp /proc/7099/fd/25 /mydir/restore.txt

If lsof didn't list your file then you could try to locate your data reading directly from the device.
But this works only if blocks containing your files haven't been claimed for something else.

To make sure no one else over writes that free blocks, immediately remount the file system with read-only and then search for your file.

$ mount -o ro,remount /dev/sda1
$ grep -a -C100 "unique string" /dev/sda1 > file.txt

Replace /dev/sda1 with the device that the file was on and replace 'string' with the unique string in your file.
This does is it searches for the string on the device and then returns 100 lines of context and puts it in file.txt.
If you need more lines returned just adjust -C options as appropriate. Alternatively you can use -A, -B options with grep to print lines before and after the matched string.

You might get a bunch of extra garbage date, mostly some binary data but you can get your data back.
If you don't want this binary data then you can apply 'string' on the device and grep for the unique string

$ strings /dev/sda1 | grep -C100 "unique string" > file.txt

Read More

Calculating time differences in python

Recently I had a situation where I need to calculate the timer efficiency.
I have a C++ timer that calls my function after every 5 sec. My function does some critical operation and logs one statement to syslog.
When I observed the logs I found that there is delay in my function execution, slowly the timer started drifting in result and the next function calls are getting delayed !!

So I wanted to calculate how many times the timer has delayed in a day. I grepped for the particular log in my syslog and redirected to a file.

file format is like this:
Jan 29 06:34:24
Jan 29 06:34:29
Jan 29 06:34:34
Jan 29 06:34:39
..
..
Now I should compare the lines in the file and log if the time difference is greater than 5 sec.
compare line 1 with line 2
line 2 with line 3, then line 3 with line 4 and so on...



Here the bad thing is f.readlines() ... It will load whole file in to list and tried to read 2 lines at a time.
If anybody reads this post :P and if you know any better working solution please share. :)

Read More

Change the screen brightness using bash script - xrandr

In my Ubuntu 14.0 (Dell Inspiron), brightness controls were not working at all. So I started searching for other alternatives to sets the brightness.

finally I found this xrandr.
it is great tool to set the brightness and even to rotate the screen.

--brightness flag can be used to set the brightness (best values: 0 - 1)

--brightness flag cab be used to rotate the screen (possible options : normal, left, right) 

Usage:
          ./brightness_control.sh +      # Increase the brightness by 0.05
          ./brightness_control.sh  -       # Decrease the brightness by 0.05
          ./brightness_control.sh 1       # Set brightness to 1
 
 

Read More

Undestanding Generators in Python

Before we understand generator we must know Iterator.

Iterator:
Iterator is any object which understands the concept of a for-loop.
Basically this means that it has a next method, when it is called it returns the next item in the sequence.
Everything you can use in "for" is an iterable: lists, strings, files...etc

>>> [x*x for x in range(3)]
[0, 1, 4]

>>> mylist = [x*x for x in range(3)]
>>> for i in mylist:
    print i


0
1
4

Generator:
Generators are iterators, but you can only iterate over them once.
It's because they do not store all the values in memory, they generate the values on the fly.

If we try to print the generator we will get the object.

>>> (x*x for x in range(3))
<generator object <genexpr> at 0x000000000268C480>

>>> mygenerator = (x*x for x in range(3))
>>> for i in mygenerator:
 print i

0
1
4

Generators can be iterated only once. If we try to iterate over them again we get nothing its because when for loop is running it actually calls next element in the tuple and loop will break when there are no more elements. Once the loop breaks generator is destroyed (object still exist but no more elements to return). So if we try to use the same generator again to print the elements we get nothing.

>>> mygenerator = (x*x for x in range(3))
>>> for i in mygenerator:
 print i

0
1
4

>>> for i in mygenerator:
 print i

In short,  if round parentheses are used, then a generator iterator is created.
If rectangular parentheses are used, then we get a list.

Read More