Showing posts with label Python. Show all posts
Showing posts with label Python. Show all posts

Writing context managers using Python

We use. We forget. This is for myself so I remember the magic called Context Manager. You already must have used the "with statement" for file handling, threads, socket and other operations. I am going to talk about creating you OWN context managers.

The magic operator to be used is : @contextmanager. It can be imported from the module "contextlib"

To create a context manager, use the decorator contextlib.contextmanager


@contextlib.contextmanager
def getListCounts(x):
    try:
        from collections import Counter
        yield Counter(x)
    finally:
        print 'No of elements in the list : %d' % len(x)


Let's now use the above context manager 
   
Example 1:
>>> with getListCount([10, 10, 12]) as c:
 print "Element occurring the most is %d" % c.most_common()[0][0]

 
Element occurring the most is 10
No of elements in the list : 3

Example 2:
>>> with getListCount([10, 10, 12]) as c:
 print "Element occurring the least is %d" % c.most_common()[-1][0]

 
Element occurring the least is 12
No of elements in the list : 3

Python: Super class examples

Hi all. We are going to see some examples of usage of keyword 'super' here . 

Below code shows two classes Employee and Developer. A parent and a child.


class Employee(object):
    def __init__(self, first, last):
        self.first = first
        self.last = last
        print 'Employee: {0}, {1}'.format(first, last)

    def hike_category(self):
        hike = 0
        if self.grade == 'A':
            hike = (0.1*self.pay) + self.pay #percent
        elif self.grade == 'B':
            hike = (0.15*self.pay) + self.pay #percent
        elif self.grade == 'C':
            hike = (0.20*self.pay) + self.pay #percent
        return '{0},{1} gets {2}'.format(self.first, self.last, hike)

class Developer(Employee):
    def __init__(self, fname, lname, grade, pay):
        super(Developer, self).__init__(fname, lname)
        self.grade = grade
        self.pay = pay

Lets run the above program.

Results:

>>> d = Developer('arc', 'roy', 'C', 10000)
Employee: arc, roy
>>> d.hike_category()
'arc,roy gets 12000.0'


Observations and things to note


  • Above programming is done in Python 2.7. 
  • In 2.7, the class needs to inherit from object , otherwise it wont work.
  • The variable names can be different in both classes (In 2.7 only).
  • super needs to have same class name as first argument. 
  • The class instance 'b' can access all class variables of both classes.


Quick guide to Logging in different files using Python

This is for quick setup of a logging system . 


Step 1: import logging 


Logging is the build-in module for implementing logging in Python.

Step 2: Create a logger for yourself. 


logger = logging.getLogger(__name__)

Step 3: Set Logger level


logger.setLevel(logging.DEBUG)

Step 4: Create a handler. 


hdlr = logging.FileHandler(settings.BASE_LOG_FILE, mode='w')

Step 5: Create a formatter


formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')

Step 6: Set formatter and handler


hdlr.setFormatter(formatter)

logger.addHandler(hdlr)



When and why to use Classes in Python

Note: All programming done in Python 2.7

Why to make a class?

When we need to use some common variables across other functions quite often . If we only one or 2 functions to write , we should not need to write a class. But if we are going to have 4-5 functions that need to use common objects between themselves, then its better to use classes.

What are class methods?

Class methods are methods that you can call on the class itself

A good example of classes . Lets have a BankAccount class for managing Bank Accounts and its related operations.


class BankAccount(object):
    #Class methods are methods that you can call on the class itself
    def __init__(self, balance=0):
        self.balance = balance
    def deposit(self, amount):
        self.balance += amount
    def withdraw(self, amount):
        self.balance -= amount

The __init__ function is automatically called when a new instance is created. We have two functions that add or subtract  money from account.

Ok....So lets create 2 accounts


a_account = BankAccount()
b_account = BankAccount()

Now lets deposit something in these accounts


>>> a_account.deposit(100)
>>> b_account.deposit(500)


Let's try to transfer some fund from B to A account. We need to add a "transfer" function to our class.


def transfer(self, other_account, amount):
    self.withdraw(amount)
    other_account.deposit(amount)


>>> b_account.transfer(a_account, 100)
>>> a_account.balance
200
>>> b_account.balance
400

As you can see, how we used classes to perform simple operations between two different accounts.

Customize looks of your object

We can have more control over our objects . Like if we print our object , what will it look.
Normally it looks like this.


>>> a_account
<__main__.BankAccount object at 0x0000000001FB3B38>

We want to show some other info when someone calls up the object itself at the time it is created. We can use the __repr__ function for this.

Lets add this function to our class .


def __repr__(self):
    return "Bank_Account Balance={}".format(self.balance)

One thing to note:

 if __repr__ is defined, and __str__ is not, the object will behave as though __str__=__repr__.

Find more info here 


>>> a_account = BankAccount(1000)
>>> a_account
Bank_Account Balance=1000

If we want to control how it will look when its printed , we should use the __str__ method.


def __str__(self):  # When the object is printed or str(object) is used
    return "Bank_Account is ${}".format(self.balance)


What is Super Class

You might have seen before the keyword being used inside classes. Super here helps us call the original base class __init__function . Whenever I have a method to call in the original baseclass.


class NamedBankAccount(BankAccount):
    def __init__(self, name, *args, **kwargs):
        self.name = name
        super(NamedBankAccount, self).__init__(*args, **kwargs)

Now we create an object of above class.


>>> a = NamedBankAccount('Arc', 1000)
>>> a.name
'Arc'
>>> a.balance
1000
>>> print a
Bank_Account is $1000



As you can see, the 'balance' was auto created by the __init__ of BankAccount class.

Note: In python 3, you can use super().__init__()   . This is much simpler.

Lets add a hello function in our base class and lets add that in our Child class.


class BankAccount(object):
    #Class methods are methods that you can call on the class itself
    def __init__(self, balance=0):
        self.balance = balance
    def deposit(self, amount):
        self.balance += amount
    def withdraw(self, amount):
        self.balance -= amount
    def transfer(self, other_account, amount):
        self.withdraw(amount)
        other_account.deposit(amount)
    def __repr__(self):
        return "Bank_Account Balance={}".format(self.balance)
    def __str__(self):  # When the object is printed or str(object) is used
        return "Bank_Account is ${}".format(self.balance)
    def hello(self):
        print 'Hi'



class NamedBankAccount(BankAccount):
    def __init__(self, name, *args, **kwargs):
        self.name = name
        super(NamedBankAccount, self).__init__(*args, **kwargs)
        super(NamedBankAccount, self).hello()


>>> a = NamedBankAccount(name='Arc')
Hi

I think this is sufficient proof enough that we can call ANY method using super. It just a convenience provided by python.

Note: Just remember that the same can be achieved simple like this, 


BankAccount.__init__(self, 234)

>>> a = NamedBankAccount(name='Arc')
>>> a
Bank_Account Balance=234


Generator functions. Why and when should we use them.

You probably have heard of them. But as a beginner python programmer, we tend to ignore it's values because we fail to find uses of it. When we have loops and itertools and other fancy toys offered, why should we bother to understand them. If you want to take your programming  or understanding of Python to the next level, you must understand and use generators more and more.

Before we start, I want to make a simple program that prints square of (hence generates and delivers) 5 numbers from 1 to 5. So what is the first idea that pops up ? Let's use a for loop.


def square(N):
    for i in range(1, N+1):
        print i**2

>>>square(5)
1
4
9
16
25

Let's say, you need all the values generated for some other calculation. I know, above example is too lame . But imagine after some complicated calculations, you have generated some result.
So you probably will return all values together in an array. And to that, the first thought in our minds is, using a LIST. Let's use it. 

Now our function will look like this.

>>> def my_function(N):

    new_list = []

    for i in range(1, N+1):

        new_list.append(i**2)

    return new_list




>>> my_function(5)
[1, 4, 9, 16, 25]


All this looks easy. You probably are thinking...If generators are going to ruin this idea, I don't need it. Think again.
  1.  This LIST business will soon start becoming a nuisance as the  number of such calculations and variables increase (believe  me...thinking of new names for variables... SUCK).
  2.  What if you don't know your requirement. How many results  do you want.?
  3.  You had to create an array and it will take memory space  which becomes bigger with length of the list and size of each  element.
  4. Once you are dealing with very large list (imagine a million), generating a list of 1 million (let's assume integers) will take around 300 mb in memory.
A generator will make it all...Simpler. Here's the same example with generator function.


def square(N):
    for i in range(1, N+1):
        yield i**2

gen = square(5)

print gen.next()
print gen.next()
print gen.next()
print gen.next()
print gen.next()
print gen.next()

Result:

1
4
9
16
25

Traceback (most recent call last):

  File "D:\mydrive\Desktop\PyScripts\generators.py", line 14, in <module>
    print gen.next()
StopIteration
>>>

You can guess the reasons for the last traceback. We got out of fuel. In order to avoid this, you can always use generator object to iterate.


def square(N):
    for i in range(1, N+1):

        yield i**2

gen_object = square(5)

for result in gen_object:

    print result

Result:

1
4
9
16
25


More fun with generators:

You can also make generator functions from list comprehensions. Use parenthesis instead of square bracket to make a gen function.


l = (x**2 for x in range(1,6))  

print l
for value in l:
    print value


Result:

<generator object <genexpr> at 0x0000000002D50F78>
1
4
9
16
25

Python: Running executables using subprocess module

There are many ways to run a bat/exe file but the most widely used is the subprocess module.
And I would like to clearly show, how exactly to do that. When should you use what arguments.
I am going to specifically talk about the check_output module .

Note : This function is used when we want the output of the bat file collected inside a variable.

Condition 1: The exe file is in the same location as of the python script that wants to run it.

Explanation: 
You are in a Folder > MyPrograms  . And the bat/exe file lies inside this folder. (Refer picture below)

Single folder with bat and python files

Content of the hello.bat file


@echo off
echo "This is a test bat"


My script contents:


import subprocess
result = subprocess.check_output('hello.bat')
print result


Now if you run it , the result will be a:

"This is a test bat"

This is easy. Let's look at another condition.



Condition 2: The exe file is in a different location from the python script that wants to run it.

Two diff folders with python script and bat file

No change in contents of hello.bat file. Some change in myscript.py file.


import subprocess
result = subprocess.check_output('hello.bat', shell=True, cwd=r'D:\inchowar\Desktop\temp')
print result


The trick here is, cwd (current working directly) argument helps you automatically switch switch to that location specified before it tries to find the bat file . 

Otherwise you would have to do it yourself . Change directory before you run the bat file using os.chdir(r'D:\inchowar\Desktop\temp'), and then run the bat file, Obviously, this becomes tedious.

Note: When you are using the cwd argument, the bat file should not have a path before it. Only the name of bat file i.e hello.bat. NOT c:\Temp\hello.bat

As to when to use shell=True, I think this article has much better explanation than I can provide here.

Thanks for reading.

Regular Expressions with Python: Look ahead and Look behind

These two important tools have been bugging me every time I come up with a situation. Because I never remember this one. Blame my memory.
So it's time I cement this idea somewhere . And no better place than a blog page.

LOOK AHEAD

LOOK AHEAD means , I want to see what lies ahead first, and then decide, to do something now.


Syntax of regex pattern:  pattern1(?=pattern2)


Where Pattern1 is the pattern for the part that we ACTUALLY want to capture. Pattern2 is the pattern which needs to be found as MANDATORY. Logically speaking, 

IF PATTERN2 is FOUND, then print/get/capture/show PATTERN1


Example: Let there be a string "Hello World"

Aim: I want to find 'Hello' ...ONLY IF its followed by 'World'



re.search(r'\w+(?= World)', 'Hello World').group()

Result : >> 'Hello'

Explanation:

  •  The \w+ is Pattern1
  •  The 'World' is Pattern2
  •  r'\w+(?= World)' means, Find anything which fits \w+ IF it is followed by 'World'

Rules:
  • The Pattern2 needs to be in parenthesis.
  • If pattern2 itself has parenthesis, then they need to be bracketed. i.e [ ( ] or [ ) ]

LOOK BEHIND


Its just the opposite of the above. It means , I want to see what lies behind me , and then decide to do something.



Syntax of regex pattern:  (?<=pattern1)pattern2


Where pattern2 is the pattern for the part that we ACTUALLY want to capture. Pattern1 is the pattern which needs to found as MANDATORY.


IF PATTERN1 is FOUND, then print/get/capture/show PATTERN2


Example : We will take the same example. 'Hello World'

Aim : I want to find 'World' ONLY if it is preceded by 'Hello ' (The space here also counts)

>>> re.search(r'(?<=Hello )\w+', 'Hello World').group()

Result : 'World'

Rules: 
  • Pattern1 needs to be in parenthesis
  • If pattern2 itself has parenthesis, then they need to be bracketed. i.e [ ( ] or [ ) ]


LOOK AHEAD & LOOK BEHIND COMBINED

Consume only if it is surrounded by the things we want.



Syntax of regex pattern: (?<=pattern_behind)pattern_middle(?=pattern_ahead)


Where pattern_middle is the pattern for the part that we ACTUALLY want to capture. Pattern_behind and Pattern_ahead are patterns which need to be found as MANDATORY.


IF PATTERN_AHEAD AND IF PATTERN_BEHIND are BOTH found, consume PATTERN_MIDDLE


Example: We will take a new example . 'Hello My World'

Aim: I want to find any word that occurs in between Hello & World


>>> re.search('(?<=Hello )\w+(?= World)', 'Hello My World').group()

Result : 'My'

More examples:


Problem : Remove all special symbols with a space,  that come in between two alphanumeric character.
Target string'This$#is% Matrix#  %!'


re.sub(r'(?<=[a-zA-Z0-9])[$#@%^\s]+(?=[a-zA-Z0-9])', ' ', 'This$#is% Matrix#  %!')

Explanation: For alphanumeric character, we used [a-zA-Z0-9]

To find special characters between them , we use a combo of look ahead and look behind.

Regular Expressions with Python: Examples of common functions

Regular Expressions are tools to filter and grab information from a heap of strings. Python has got a lot of toys to deal with it. The module "re". I am going to give you a quick ride through it. The ones that we use/gonna use the most.

So just immediately launch your favourite IDE and type "import re". Let's get started.

re.search:


for the object r where r = re.search(pattern, subject),

r.start() - Gives the start point where the match was found.
r.end() - Gives the end point where the match was found.
r.group(0) - Returns the full matched string
r.group(1) - Returns the matched string for the 1st group
r.group(2) - Returns the matched string for the 2nd group

And so on....

Note : r.group() is same as r.group(0)

The number of groups available depends on the number of groups used in your pattern.

Example:


>>> regex = r"([a-zA-Z]+) (\d+)"
>>> re.search(regex, 'July 4')
<_sre.SRE_Match object at 0x0000000002E4D140>
>>> r = re.search(regex, 'July 4')
>>> r.end()
6
>>> r.start()
0
>>> 'July 4'[0:6]
'July 4'
>>> 'July 4'[0:5]
'July '
>>> r.group(0)
'July 4'
>>> r.group(1)
'July'
>>> r.group(2)
'4'
>>> r.group(3)

Traceback (most recent call last):
  File "<pyshell#12>", line 1, in <module>
    r.group(3)
IndexError: no such group

re.findall


>>> r = re.findall(regex, 'July 4 August 10 Sept')
>>> r
[('July', '4'), ('August', '10')]

Same example with findall...observe how we got two results since there are two patterns in the string .

Noteworthy : If you are using groups with findall, then the result will be a list containing tuples as above.
But if you are just using single regex (no grouping), the result will be a list with strings .

Example:


>>> re.findall('\w+', 'July 4 August 10')
['July', '4', 'August', '10']

re.finditer

The finditer returns an iterator of re.MatchObjects.


>>> it = re.finditer('\w+', 'July 4 August 10')
>>> for i in it:
        print 'July 4 August 10'[i.start():i.end()]

>>> for i in it:
       print i.group()

 
July
4
August
10

re.compile

If you want to repeat searching with a common pattern , its better to create a separate object (known as compiled regular expression object).

Python official help says:

Compile a regular expression pattern, returning a pattern object.


>>> patt = re.compile('abc')


Lets use the 'patt' object now.


>>> r = patt.search('abcXYZabc1234hjhjkabc')
>>> r.group()
'abc'
>>> r.start()
0
>>> r.end()
3

Things to note:
  • Search only needs a target search string as argument. 
  • Functions start() and end() returns the start index number , and end return the position as i + 1 starting from 0.
Now what if you want to know if there are more matches possible in the same string with our pattern.

>>> r = patt.search('abcXYZabc1234hjhjkabc', 1)
>>> r.group()
'abc'
>>> r.start()
6
>>> r.end()
9

As you can see, I can now traverse in the string to find all possible matches.

re.sub

If you want to replace some text that matches a certain pattern , the re.sub functions comes handy.


So by the syntax, we need to provide a pattern , then a text to be replaced if a match is found, and a target string. Let's see a simple example.


>>> string = "Let have a big party on my birthday"
>>> re.sub('(?<=\s)\w', 'X', string)
"Let's Xave X Xig Xarty Xn Xy Xirthday"

Explanation:

  • We have a sentence. I want to replace all first letters with some letter only if that letter has a space before it. 
  • The pattern : (?<=\s)\w finds all letters that have a space before it. 
  • 'x' is the thing to be replaced with.
  • The variable "string" is the target. 
  • The result shown is the return of the re.sub(). It needs to be stored if you want to use it. The variable "string" remains unchanged.


Art of Unittest writing : Auto-generation of unit test cases


How to auto create test cases with minimum coding 

Lets say , you want to run a series of test cases that undergo similar kind of testing  . but you need to generate different test cases for each comparison (or whatever test procedure you are using).

So imagine. We want to test a function. A function that accepts two variables, and an operator for comparison. Here's our test function:

    def _test_func(self, a, b, exp, operator):
        ans = 0
        exec("ans = %d %s %d" % (a, operator, b))
        self.assertEqual(ans, exp)


We want to test above function with various inputs....And...we want each scenario as a test case...You might need it for test report or something...


import unittest
from collections import namedtuple

class TestAuto(unittest.TestCase):
    def setUp(self):
        pass

    def tearDown(self):
        pass

# Adder function to add test case
def _add_test(cls, generator):
    for func, name, doc, a, b, exp, operator in generator():
        test = lambda self, a=a, b=b,  expected=exp, oper=operator, func=func : func(self, a, b, expected, oper)
        test.__name__ = "test_{}".format(name)
        test.__doc__ = doc
        setattr(cls, test.__name__, test)

def _test_generator_normal():
    """Generator function for testing queries"""        


    def _test_func(self, a, b, exp, operator):
        ans = 0
        exec("ans = %d %s %d" % (a, operator, b))
        self.assertEqual(ans, exp)

       
    tr = namedtuple('TestCase', 'name doc a b exp operator')
    test_list = (
        tr(name="sub", doc='Test for sub', a=10, b=20, exp=-10,  operator='-'),
        tr(name="add", doc='Test for add', a=30, b=50, exp=80,  operator='+'),
        tr(name="div", doc='Test for divide', a=10, b=2, exp=5, operator='/')
        )
    for ti in test_list:
        yield _test_func, ti.name, ti.doc, ti.a, ti.b, ti.exp, ti.operator

_add_test(TestAuto, _test_generator_normal)

if __name__ == '__main__':
    unittest.main(verbosity=2)


Results:


>>>
test_add (__main__.TestAuto)
Test for add ... ok
test_div (__main__.TestAuto)
Test for divide ... ok
test_sub (__main__.TestAuto)
Test for sub ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.010s

OK

Recently , we (me and my friend Sumant)came up with one more simpler model for the same purpose.


import unittest

def generator(test_class, a, b):
    def test(self):
        self.assertEqual(a, b)
    return test

def add_test_methods(test_class):
    #First element of list is variable "a", then variable "b", then name of test case that will be used as suffix.
    test_list = [[2,3, 'one'], [5,5, 'two'], [0,0, 'three']]
    for case in test_list:
        test = generator(test_class, case[0], case[1])
        setattr(test_class, "test_%s" % case[2], test)
        

class TestAuto(unittest.TestCase):
    def setUp(self):
        print 'Setup'
        pass

    def tearDown(self):
        print 'TearDown'
        pass

_add_test_methods(TestAuto)  # It's better to start with underscore so it is not detected as a test itself

if __name__ == '__main__':
    unittest.main(verbosity=1)


RESULT:



>>>
Setup
FTearDown
Setup
TearDown
.Setup
TearDown
.
======================================================================
FAIL: test_one (__main__.TestAuto)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:/inchowar/Desktop/PyTrash/test_auto_3.py", line 5, in test
    self.assertEqual(a, b)
AssertionError: 2 != 3

----------------------------------------------------------------------
Ran 3 tests in 0.019s

FAILED (failures=1)

In case , you want to be extra cautious, you can use the decorator @nottest to indicate that the function is not a test method.

"from nose.tools import notttest"

Total Pageviews