python

This differs from the site mentioned above. I do the same for every project so when I'm in each of my project directories they all have an env directory. Then everything is the same. This keeps things simple and easily remembered.

Step#3: start the virtual env - run the following from the project directory

$ source ./env/bin/activate

Once your virtual environment is activated anything you install will be put into the environment. If you look you'll notice new directories inside of ./env/lib/python2.7/site-packages after each install.

Now, install what you need. For example,

$ pip install zipline

To see all the libraries in your environment you can run pip list.

$ pip list

When you run python in your active environment you'll have access to all the libraries you've just installed.

Lastly, to get out of the virtual env you can run deactivate.

$ deactivate

https://tryolabs.com/blog/2015/12/15/top-10-python-libraries-of-2015/

https://tryolabs.com/blog/2016/12/20/top-10-python-libraries-of-2016/

https://tryolabs.com/blog/2017/12/19/top-10-python-libraries-of-2017/

https://python-3-patterns-idioms-test.readthedocs.io/en/latest/index.html

http://dev.mysql.com/doc/connector-python/en/connector-python-installation.html

pip install mysql-connector

http://dev.mysql.com/downloads/connector/odbc/

https://pythondigest.ru/

https://hashedin.com/training/designing-modules-in-python-ebook/

https://www.pythonsheets.com/

https://awesome-python.com

http://code.activestate.com/recipes/580705-get-names-and-types-of-all-attributes-of-a-python-/

https://jeffersonheard.github.io/2016/11/simplifying-complex-business-logic-with-pythons-kanren/

http://www.grantjenks.com/docs/sortedcontainers/

https://habrahabr.ru/post/319200/

https://habrahabr.ru/post/319876/

https://habrahabr.ru/post/320288/

http://odo.readthedocs.io/en/latest/

# FIND 2 elements in list where sum=target

l=[6,2,3,4,5,1]

target=6

d={}

for el in l:

if d.has_key(target-el):

print "Found:", el, d[target-el]

else:

d[el]=el

################

print " Dictionary examples"

################

print " dict traversali #1"

for key in d.keys():

print "key=", key, "value=", d[key]

print " dict traversali #2"

for key, value in d.items():

print "key=", key, "value=", value

missing=d.get(15,0) //default

print "missing key=15 default value=", missing

print d.keys()

print d.values()

print d.items()

print "Count the number of times each word is seen in a file"

myfile=("A","B","C","D","A")

words = {}

for word in myfile:

occurrences = words.setdefault(word, 0)

words[word] = occurrences + 1

for key, value in words.items():

print "key=", key, "value=", value

##################

print " List examples"

###############

print "list=", l

l.sort()

print "sorted=", l

l.sort(reverse=True)

print "reverse sorted=",l

#################

print " SET examples"

################

my_list=[1,2,2,3,3,3,4]

a=set(my_list)

print a

b=set([3,4,22])

print b.issubset(a)

print a.difference(b)

print b.difference(a)

if len(a.difference(a)) == 0:

print "No difference with myself"

a.discard(4)

a.discard(8)

union= a | b

print union

print a.union(b)

####################

print "String examples"

####################

s='abcdef'

print len(s)

print "find example exists", s.find('a')

print "find example not exists", s.find('g')

try:

s.index('g')

except:

print "exceptioni in index()"

else:

print "else"

def permutations(word):

if len(word)<=1:

return [word]

#get all permutations of length N-1

perms=permutations(word[1:])

char=word[0]

result=[]

#iterate over all permutations of length N-1

for perm in perms:

#insert the character into every possible location

for i in range(len(perm)+1):

result.append(perm[:i] + char + perm[i:])

return result

print permutations("123")

Sorting

lst = ["a", 6.99], ["b", 1.49], ["c", 7.99], ["d", 0.49]

for x, y in sorted(lst, key=lambda x: x[1]): print x

for x, y in sorted(lst, key=lambda x: x[0]): print x

String formatting

cycle_val=10

station_val='XXX'

SQL="""

SELECT a FROM A WHERE cycle={cycle_param} and station='{station_param}' and x={cycle_param}

""".format(cycle_param=cycle_val, station_param = station_val)

Concurrency

http://n-s-f.github.io/2016/12/23/starmap-pattern.html

https://habrahabr.ru/post/314606/ Async

http://masnun.rocks/2016/10/06/async-python-the-different-forms-of-concurrency/

https://www.oreilly.com/learning/thinking-in-coroutines

http://www.jacksimpson.co/2016/11/23/multiprocessing-in-python/

http://stackoverflow.com/questions/372885/how-do-i-connect-to-a-mysql-database-in-python

https://github.com/jeffknupp/sandman2 REST API FOR databases

https://github.com/hashedin/jinjasql

Conferences

https://www.youtube.com/channel/UC51aOZF5nnderbuar5D5ifw/videos?sort=dd&view=0&shelf_id=0

https://www.youtube.com/results?search_query=scipy+2016

https://www.youtube.com/playlist?list=PLYx7XA2nY5Gcpabmu61kKcToLz0FapmHu

http://awesome-python.com/

http://book.pythontips.com/en/latest/

https://news.ycombinator.com/item?id=12180301 PyObject

http://www.blog.pythonlibrary.org/2016/04/20/python-201-an-intro-to-itertools/

Know the Python Path

Knowing about the PYTHONPATH is key to installing and importing third-party packages. When an import command is passed, python looks for the module/package in a list of places. The default path(s) where python would search for modules can be found out by:

For my computer it gives me the the following paths:

[”, ‘C:\\Python27\\Lib\\idlelib’, ‘C:\\Windows\\system32\\python27.zip’, ‘C:\\Python27\\DLLs’, ‘C:\\Python27\\lib’, ‘C:\\Python27\\lib\\plat-win’, ‘C:\\Python27\\lib\\lib-tk’, ‘C:\\Python27’, ‘C:\\Python27\\lib\\site-packages’]

Starters would do good to install or add modules in one of these paths. If you get this wrong, you will get an error like: ImportError: No module named “foo” . By convention, all third-party packages go into Pythonfolder/Lib/site-packages.

Importing Module or Importing packages ?

When you add a third-party module to your python library, remember that simply dumping a folder with bunch of .py files will not make it a package. When you use the import command, python does not look inside folders, even if the folders are located in the PYTHONPATH. However, if a folder in the PythonPath has a file called __init__.py, Python jumps inside it and treats the complete folder as a package.

Changing the Pythonpath

If you want to add packages in any other directory (other than the python paths listed above) – you will have to first export the python path in your code with:

Hardcoding path

ImportError: No module named 'monster'

import sysprint(sys.path)sys.path.append("F:\\USERS\BR\Documents\PYTHON\OBJECTS")print(sys.path)

Python 3+ Python 2 on same windows box

https://docs.python.org/3.5/using/windows.html#python-launcher-for-windows

PYTHONPATH=$PYTHONPATH:C/Your new path/yournewmodulefolder

import sys

print sys.path

After installation of Python 3.3, the py.exe and pyw.exe is copied to your c:\Windows directory, and the associations are set for the .py extension so that it uses the launcher. By default, Python 2 is launched for py script.py. The py -3 script.py launches Python 3. (This also means that no path for Python must be added to the environment -- the C:\Windows already is in the PATH.)

The best of all is that #!python2 in the script causes lauching via Python 2, the #!python3 causes launching the script via Python 3. This way, you can use scripts for both versions of Python, and you can lauch them the unified way -- py script.py or by just clicking on the script icon.

https://bespohk.com/blog/configuring-python-visual-studio-code

http://book.pythontips.com/en/latest/

http://www.grantjenks.com/docs/sortedcontainers/pycon-2016-talk.html

https://www.reddit.com/r/Python/comments/4l3in7/nice_curation_of_19_free_ebooks_to_learn_python/

http://textbooks.opensuny.org/introduction-to-the-modeling-and-analysis-of-complex-systems/

https://python.libhunt.com/

https://news.ycombinator.com/item?id=11526563

http://akaptur.com/blog/2013/12/03/introduction-to-the-python-interpreter-4/

https://habrahabr.ru/post/281332/ plugins in python

http://www.blog.pythonlibrary.org/2016/01/26/python-101-how-to-traverse-a-directory/

http://brunorocha.org/python/microservices-with-python-rabbitmq-and-nameko.html

https://github.com/domspad/ctci_solutions

RPC with python

http://alex.vector57.net/python-jrpc/

https://rpyc.readthedocs.org/en/latest/

'correct' way to install package on windows is to first install Anaconda Python (https://www.continuum.io/downloads) and then:

0) Check if the package came preinstalled with Anaconda

1) if not try to use conda to install packages,

2) if conda doesn't have the package, see if it's been uploaded here:http://www.lfd.uci.edu/~gohlke/pythonlibs/

3) try pip.

import sys print(sys.prefix) print(sys.executable) print(sys.path)

s="aaa {} bbb {}"

s.format('X', 'Y')

OUTPUT 'aaa X bbb Y'

g="aaa {1} bbb {0}"

g.format('X', 'Y')

OUTPUT 'aaa Y bbb X'

name = "John"

age = 23

print "%s is %d years old." % (name, age)

John is 23 years old.

http://pythonwise.blogspot.com/2015/01/using-supervisord-to-manage-you-daemons.html

https://techietweak.wordpress.com/2015/11/11/python-collections/

http://asvetlov.blogspot.com/

https://faster-cpython.readthedocs.org/

http://eli.thegreenplace.net/tag/python-internals

https://www.youtube.com/watch?v=lYe8W04ERnY

http://talkpython.fm/episodes/show/22/cpython-internals-and-learning-python-with-pythontutor.com

https://codedeposit.wordpress.com/2015/10/28/building-a-shared-library-in-c-and-using-it-in-a-python-program/

http://awesome-python.com/

https://automatetheboringstuff.com/

https://news.ycombinator.com/item?id=10688556 working with Binary data

http://toly.github.io/blog/2014/02/13/parallelism-in-one-line/

2345

from collections import defaultdictcolors = ["brown", "red", "green", "yellow", "yellow", "brown", "brown", "black"]color_counts = defaultdict(int)for c in colors: color_counts[c] += 1

from collections import Countercolors = ["brown", "red", "green", "yellow", "yellow", "brown", "brown", "black"]color_counts = Counter(colors)

colors = ["brown", "red", "green", "yellow", "yellow", "brown", "brown", "black"]color_counts = dict.fromkeys(colors, 0)for c in colors: color_counts[c] += 1

https://gumroad.com/l/WOvyt

http://jnosal.logdown.com/posts/286714-python-in-production-i-linux

http://blog.dimroc.com/2015/05/07/etl-language-showdown-pt2/

https://news.ycombinator.com/item?id=10145880 tabular data in python

Algo

https://github.com/ztgu/sorting_algorithms_py

http://incolumitas.com/2015/01/24/implementing-two-graph-traversal-algorithms-in-python-depth-first-search-and-breadth-first-search/

https://github.com/QuantumFractal/Data-Structure-Zoo

https://github.com/lorien/awesome-web-scraping/blob/master/python.md

python -m SimpleHTTPServer

Serving HTTP on 0.0.0.0 port 8000 ...

Open browser and put in any of the following addresses:

http://your_ip_address:8000

http://127.0.0.1:8000

If you wish to change the port that's used start the program via:

$ python -m SimpleHTTPServer 8080

check if element exists in list

Number of rows in file

nrows = sum(1 for _ in open('POIWorld.csv'))

found = el in somelist # returns True/False

index= somelist.index(el) if el in somelist else None

>>> d={ 'a':33, 'b':44} >>> d {'a': 33, 'b': 44} >>> d['a'] 33 >>> d.values() [33, 44] >>> d['c']=33 >>> set(d.values()) set([33, 44]) >>> for key in d: ... print key, d[key] ... a 33 c 33 b 44 >>> d.keys() ['a', 'c', 'b'] >>> for k, v in d.iteritems(): ... print k,v ... a 33 c 33 b 44 >>> l=[3,5,1] >>> l.sort() >>> l [1, 3, 5]

http://www.oreilly.com/programming/free/files/functional-programming-python.pdf

https://habrahabr.ru/post/305750/ functional python lib

http://docs.quantifiedcode.com/python-anti-patterns/index.html

https://news.ycombinator.com/item?id=9896369 good python code to read

https://www.youtube.com/watch?v=x-kB2o8sd5c

http://www.reddit.com/r/Python/comments/32yvjf/experienced_python_users_whats_the_most_recent/

https://freepythontips.wordpress.com/2015/04/19/nifty-python-tricks/

http://sahandsaba.com/thirty-python-language-features-and-tricks-you-may-not-know.html

http://blog.stuartowen.com/pipelining-a-successful-data-processing-model

http://lwn.net/SubscriberLink/643786/7dbcc64524d979b1/ await/async

https://github.com/duerrp/pyexperiment

pyenv does basically this:

mkdir libs pip install -t libs <package> PYTHONPATH=libs python app.py

Regular Expression

http://tonysyu.github.io/readable-regular-expressions-in-python.html

http://avxhome.se/ebooks/programming_development/3601177.html

Celsius = [39.2, 36.5, 37.3, 37.8] Fahrenheit = map(lambda x: (float(9)/5)*x + 32, Celsius)

fib = [0,1,1,2,3,5,8,13,21,34,55] result = filter(lambda x: x % 2, fib)

reduce:

http://www.python-course.eu/lambda.php

http://www.bogotobogo.com/python/python_interview_questions_2.php

python utils

https://github.com/mahmoud/boltons

http://www.reddit.com/r/Python/comments/3ej9u2/small_functions_that_live_in_my_pythonpath_share/

https://github.com/cool-RR/python_toolbox

http://www.reddit.com/r/Python/comments/32yvjf/experienced_python_users_whats_the_most_recent/

https://codefisher.org/catch/blog/2015/04/22/python-how-group-and-count-dictionaries/

https://github.com/faif/python-patterns

http://stackoverflow.com/questions/14257373/skip-the-headers-when-editing-a-csv-file-using-python

Python version management

https://github.com/qw3rtman/p/

Automated UI for Python

https://github.com/mfitzp/Wooey

http://pyinformatics.blogspot.com/2015/05/djangui-empower-your-python-scripts.html

https://github.com/nucleic/enaml

http://www.reddit.com/r/Python/comments/3bak0g/sandals_a_gui_wrapper_for_tkinter_inspired_by_the/

http://chimera.labs.oreilly.com/books/1230000000393/index.html Python CookBook

https://www.youtube.com/watch?v=OcSrJTdNlPQ

https://www.youtube.com/channel/UCgxzjK6GuOHVKR_08TT4hJQ PyCon 2015

https://www.youtube.com/watch?v=IGwNQfjLTp0 Hash functions

http://pansop.com/1019/ common pitfalls

http://nedbatchelder.com/text/names1.html

https://www.youtube.com/watch?v=HTLu2DFOdTg

https://www.youtube.com/watch?v=YAFGQurdJ3U Distributed Systems 101

http://www.fullstackpython.com/websockets.html

https://codefisher.org/catch/blog/2015/02/10/python-decorators-and-context-managers/

http://wooey.martinfitzpatrick.name/

http://intermediatepythonista.com/the-function

http://intermediatepythonista.com/object-orientation-in-python

http://intermediatepythonista.com/classes-and-objects-ii-descriptors

http://blog.lerner.co.il/python-attributes/

http://www.reddit.com/r/Python/comments/316dm5/simple_profound_python_tricks/

https://pypi.python.org/pypi/sortedcontainers

http://lucumr.pocoo.org/2015/4/8/microservices-with-nameko/

https://csvkit.readthedocs.org/en/0.9.0/

http://ruslanspivak.com/lsbaws-part2/ let build a web server

EXTERNAL

https://amoffat.github.io/sh/

http://fastml.com/how-to-run-external-programs-from-python-and-capture-their-output/

http://pythonadventures.wordpress.com/2014/01/08/capture-the-exit-code-the-stdout-and-the-stderr-of-an-external-command/

http://blog.endpoint.com/2015/01/getting-realtime-output-using-python.html

import sys

import shlex

from subprocess import Popen, PIPE

def get_exitcode_stdout_stderr(cmd):

args = shlex.split(cmd)

proc = Popen(args, stdout=PIPE, stderr=PIPE)

out, err = proc.communicate()

exitcode = proc.returncode

return exitcode, out, err

def main(params):

cmd = ' '.join(params)

exitcode, out, err = get_exitcode_stdout_stderr(cmd)

if __name__ == "__main__":

if len(sys.argv) < 2:

print('Usage: {} <external_command>'.format(sys.argv[0]))

sys.exit(1)

main(sys.argv[1:])

http://scottsievert.github.io/blog/2014/07/30/simple-python-parallelism/

How to remove all entries of character from string:

newstr = oldstr.replace("M", "")

https://www.airpair.com/python/posts/python-tips-and-traps

https://docs.python.org/2/tutorial/modules.html

http://sebastianraschka.com/

Idiomatic Python

https://github.com/kennethreitz/python-guide

http://safehammad.com/downloads/python-idioms-2014-01-16.pdf

http://preshing.com/20110920/the-python-with-statement-by-example/

https://news.ycombinator.com/item?id=8665820

https://speakerdeck.com/pyconslides/transforming-code-into-beautiful-idiomatic-python-by-raymond-hettinger-1

http://www.dtic.mil/dtic/tr/fulltext/u2/a608708.pdf plugin in python

http://www.reddit.com/r/Python/comments/2nneix/how_to_work_with_csv_like_a_twodimensional_array/

http://stackoverflow.com/questions/3828723/why-we-need-sys-setdefaultencodingutf-8-in-a-py-script

LRU

http://www.kunxi.org/blog/2014/05/lru-cache-in-python/ LRU cache

http://www.blog.pythonlibrary.org/2016/02/25/python-an-intro-to-caching/

http://www.reddit.com/r/Python/comments/26pxwk/lru_cache_in_python/

http://www.bitdance.com/blog/2014/09/30_01_asycio_overview/

https://speakerdeck.com/lukasa/a-deep-dive-into-python-requests

pip search yaml # to find project on pip

http://prooffreaderplus.blogspot.com/2014/11/top-10-python-idioms-i-wished-id.html

with open(file_name,'r') as infile:

count = 0 #bad practice

for line in infile: # use enumerate here!!!

count = count + 1 # bad practice

words = line.rstrip().split('\t')

# good practice

with open('foobar.txt') as f:

for lineno, line in enumerate(f, start=1):

print('Reading line {0}'.format(lineno))

inotify

https://github.com/joh/when-changed (inotify)

http://timstaley.co.uk/posts/automated-incoming-file-processing-with-python/

https://github.com/kctess5/file_watcher

https://gist.github.com/swarminglogic/8963507

http://git.z3bra.org/cgit.cgi/wendy/

https://github.com/austinpickett/watch-directory

namedtuple

http://users.livejournal.com/_winnie/447767.html

https://www.reddit.com/r/Python/comments/44fu16/using_namedtuple_as_a_poormans_parser/

http://habrahabr.ru/post/247843/ how dictionary is implemented in python

# Copy all files from local directory to a server using FTPfrom ftplib import FTP import os # ftp settings settings = { 'ftp': { 'url': 'ftp.some-server.com', 'username': 'your-account-name', 'password': 'your-password', 'remote-directory': '/path/to/files' }} # local paths paths = { 'local-directory': 'my-files/'} # list of local files files = os.listdir(paths['local-directory']) # connect and storefor f in files: ftp = FTP(settings['ftp']['url']) ftp.login(settings['ftp']['username'], settings['ftp']['password']) ftp.cwd(settings['ftp']['remote-directory']) ftp.storbinary('STOR ' + f, open(paths['local-directory'] + f, 'rb')) ftp.close()

http://www.leaseweblabs.com/2014/07/measuring-context-switching-linux-application/

https://news.ycombinator.com/item?id=7715349

http://www.toptal.com/python/top-10-mistakes-that-python-programmers-make

http://pyvideo.org/video/1780/transforming-code-into-beautiful-idiomatic-pytho

https://gist.github.com/JeffPaine/6213790

http://lignos.org/py_antipatterns/

http://easy-python.readthedocs.org/en/latest/

http://www.infoq.com/presentations/A-Few-of-My-Favorite-Python-Things

http://habrahabr.ru/company/wargaming/blog/202170/

https://github.com/vinta/awesome-python

http://matthewrocklin.com/blog/work/2014/05/01/Fast-Data-Structures/

http://www.fullstackpython.com/

http://stevenloria.com/three-command-line-tools-for-productive-python-development/

http://www.packtpub.com/mastering-object-oriented-python/book

Video

http://around-technology.blogspot.com/2014/03/a-collection-of-python-must-reads.html

http://jessenoller.com/good-to-great-python-reads/

https://speakerdeck.com/pycon2014

https://www.youtube.com/channel/UCadZ6_NWdCN6YolgQdfV8Pg PyEuro2004

timeout (see man timeout as well)

https://pypi.python.org/pypi/stopit Timeout control decorator and context managers, raise any exception in another thread

https://wiki.python.org/moin/PythonDecoratorLibrary#Function_Timeout

MS EXCEL

http://www.devdungeon.com/content/working-spreadsheets-python

http://www.blog.pythonlibrary.org/2014/04/30/reading-excel-spreadsheets-with-python-and-xlrd/ MS EXCEL

xls_file='LPO1606161658_alumina_screen.xlsx'

import xlrd

workbook = xlrd.open_workbook(xls_file)

for sheet_name in workbook.sheet_names():

print sheet_name

#worksheet = workbook.sheet_by_name('My_Sheet_Name')

n_sheets = workbook.nsheets

print "######### Total # of sheets=", n_sheets

for i in range (n_sheets):

sheet = workbook.sheet_by_index(i)

print sheet.name, sheet.nrows

for s in workbook.sheets():

if s.name in ['Statistics_1-029','Channel_1-029']:

print s.name , " number of rows=", s.nrows

for row in range (s.nrows):

values=[]

for col in range (s.ncols):

values.append(s.cell(row,col).value)

#print row, ','.join(str(values))

print row, values

>>> xls_file='LPO1606161658_alumina_screen.xlsx'

>>>

>>> import xlrd

>>> workbook = xlrd.open_workbook(xls_file)

>>> dir(workbook)

['__class__', '__delattr__', '__dict__', '__doc__', '__enter__', '__exit__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_all_sheets_count', '_all_sheets_map', '_externsheet_info', '_externsheet_type_b57', '_extnsht_count', '_extnsht_name_from_num', '_repr_these', '_resources_released', '_rich_text_runlist_map', '_sh_abs_posn', '_sharedstrings', '_sheet_list', '_sheet_names', '_sheet_num_from_name', '_sheet_visibility', '_sheethdr_count', '_supbook_addins_inx', '_supbook_count', '_supbook_locals_inx', '_supbook_types', '_xf_epilogue_done', '_xf_index_to_xl_type_map', 'actualfmtcount', 'addin_func_names', 'biff2_8_load', 'biff_version', 'builtinfmtcount', 'codepage', 'colour_map', 'countries', 'datemode', 'derive_encoding', 'dump', 'encoding', 'fake_globals_get_sheet', 'filestr', 'font_list', 'format_list', 'format_map', 'formatting_info', 'get2bytes', 'get_record_parts', 'get_record_parts_conditional', 'get_sheet', 'get_sheets', 'getbof', 'handle_boundsheet', 'handle_builtinfmtcount', 'handle_codepage', 'handle_country', 'handle_datemode', 'handle_externname', 'handle_externsheet', 'handle_filepass', 'handle_name', 'handle_obj', 'handle_sheethdr', 'handle_sheetsoffset', 'handle_sst', 'handle_supbook', 'handle_writeaccess', 'initialise_format_info', 'load_time_stage_1', 'load_time_stage_2', 'logfile', 'mem', 'name_and_scope_map', 'name_map', 'name_obj_list', 'names_epilogue', 'nsheets', 'on_demand', 'palette_record', 'parse_globals', 'props', 'ragged_rows', 'raw_user_name', 'read', 'release_resources', 'sheet_by_index', 'sheet_by_name', 'sheet_loaded', 'sheet_names', 'sheets', 'style_name_map', 'unload_sheet', 'use_mmap', 'user_name', 'verbosity', 'xf_list', 'xfcount']

>>> sheet = workbook.sheet_by_index(0)

>>> dir(sheet)

['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_cell_attr_to_xfx', '_cell_types', '_cell_values', '_cell_xf_indexes', '_dimncols', '_dimnrows', '_first_full_rowx', '_ixfe', '_maxdatacolx', '_maxdatarowx', '_position', '_repr_these', '_xf_index_stats', '_xf_index_to_xl_type_map', 'automatic_grid_line_colour', 'bf', 'biff_version', 'book', 'bt', 'cached_normal_view_mag_factor', 'cached_page_break_preview_mag_factor', 'cell', 'cell_note_map', 'cell_type', 'cell_value', 'cell_xf_index', 'col', 'col_label_ranges', 'col_slice', 'col_types', 'col_values', 'colinfo_map', 'columns_from_right_to_left', 'computed_column_width', 'cooked_normal_view_mag_factor', 'cooked_page_break_preview_mag_factor', 'default_additional_space_above', 'default_additional_space_below', 'default_row_height', 'default_row_height_mismatch', 'default_row_hidden', 'defcolwidth', 'dump', 'fake_XF_from_BIFF20_cell_attr', 'first_visible_colx', 'first_visible_rowx', 'fixed_BIFF2_xfindex', 'formatting_info', 'gcw', 'get_rows', 'gridline_colour_index', 'gridline_colour_rgb', 'handle_feat11', 'handle_hlink', 'handle_msodrawingetc', 'handle_note', 'handle_obj', 'handle_quicktip', 'handle_txo', 'has_pane_record', 'horizontal_page_breaks', 'horz_split_first_visible', 'horz_split_pos', 'hyperlink_list', 'hyperlink_map', 'insert_new_BIFF20_xf', 'logfile', 'merged_cells', 'name', 'ncols', 'nrows', 'number', 'panes_are_frozen', 'put_cell', 'put_cell_ragged', 'put_cell_unragged', 'ragged_rows', 'read', 'remove_splits_if_pane_freeze_is_removed', 'req_fmt_info', 'rich_text_runlist_map', 'row', 'row_label_ranges', 'row_len', 'row_slice', 'row_types', 'row_values', 'rowinfo_map', 'scl_mag_factor', 'sheet_selected', 'sheet_visible', 'show_formulas', 'show_grid_lines', 'show_in_page_break_preview', 'show_outline_symbols', 'show_sheet_headers', 'show_zero_values', 'split_active_pane', 'standardwidth', 'string_record_contents', 'tidy_dimensions', 'update_cooked_mag_factors', 'utter_max_cols', 'utter_max_rows', 'verbosity', 'vert_split_first_visible', 'vert_split_pos', 'vertical_page_breaks', 'visibility']

Count the frequency of words using Counter

>>>from collections import Counter >>>frequency = Counter(words) >>>print(frequency) Counter({'foo': 2, 'a': 1, 'the': 1, 'bar': 1})

http://koder-ua.blogspot.com/

http://www.youtube.com/watch?v=HTLu2DFOdTg

https://news.ycombinator.com/item?id=7319590

http://www.toptal.com/python/python-class-attributes-an-overly-thorough-guide

http://pypix.com/roundups/best-python-2013/

http://lukauskas.co.uk/articles/2014/02/12/how-to-make-python-faster-without-trying-that-much/

http://www.machinalis.com/blog/how-to-unit-test-python/ measuring time

sw Kindle Books

https://read.amazon.com/?asin=B00BT95CWM

http://www.amazon.com/gp/product/B006ZHJSIM/

http://www.jeffknupp.com/writing-idiomatic-python-ebook/

http://chimera.labs.oreilly.com/books/1230000000393/index.html CookBook

http://rhodesmill.org/brandon/slides/2013-07-pyohio/

http://www.pythonweekly.com/archive/

http://pythonarticles.com/

https://github.com/devsnd/tinytag reading mp3 headers

http://radimrehurek.com/2014/03/data-streaming-in-python-generators-iterators-iterables

HTTP from command line

https://github.com/jkbr/httpie

http://httpkit.com/resources/HTTP-from-the-Command-Line/

http://pypix.com/tools-and-tips/http-fundamentals/

JSON FROM COMMAND LINE

http://stedolan.github.io/jq/

http://www.reddit.com/r/programming/comments/1xnjug/jq_a_lightweight_and_flexible_commandline_json/

json_results = [] for result in results: d = {'sighted_at': result.sighted_at, 'reported_at': result.reported_at, 'location': result.location, 'shape': result.shape, 'duration': result.duration, 'description': result.description, 'lat': result.lat, 'lng': result.lng} json_results.append(d)

http://rhodesmill.org/brandon/

flake8 pyflakes, pylint

https://github.com/timothycrosley/frosted is fork of pyflakes

PyCharm http://pedrokroger.net/getting-started-pycharm-python-ide/

if reference == None

should be:

if reference is None

http://around-technology.blogspot.com/2014/03/a-collection-of-python-must-reads.html

http://stackoverflow.com/questions/101268/hidden-features-of-python

http://pypix.com/roundups/best-python-2013/

http://habrahabr.ru/post/207988/

try: function() except Error: # Если не сработал try и объявлена ошибка Errorelse: # Если сработал try и не сработал exceptfinally: # Выполняется в любом случае

Python allows to return functions from other functions:

def parent(num): def first_child(): return "Printing from the first_child() function." def second_child(): return "Printing from the second_child() function." try: assert num == 10 return first_child except AssertionError: return second_child foo = parent(10) bar = parent(11)print foo print bar print foo()print bar()

The output of the first two print statements is:

This simply means that foo points to the first_child() function, while bar points to thesecond_child() function.

The output of the second two functions confirms this:

Printing from the first_child() function.Printing from the second_child() function.

Let's start with this fibonacci function:

def fib(n): if n <= 1: return 1 else: return fib(n-1) + fib(n-2)

Slow as hell, right? With the fibonacci function, it's easy to turn it into a version that doesn't re-calculate entries a lot, but let's pretend that this is one of those functions where it's a lot harder to do that. What we want is to be able to remember previously-calculated results, so any recursive calls that have already been calculated can be skipped. So we can modify it like this:

fib_cache = dict() # empty associative array def cached_fib(n): if n not in fib_cache: if n <= 1: fib_cache[n] = 1 else: fib_cache[n] = cached_fib(n-1) + cached_fib(n-2) return fib_cache[n]

Now you have a cache storing all previously-calculated results. But you're also mixing the "calculating the results" part of the function with the "caching shit" part. You can separate out the caching behavior like this:

def make_cached(f): cache = dict() def cached_function(arg): # There's syntax to let you use any arguments, but for simplicity let's pretend there's always one. if arg not in cache: cache[arg] = f(arg) return cache[arg] return cached_function # NOT cached_function()

Now you have a function that takes a function and returns a function that has different behavior. You can use it like this:

cached_fib = make_cached(fib) cached_fib(100)

Python has special syntax that you can use for functions that modify functions, called "decorators":

@make_cached def whatever: # stuff

This works the same as:

def whatever: #stuff whatever = make_cached(whatever)

Import in Python

http://blog.amir.rachum.com/post/63666832095/python-importing

http://pydev.blogspot.mx/2016/11/python-import-world-anti-pattern.html

http://www.blog.pythonlibrary.org/2016/03/01/python-101-all-about-imports/

http://www.reddit.com/r/Python/comments/1o62wa/python_importing/

http://asvetlov.blogspot.com/2010/05/blog-post.html

http://asvetlov.blogspot.com/2010/05/2.html

http://eli.thegreenplace.net/2012/03/23/python-internals-how-callables-work/

Decorators

http://freepythontips.wordpress.com/2013/10/10/all-about-decorators-in-python/

http://www.codingismycraft.com/2014/05/23/python-decorators-basics/

https://github.com/GrahamDumpleton/wrapt/tree/master/blog Python Decorators

http://www.blog.pythonlibrary.org/2014/03/13/python-201-decorators/

http://blog.endpoint.com/2014/02/python-decorator-basics-part-ii.html

http://hackflow.com/blog/2013/11/03/painless-decorators/

http://www.jeffknupp.com/blog/2013/11/29/improve-your-python-decorators-explained/

http://hackflow.com/blog/2013/11/03/painless-decorators/

In python you can write this code:

@my_decorator def my_function(*args, **kwargs): pass

The @my_decorator syntax is just sugar for the following code

def my_function(*args, **kwargs): pass my_function = my_decorator(my_function)

Namely, my_decorator is a function that takes a function as an argument and returns a new function. This allows you to wrap your functions with added functionality, and abstract that extra functionality away and reuse it elsewhere.

use this decorator when debugging:

def print_args(func): def wrapper(*args, **kwargs):- print "Calling", func.__name__ print "args:", args print "kwargs" ".join(["%s=%s" % (key, value) for (key, value) in a.iteritems()]) return func(*args, **kwargs) return wrapper

Adding @print_args before the declaration of any function results in the name of the function and its arguments being printed before its invocation. Other things to implement with decorators include: timeouts, acquiring/releasing resources (see also contextmanagers),

http://stackoverflow.com/questions/739654/how-can-i-make-a-chain-of-function-decorators-in-python#1594484

http://habrahabr.ru/post/187482/

http://www.quora.com/Python-programming-language-1/What-are-common-uses-of-Python-decorators

http://cuppster.com/2013/01/30/decorators-scrapers-and-generators/

http://blaag.haard.se/Python-Closures-and-Decorators--Pt--2/

http://www.reddit.com/r/Python/comments/cp3zc/common_uses_of_python_decorators/

http://toumorokoshi.github.io/dry-principles-through-python-decorators.html

http://stackoverflow.com/questions/739654/understanding-python-decorators#1594484

@foo

def bar:

...

is just syntactic sugar for

def bar:

...

bar = foo(bar)

More complex example:

@foo('baz')

def bar:

is just syntactic sugar for

bar = foo('baz')(bar)

https://www.youtube.com/watch?v=f_6vDi7ywuA PYTHON 3

http://www.jeffknupp.com/ blog

http://docs.python.org/reference/datamodel.html

http://www.jeffknupp.com/blog/2013/02/14/drastically-improve-your-python-understanding-pythons-execution-model/

http://www.stevenloria.com/python-best-practice-patterns-by-vladimir-keleshev-notes/

http://www.giantflyingsaucer.com/blog/?p=5117#more-5117 observe patern

http://habrahabr.ru/company/buruki/blog/189972/ Python Internals

http://habrahabr.ru/post/206420/ python internals

http://habrahabr.ru/company/buruki/blog/191032/

http://freepythontips.wordpress.com/2013/09/29/the-python-yield-keyword-explained/

http://www.reddit.com/r/Python/comments/1tyzw3/parallelism_in_one_line/

TESTING

http://pythontesting.net/

http://www.drdobbs.com/testing/unit-testing-with-python/240165163

http://lists.idyll.org/pipermail/testing-in-python/

http://www.jeffknupp.com/blog/2013/12/09/improve-your-python-understanding-unit-testing/

http://dustinrcollins.com/post/62564949954/testing-python-command-line-apps

http://blog.safaribooksonline.com/2013/12/05/flask-with-mock-and-nose/

http://packages.python.org/Attest/

В python 2.7 в unittest добавили много новых интересных вещей:

- дополнительные проверки (assert*),

- декораторы, позволяющие пропустить отдельный тест (@skip, @skipIf) или

обозначить сломанные тесты, о которых разработчику известно (@expectedFailure)

http://stackoverflow.com/questions/19144235/python-unitest-use-variables-defined-in-module-and-class-level-setup-functions

http://stackoverflow.com/questions/4891671/how-do-i-use-the-unittest-setupclass-method

http://pythontesting.net/framework/unittest/unittest-fixtures/

http://gahcep.github.io/blog/2013/02/10/qa-in-python-unittest/

import unittestclass Test(unittest.TestCase): @classmethod def setUpClass(cls): cls._connection = createExpensiveConnectionObject() @classmethod def tearDownClass(cls): cls._connection.destroy()

def setUpModule(): createConnection()def tearDownModule(): closeConnection()

nose автоматически собирает тесты из файлов начинающихся на test_,

он достаточно умен чтобы заглянуть в папочку tests, если такая присутствует,

может показать покрытие кода тестами (модуль coverage).

http://www.reddit.com/r/Python/comments/1sh8i6/can_someone_explain_the_relationship_between/

Имеет удобный режим, когда запускаются только не прошедшие в прошлый прогон тесты

How to use assert:

https://mail.python.org/pipermail/python-list/2013-November/660401.html

Python Gems

http://www.stevenloria.com/lazy-evaluated-properties-in-python/

http://habrahabr.ru/post/196382/

http://www.reddit.com/r/Python/comments/1knw7z/python_interview_questions/

https://github.com/faif/python-patterns

http://devcharm.com/pages/11-python-modules-you-should-know

http://pypix.com/tools-and-tips/python-functions/

Python resources

https://github.com/kirang89/pycrumbs/blob/master/pycrumbs.md

http://freepythontips.wordpress.com/2013/09/01/best-python-resources/

Best Python code:

http://docs.python-guide.org/en/latest/writing/reading/

http://www.reddit.com/r/Python/comments/1ls7vq/best_written_projects_on_python_github/

https://github.com/rg3/youtube-dl

https://github.com/timvieira/arsenal

Python Extensions (C/C++/Java)

http://starship.python.net/crew/arcege/extwriting/pyext.html

http://www.tutorialspoint.com/python/python_further_extensions.htm

http://pythonconquerstheuniverse.wordpress.com/2012/02/15/mutable-default-arguments/

def foobar(arg_string = "abc", arg_list = []):

whenever foobar is called without arguments, arg_string will be bound to the default string object, and arg_list will be bound to the default list object. In such a case,arg_string will always be “abc”, but arg_list may or may not be an empty list. There is a crucial difference between a string object and a list object. A string object is immutable, whereas a list object is mutable. That means that the default for arg_string can never be changed, but the default for arg_list can be changed: list — a mutable object. On each pass, we append a member to the list, and the list grows.

solution is straightforward. The mutable objects used as defaults are replaced by None, and then the arguments are tested for None.

def foobar(arg_string="abc", arg_list = None):

if arg_list is None: arg_list = []

...

Reading files in Python:

http://docs.python.org/2/library/fileinput.html

http://docs.python.org/2/library/mmap.html

Reading file from stdin:

import sys for lineno, line in enumerate(sys.stdin, 1): print('{:>6} {}'.format(lineno, line[:-1]))

# a simple filter that prepends line numbers

import sys

for fname in ( 'file.txt', 'file2.txt, ):

with open(fname, 'r+') as f:

lineno = 0

# this reads in one line at a time from stdin

for line in f:

lineno += 1

print '{:>6} {}'.format(lineno, line[:-1])

#-------------------------------------------------------------------------------

"A simple filter that prepends line numbers" # <-- Docstring

import sys

for fname in sys.argv[1:]: # ./program.py file1.txt file2.txt ...

with open(fname) as f:

# This reads in one line at a time from stdin

for lineno, line in enumerate(f, 1): # Start at 1

print '{:>6} {}'.format(lineno, line[:-1])

https://news.ycombinator.com/item?id=5998750

http://www.reddit.com/r/Python/comments/1e8xw5/common_misconceptions_in_python/

http://blog.moertel.com/posts/2013-06-03-recursion-to-iteration-3.html

http://www.quora.com/Python-programming-language-1/What-are-some-cool-Python-tricks

MULTIPROCESSING

http://rnovitsky.blogspot.co.il/2012/10/python-event-listener-multiprocessing.html

http://www.giantflyingsaucer.com/blog/?p=5008 multiprocessing

http://www.blog.pythonlibrary.org/2016/08/02/python-201-a-multiprocessing-tutorial/

http://adambard.com/blog/Reducers-explained-through-Python/

http://www.spkrbar.com/talk/11 SuperAdvancedPython

http://jmduke.net/post/42448812417/fun-with-itertools

http://scipy-lectures.github.com/advanced/advanced_python/index.html

http://stackoverflow.com/questions/231767/the-python-yield-keyword-explained

http://stackoverflow.com/questions/101268/hidden-features-of-python/113198#113198

http://stackoverflow.com/questions/2573135/python-progression-path-from-apprentice-to-guru

http://ivory.idyll.org/articles/advanced-swc/

http://www.youtube.com/watch?v=OSGv2VnC0go&feature=youtu.be

http://kachayev.github.com/talks/uapycon2012/index.html#/

http://stackoverflow.com/questions/3299648/python-compilation-interpretation-process

VIDEO

http://pyvideo.org/video/1780/transforming-code-into-beautiful-idiomatic-pytho

http://pyvideo.org/category/33/pycon-us-2013

https://github.com/s16h/py-must-watch

http://vimeo.com/pydata/videos

https://speakerdeck.com/pyconslides

http://pyvideo.org/video/2268/atypes-btypes-and-ctypes CTYPES

High Performance Python

http://www.youtube.com/watch?v=Iw9-GckD-gQ

http://www.youtube.com/watch?v=xHqlzuPq_qQ

http://lanyrd.com/2013/pycon/coverage/

>>> def deriv(f): ... dx = 0.0001 ... def fp(x): ... return (f(x + dx) - f(x)) / dx ... return fp ... >>> cube = lambda x: x**3 >>> deriv(cube)(2.0) 12.000600010022566

http://stackoverflow.com/questions/100003/what-is-a-metaclass-in-python/6581949#6581949

http://www.jeffknupp.com/blog/2013/01/24/the-writing-idiomatic-python-ebook-is-finally-available/

http://www.reddit.com/r/programming/comments/145m8o/improving_your_python_productivity/

http://www.reddit.com/r/Python/comments/17auys/grow_as_a_python_developer_by_reading_other/

[] list

() tuples

{} dict

Tuples are ordered collections of possibly dissimilar types where position of elements has meaning.

Tuples allows multiple return values, especially with pattern matching to do tuple unpacking

Tuples are used as dict keys; tuples are immutable and as a result hashable (could be used as keys in dictionary).

Difference between frozendict and namedtuple:

a namedtuple has fields accessible as foo.bar or foo[0], whereas a dict has fields accessible as foo['bar'].

A namedtuple is still a tuple, which means it's an ordered sequence container,

whereas a dict is unordered and does not support access by integer index.

byte_to_bits = dict((chr(i), bin(i)[2:].zfill(8)) for i in range(256))

byte_to_bits["A"]

'01000001'

#equivalent to: for i in ['bar','baz']: results.append(foo(i))

results = map(foo, ['bar',baz'])

results = [foo(i) for i in ['bar', 'baz']]

# returns a list of even numbers

even = filter(lambda x: x % 2 == 0, [1,2,3,4,5,6,7,8,9])

even = [i for i in [1,2,3,4,5,6,7,8,9] if i % 2 == 0]

sum = reduce(lambda x, y: x + y, [1,2,3,4,5,6,7,8,9]) # sums up numbers 1 through 9

json beautifier cat a.json | python -mjson.tool

I usually find comprehensions more readable and lately they work for generating sets and dictionaries as well.

lengths = {i:len(i) for i in ('foo', 'bar', 'foobar')} #dictionary mapping strings to their lengths even_set = {i for i in [1,2,3,4,5,6,7,8,9] if i % 2 == 0}

http://www.reddit.com/r/Python/comments/10t39i/assuming_a_fresh_linux_installation_how_should_i/

http://uberpython.wordpress.com/2012/09/23/why-im-not-leaving-python-for-go/

http://www.reddit.com/r/Python/comments/zq1o5/favorite_tools_in_your_python_toolbox/

http://eli.thegreenplace.net/articles/

http://users.softlab.ntua.gr/~ttsiod/pythonTypeChecking.html

https://github.com/languages/Python/most_watched

http://pythonpackages.com/

http://docs.python-guide.org

http://agiliq.com/blog/2012/06/understanding-args-and-kwargs/

http://www.shutupandship.com/

http://en.wikipedia.org/wiki/List_of_Python_software

http://stackoverflow.com/questions/101268/hidden-features-of-python

http://www-inst.eecs.berkeley.edu/~cs61a/sp12/book/streams.html

http://eli.thegreenplace.net/2012/03/30/python-objects-types-classes-and-instances-a-glossary/

http://late.am/post/2012/03/26/exploring-python-code-objects

http://www.pythonlearn.com/

http://python3porting.com/toc.html

http://nedbatchelder.com/text/iter.html

http://satyajit.ranjeev.in/2012/05/17/python-a-few-things-to-remember.html

http://me.veekun.com/blog/2011/07/22/python-faq/

http://isbullsh.it/2012/05/05-Python-built-in-functions/

http://docs.python.org/release/3.1.5/howto/functional.html

http://www.leancrew.com/all-this/

http://sdiehl.github.com/gevent-tutorial/ gevent

http://www.youtube.com/watch?v=52wxGESwQSA Web Scrapping

YELD

http://stackoverflow.com/questions/231767/the-python-yield-keyword-explained

http://stackoverflow.com/questions/231767/the-python-yield-keyword-explained/231855#231855

http://habrahabr.ru/post/132554/

http://www.jeffknupp.com/blog/2013/04/07/improve-your-python-yield-and-generators-explained/

DataStreams processing

LiveStats solves the problem of generating accurate statistics for when your data set is too large to fit in memory, or too costly to sort. Just add your data to the LiveStats object, and query the methods on it to produce statistical estimates of your data:

https://bitbucket.org/scassidy/livestats

https://news.ycombinator.com/item?id=5554804 http://godoc.org/github.com/bmizerany/perks/quantile (Go)

http://stackoverflow.com/questions/1790550/running-average-in-python

http://docs.python.org/library/collections.html#deque-recipes

#!/usr/bin/env bash

http://docs.python.org/library/

http://docs.python.org/reference/datamodel.html

http://pypi.python.org/pypi/pep8

SSH

http://pypi.python.org/pypi/ssh/1.7.14 ssh from python

https://kushaldas.in/posts/working-over-ssh-in-python.html

http://www.freenetpages.co.uk/hp/alan.gauld/tutos.htm working with OS

map/ filter / reduce can eliminate loops in Python

items = [1, 2, 3, 4, 5]

def visit(item): print item

map(visit, items)

def is_odd(item): return item % 2 == 0

assert [2,4] == filter(is_odd, items)

def add(accumulator, item): return accumulator + item

assert 15 == reduce(add, items)

Named Tuples

http://jmduke.com/blog/2013/12/19/the-best-secret-weapon-for-pythonic-slackers

http://docs.python.org/library/collections.html#collections.namedtuple

user = dict(name="John", age=20) # dict is mutable, slower then tuples

user = ("John", 20) #immutable, but luck of readibility without seeing the context of the tuple packing.

namedtuples are the perfect compromise for the two approaches:

- the have great readability

- lightweightness

- immutability

from collections import namedtuple

Point = namedtuple('Point', 'x y')

p= Point(1.0, 5.0)

p._fields

print p[1] #5.0 access by index, as in regular tuple is not always convenient

print p.y #5.0 access by names: more convenient

http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html

def increment(n):

n = n + 1 # creates new object "n" in new place !!

i = 1

print i # 1

increment(i)

print i #1

-----this will just print 1 twice

def lengthen(n):

n.append(2)

i = [1]

print i # [1]

lengthen(i)

print i # [1, 2]

ints, strs, tuples, are immutable, so they implement +=

by creating a new object and assigning it.

But lists are mutable, so += acts like .extend() and changes the list in-place

args = (1, 2, 3) # usually a tuple, always an iterable*

f(*args) ==> f(1, 2, 3)

kwargs = {"a": 1, "b": 2, "c": 3} # usually a dict, always a mapping*

f(**kwargs) -> f(1, 2, 3) # if you have `def f(a=0, b=0, c=0)`

Iterables are objects that implement the __iter__() method.

Mappings are objects that implement __iter__() and __getitem__().

Any object that supports this protocol will be understood by the constructors tuple() and dict(),

so they can be used for unpacking arguments.

[s for s in strings if any(re.match(f, s) for f in filters)]

Find the first element which satisfies predicate ( len(i) > 2)

>>> lista = ['a', 'b', 'foo', 'c', 'd', 'e', 'bar']

>>> next (i for i in lista if len(i) > 2)

0: 'foo'

Printing all env variables from script

for name, value in sorted(os.environ.items()):

print "%s\t= %s <br/>" % (name, value)

http://mirnazim.org/writings/python-ecosystem-introduction/

http://code.google.com/edu/languages/google-python-class/

http://docs.python-guide.org/en/latest/index.html

http://habrahabr.ru/blogs/hr/137733/#habracut

http://nullege.com/

http://rgruet.free.fr/PQR26/PQR2.6.html Python Quick Reference

http://www.markus-gattol.name/ws/python.html Python on 1 page

http://code.google.com/edu/languages/google-python-class/introduction.html

http://habrahabr.ru/blogs/python/137415/

Python underscore (magic) methods

http://blog.lerner.co.il/making-init-methods-magical-with-autoinit/

https://news.ycombinator.com/item?id=7004864

http://habrahabr.ru/post/186608/

http://www.rafekettler.com/magicmethods.html

http://docs.python.org/reference/datamodel.html#special-method-names

http://pythonconquerstheuniverse.wordpress.com/2012/03/09/pythons-magic-methods/

http://www.siafoo.net/article/57

http://jessenoller.com/good-to-great-python-reads/

http://ivory.idyll.org/articles/advanced-swc/

http://www.reddit.com/r/Python/comments/nnsej/heres_a_weird_question_what_are_some_cool_python/

http://ozkatz.github.com/improving-your-python-productivity.html

http://pytools.codeplex.com/releases/view/76091 Visual Studio Plugin

Formatting

http://docs.python.org/tutorial/inputoutput.html

Sorting

http://habrahabr.ru/blogs/python/138535/

http://habrahabr.ru/blogs/python/138625/

from operator import itemgetter

my_sorted = sorted(my_object, key = itemgetter(calday))

Filtering

filtered_file_object = filter(lambda row: (row[bpartner] in bpartner_master_filter) and (row[ucpremise] in ucpremise_master_filter), my_file_object)

filtered_object = filter(lambda row: (cons_date_from <= row[calday] <= cons_date_to) and \

(row[calday].weekday() in cons_days_of_week) and \

(row[bpartner] in bp_list_by_frequent_pattern_filter), my_file_object )

Logging :

import logging logging.basicConfig(level=logging.INFO, filename='/temp/myapp.log', filemode='w') logger = logging.getLogger("program.stuffPart") try: do_stuff() except: logger.exception("Stuff didn't do")

-----------------------

try: do_something() except: logger.debug('Something something', exc_info=True)

http://eric.themoritzfamily.com/learning-python-logging.html

http://victorlin.me/2012/08/good-logging-practice-in-python/

http://victorlin.me/posts/2012/08/good-logging-practice-in-python/

http://www.youtube.com/watch?v=tJqA6FgqJXs&feature=youtube_gdata_player

http://pingbacks.wordpress.com/2010/12/21/python-logging-tutorial/

http://www.5dollarwhitebox.org/drupal/node/56

http://docs.python.org/library/logging.html

http://docs.python.org/howto/logging-cookbook.html

http://mike.pirnat.com/static/joy-of-logging.pdf

http://habrahabr.ru/blogs/python/135408/

Logging + Multiprocessing

http://stackoverflow.com/questions/641420/how-should-i-log-while-using-multiprocessing-in-python

http://stackoverflow.com/questions/1501651/log-output-of-multiprocessing-process

http://plumberjack.blogspot.com/2010/09/using-logging-with-multiprocessing.html

#!/usr/bin/python import logging import time # basic setup with ISO 8601 time format logging.basicConfig(filename='/tmp/test.log', format='%(asctime)sZ pid:%(process)s module:%(module)s %(message)s', level=logging.INFO) # switch to UTC / GMT logging.Formatter.converter = time.gmtime logging.info('blah blah') logging.debug('you won\'t see this') # change the logging levellogging.getLogger().setLevel(logging.DEBUG) logging.debug('you should see this')

---- output: cat /tmp/test.log

2012-02-22 21:45:51,066Z pid:7328 module:log blah blah

2012-02-22 21:45:51,066Z pid:7328 module:log you should see this

this code has an issue if logging message > 4k

#!/usr/bin/env python

import multiprocessing, logging

logging.basicConfig(format='%(message)s', filename='log.txt')

logging.root.setLevel(logging.INFO)

from multiprocessing import Pool

def f(x):

logging.info(str(x))

if __name__ == '__main__':

logging.info('info from main process')

p = multiprocessing.Pool(5)

p.map(f, [1,2,3,4,5])

Subprocess

http://pythonadventures.wordpress.com/2014/01/08/capture-the-exit-code-the-stdout-and-the-stderr-of-an-external-command/

http://zamotivator.livejournal.com/608524.html

http://stackoverflow.com/questions/89228/how-to-call-external-command-in-python

http://sharats.me/the-ever-useful-and-neat-subprocess-module.html

http://stackoverflow.com/questions/636561/how-can-i-run-an-external-command-asynchronously-from-python

http://jimmyg.org/blog/2009/working-with-python-subprocess.html

http://www.5dollarwhitebox.org/drupal/node/115

https://nedbatchelder.com/text/shell-maybe.html

import subprocess

subprocess.call(['ping', 'localhost'])

subprocess.call(['dir', '/?']) # it does not work because suprocess does not use OS shell

subprocess.call(['cmd', 'dir /?']) # it works !!! because suprocess does not use OS shell

r = subprocess.Popen('dir', shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

for line in r.stdout.readlines():

print line,

retval = r.wait()

prefer to use subprocess.check_call(["./cloudy", input_filename])or subprocess.check_output(["./cloudy", input_filename]). They do error checking on the return code, handle things like spaces or other special characters in filenames, and check_output() provides an easy way to read what the program prints to the screen in python

Non-blocking read

http://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python

Async subprocess

https://stackoverflow.com/questions/21936597/blocking-and-non-blocking-subprocess-calls

http://stackoverflow.com/questions/7581951/python-subprocess-popen-and-asynchronous-output

http://stackoverflow.com/questions/636561/how-can-i-run-an-external-command-asynchronously-from-python

http://stackoverflow.com/questions/546017/how-do-i-run-another-script-in-python-without-waiting-for-it-to-finish

p = subprocess.Popen([sys.executable, '/path/to/script.py'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

Async subprocess on Windows

http://stackoverflow.com/questions/912830/using-subprocess-to-run-python-script-on-windows

process = subprocess.Popen("python /the/script.py")

DETACHED_PROCESS = 0x00000008

pid = subprocess.Popen(["python", "t.py"], creationflags=DETACHED_PROCESS).pid

Multiprocessing

https://news.ycombinator.com/item?id=7690885

http://habrahabr.ru/post/149420/

https://pypi.python.org/pypi/Pebble/

http://habrahabr.ru/post/260431/

http://www.quantstart.com/articles/Parallelising-Python-with-Threading-and-Multiprocessing

http://habrahabr.ru/post/149420/

https://medium.com/p/40e9b2b36148

http://zulko.github.io/blog/2013/09/19/a-basic-example-of-threads-synchronization-in-python/

http://eventlet.net/doc/

http://blip.tv/pycon-us-videos-2009-2010-2011/introduction-to-multiprocessing-in-python-1957019

http://jeetworks.org/node/81

https://github.com/taiste/pyconfi-workshop

http://habrahabr.ru/post/141181/#habracut Affynity

How to share some data between processes?

http://blog.doughellmann.com/2009/04/implementing-mapreduce-with.html

http://mikecvet.wordpress.com/2010/07/02/parallel-mapreduce-in-python/

https://medium.com/@shashwat_ds/a-tiny-multi-threaded-job-queue-in-30-lines-of-python-a344c3f3f7f0#.s3svugwtw

Queue.Queue is made for data interchange between different threads inside the same process (using the threading module).

The multiprocessing queues are for data interchange between different Python processes.

While the API looks similar the underlying mechanisms are fundamentally different.

multiprocessing.Queue exchange data by pickling (serializing) objects and sending them through pipes.

Queue.Queue uses a data structure that is shared between threads and locks/mutexes for correct behavior.

http://eli.thegreenplace.net/2012/01/04/shared-counter-with-pythons-multiprocessing/

import time

from multiprocessing import Process, Value, Lock

def func(val, lock):

for i in range(50):

time.sleep(0.01)

with lock:

val.value += 1

if __name__ == '__main__':

v = Value('i', 0)

lock = Lock()

procs = [Process(target=func, args=(v, lock)) for i in range(10)]

for p in procs: p.start()

for p in procs: p.join()

print v.value

---------Example: Returning results from pool----------------------

import os

from multiprocessing import Pool, cpu_count

from collections import defaultdict

from pprint import pprint as pp

def f(i):

return os.getpid(), i*i

if __name__ == '__main__':

cpus = cpu_count()

print "You have {} cores".format(cpus)

pool = Pool(processes=cpus)

result = pool.map(f, range(20))

data = defaultdict(list)

for pid, i in result:

data[pid].append(i)

pp(dict(data))

http://stackoverflow.com/questions/2080660/python-multiprocessing

http://docs.python.org/library/multiprocessing.html#sharing-state-between-processes

http://stackoverflow.com/questions/1675766/how-to-combine-pool-map-with-array-shared-memory-in-python-multiprocessing

http://stackoverflow.com/questions/659865/python-multiprocessing-sharing-a-large-read-only-object-between-processes

http://slott-softwarearchitect.blogspot.com/2012/02/multiprocessing-goodness-part-1-use.html

http://slott-softwarearchitect.blogspot.com/2012/02/multiprocessing-goodness-part-2-class.html

http://www.ibm.com/developerworks/aix/library/au-multiprocessing/

http://eric.lubow.org/2009/python/python-multiprocessing-pools-and-mysql/

http://blog.perplexedlabs.com/2010/03/04/python-data-sharing-in-the-multiprocessing-module/

https://github.com/ask/billiard/blob/master/README

http://stackoverflow.com/questions/3447846/sharing-data-between-processes-in-python

http://broadcast.oreilly.com/2009/04/pymotw-multiprocessing-part-2.html

http://python.6.n6.nabble.com/Multiprocessing-Pool-and-functions-with-many-arguments-td1562058.html

http://stackoverflow.com/questions/8533318/python-multiprocessing-pool-when-to-use-apply-apply-async-or-map

http://stackoverflow.com/questions/1239035/asynchronous-method-call-in-python

http://stackoverflow.com/questions/6075451/how-do-i-run-cleanup-code-in-a-python-multiprocessing-pool

http://www.frozentux.net/2010/05/python-multiprocessing/

http://grahamstratton.org/straightornamental/entries/multiprocessing

http://www.doughellmann.com/PyMOTW/multiprocessing/communication.html

http://pyinsci.blogspot.com/2009/02/usage-pattern-for-multiprocessing.html

http://www.laurentluce.com/posts/python-threads-synchronization-locks-rlocks-semaphores-conditions-events-and-queues/

https://github.com/laurentluce/python-tutorials/tree/master/threads

http://eli.thegreenplace.net/2012/01/24/distributed-computing-in-python-with-multiprocessing/

http://eli.thegreenplace.net/2012/01/16/python-parallelizing-cpu-bound-tasks-with-multiprocessing/

http://b.atcg.us/blog/2009/12/19/python-multiprocessingjoinablequeue-example.html

import multiprocessing as mul

from time import time

def f(x):

for i in range(10):

x = x * x

def f2(a, b, c):

return a + b * c

if __name__ == '__main__':

print "--- testing multiprocessing on ",mul.cpu_count(),"cores--"

print ""

elements = 100000

pool = mul.Pool(processes=mul.cpu_count())

t1 = time()

res_par = pool.map(f, range(elements))

t2 = time()

res_seq = map(f, range(elements))

t3 = time()

res_app = [pool.apply_async(f,(x,)) for x in range(elements)]

res_app = [result.get() for result in res_app]

t4 = time()

#--------------------

result = pool.apply(f2, (2, 3, 4)) # evaluate "f2(2, 3, 4)"

#--------------------

args = [ (2, 3, 4), # arguments for call 1

(5, 6, 7) # arguments for call 2

]

print [pool.apply(f2, a) for a in args]

#------------------------

results = [pool.apply_async(f, a) for a in args]

print [r.get() for r in results]

#------------------------

print len(res_seq),"elements","map() time",(t3-t2),"s"

print len(res_par),"elements","pool.map() time",(t2-t1),"s"

print len(res_app),"elements","pool.apply_async() time", (t4-t3),"s"

raw_input("press enter to exit...")

/-----------------------------------------------------------------------------------------

Notice also that you could call a number of different functions with Pool.apply_async (not all calls need to use the same function).

import multiprocessing as mp

import time

def foo_pool(x):

time.sleep(2)

return x*x

result_list = []

def log_result(result):

# This is called whenever foo_pool(i) returns a result.

# result_list is modified only by the main process, not the pool workers.

result_list.append(result)

def apply_async_with_callback():

pool = mp.Pool()

for i in range(10):

pool.apply_async(foo_pool, args = (i, ), callback = log_result)

pool.close()

pool.join()

print(result_list)

if __name__ == '__main__':

apply_async_with_callback()

each worker process will call initializer(*initargs) when it starts

def init(args):

p = Pool(initializer = init, initargs = (counter, ))

p.apply_async(async_func, [sql_command, arg1, arg2, arg3, ], callback_func)

def callback_func(self, data):

self.write(data)

def async_func(sql_command, arg1, arg2, arg3)

/----------------------------------------------------------------------------------------------

import multiprocessing

from time import time

#--------------

def do_calculation(data):

return data * 2

#----------------

def start_process():

print 'Starting', multiprocessing.current_process().name

#--------------------

def pool_map():

inputs = list(range(10))

print 'Input :', inputs

builtin_outputs = map(do_calculation, inputs)

print 'Built-in:', builtin_outputs

pool_size = multiprocessing.cpu_count() * 2

print "pool_size =", pool_size

pool = multiprocessing.Pool(processes=pool_size,

initializer=start_process,

)

pool_outputs = pool.map(do_calculation, inputs)

pool.close() # no more tasks

pool.join() # wrap up current tasks

print 'pool_map :', pool_outputs

#--------------------------------

K = 50

def CostlyFunction((z,)):

r = 0

for k in xrange(1, K+2):

r += z ** (1 / k**1.5)

return r

#---------------------------------

def pool_map_async():

currtime = time()

N = 10

po = multiprocessing.Pool()

res = po.map_async(CostlyFunction,((i,) for i in xrange(N)))

w = sum(res.get())

print "pool_map_async=",w

print 'pool_map_async: time elapsed:', time() - currtime

if __name__ == "__main__":

pool_map()

pool_map_async()

Pickling and multiprocessing

http://grahamstratton.org/straightornamental/entries/multiprocessing

http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-pythons-multiprocessing-pool-constructor

http://www.rueckstiess.net/research/snippets/show/ca1d7d90

PYTHONPATH

http://docs.python.org/whatsnew/2.5.html#pep-328-absolute-and-relative-imports

import sys

from pprint import pprint as pp

pp(sys.path)

http://www.blog.pythonlibrary.org/2010/10/14/python-101-introspection/

http://stackoverflow.com/questions/2573135/python-progression-path-from-apprentice-to-guru

http://blog.dispatched.ch/2011/06/12/how-to-become-a-proficient-python-programmer/

http://jessenoller.com/good-to-great-python-reads/

http://jasonmbaker.com/pimp-my-interactive-interpreter

pexpect

http://www.noah.org/wiki/Category:Python pexpect ssh automation Fabric

http://www.noah.org/wiki/pexpect

http://docs.python.org/howto/webservers.html

http://babbledrive.appspot.com/ http://www.gotapi.com/html

http://ivory.idyll.org/articles/advanced-swc/

http://www.dreamincode.net/forums/topic/233445-python-tips-and-tricks-first-edition/

http://code.google.com/edu/languages/google-python-class/

http://habrahabr.ru/blogs/python/114587/ Python object system

http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html

http://www.python.org/community/sigs/current/edu-sig/

http://gael-varoquaux.info/computers/python_advanced/index.html

http://www.cloud-computing-with-python.com/

http://openbookproject.net/py4fun/index.html

http://homepage.mac.com/s_lott/books/oodesign.html

http://www.voidspace.org.uk/python/articles.shtml

http://pleac.sourceforge.net/pleac_python/index.html

www.artima.com/weblogs/viewpost.jsp?thread=299551

http://www.reddit.com/r/Python/comments/dcjeg/what_tips_should_every_new_python_programmer_know/

WITH statement CONTEXT MANAGER

http://preshing.com/20110920/the-python-with-statement-by-example

http://pypix.com/python/context-managers/

http://pyvideo.org/video/883/decorators-and-context-managers

http://jessenoller.com/2009/02/03/get-with-the-program-as-contextmanager-completely-different/

http://www.doughellmann.com/PyMOTW/contextlib/

http://docs.python.org/library/contextlib.html

Timing with context manager

http://coreygoldberg.blogspot.com/2012/06/python-timer-class-context-manager-for.html

http://preshing.com/20110924/timing-your-code-using-pythons-with-statement

http://fossee.in/

@contextlib.contextmanager decorator. It lets you write a context manager that would otherwise look like this:

class withresource(object): def __init__(self, resource): self.resource = resource def __enter__(self): self.resource.acquire() return self.resource def __exit__(self, exc_typ, exc_info, exc_tb): self.resource.release

As a simple function with yield like this:

@contextlib.contextmanager def withresource(resource): # This gets executed when the context-manager is entered resource.acquire() try: # This is the value produced by the 'with' statement yield resource # Execution resumes here when the context manager is exited finally: resource.release()

It smooshes decorators, context-managers and 'yield' together into a single recipe for succinct code reuse; it's beautiful.

main template and skeleton

http://www.jeffknupp.com/blog/2014/02/04/starting-a-python-project-the-right-way/

http://naked-py.com/

http://slott-softwarearchitect.blogspot.com/2011/10/command-line-applications.html

http://cement.readthedocs.org/en/portland/

if __name__ == "__main__":

logging.basicConfig( stream=sys.stderr )

args= parse_args()

logging.getLogger().setLevel( args.verbosity )

try:

for file in args.file:

with open( file, "r" ) as source:

process_file( source, args )

status= 0

except Exception as e:

logging.exception( e )

status= 3

logging.shutdown()

sys.exit( status )

DB API

http://www.reddit.com/r/Python/comments/oovjv/is_there_a_more_efficient_way_to_insert/

con = dbapi.connect(host, port, user, password)

cur = con.cursor()

cur.execute("select count(*) from A");

ret = cur.fetchone()

print "select count(*) from A=", ret[0]

cur.close()

con.close()

conn = sqlite3.connect("mydatabase.db") cursor = conn.cursor() # create a table cursor.execute("""CREATE TABLE albums (title text, artist text, release_date text, publisher text, media_type text) """) # insert some data cursor.execute("INSERT INTO albums VALUES " "('Glow', 'Andy Hunter', '7/24/2012'," "'Xplore Records', 'MP3')") # save data to database conn.commit() # insert multiple records using the more secure "?" method albums = [('Exodus', 'Andy Hunter', '7/9/2002', 'Sparrow Records', 'CD'), ('Until We Have Faces', 'Red', '2/1/2011', 'Essential Records', 'CD'), ('The End is Where We Begin', 'Thousand Foot Krutch', '4/17/2012', 'TFKmusic', 'CD'), ('The Good Life', 'Trip Lee', '4/10/2012', 'Reach Records', 'CD')] cursor.executemany("INSERT INTO albums VALUES (?,?,?,?,?)", albums) conn.commit()

XML Parsing

http://stackoverflow.com/questions/597058/xml-dom-minidom-getting-cdata-values

http://stackoverflow.com/questions/174890/how-to-output-cdata-using-elementtree

<xml>

<a> -aaa-

<m>-mmm-</m>

</a>

<b> -bbb- </b>

<c> -ccc- </c>

</xml>

import os.path

import xml.dom.minidom

def getText(nodelist):

rc = []

for node in nodelist:

if node.nodeType == node.TEXT_NODE:

rc.append(node.data)

return ''.join(rc)

xmlfile="some.xml"

if not os.path.exists(xmlfile):

print "file not exists", xmlfile

exit(1)

file=open(xmlfile,'r')

data=file.read()

file.close()

dom = xml.dom.minidom.parseString(data)

tracks = dom.getElementsByTagName("a")

print tracks[0].firstChild.data

for track in tracks:

print getText(track.getElementsByTagName("m")[0].childNodes)

Videos

http://pyvideo.org/

http://www.youtube.com/user/PythonItalia/feed

http://pyvideo.org/video/618/optimize-performance-and-scalability-with-paralle

http://www.youtube.com/watch?v=b7R3-_ViNxk&feature=related Coroutines, event loops

Advanced Python

http://fulmicoton.com/posts/lazy/ http://fulmicoton.com/posts/fibonacci/

http://www.youtube.com/watch?v=E_kZDvwofHY&feature=related

http://www.youtube.com/watch?v=23s9Wc3aWGY&feature=relmfu

http://www.youtube.com/watch?v=Jpjo0JX797Q&feature=related

http://www.youtube.com/watch?v=Pi9NpxAvYSs&feature=player_embedded Python Epiphanies

http://www.youtube.com/watch?v=tY1fokupHQ4&feature=related

http://www.youtube.com/watch?v=KYuqQ3S7QUw&feature=relmfu

http://python.mirocommunity.org/video/1510/pycon-2010-turtles-all-the-way

http://www.youtube.com/watch?v=i0FCn889ucs&feature=player_embedded Python in big data

http://python.mirocommunity.org/

http://pycon.blip.tv/file/4883162/

http://pyvideo.org/category/17/pycon-us-2012

http://rbw.goe.net/pycon/

http://www.reddit.com/r/Python/comments/hbn0b/intermediateadvanced_python_video_lecture/

http://blip.tv/file/2940638/ Python Data Structures

http://python.mirocommunity.org/category/pyohio-2010

http://europythonvideos.blip.tv/

http://us.pycon.org/2010/conference/schedule/event/12/

http://wwwsearch.sourceforge.net/mechanize/doc.html Parsing web pages

SVG picture of installed packages http://kmkeen.com/pacgraph/

Python IDE

https://pytools.codeplex.com/ VisualStudio Tools

http://www.hanselman.com/blog/OneOfMicrosoftsBestKeptSecretsPythonToolsForVisualStudioPTVS.aspx?repost=true

PyCharm community Edition

http://www.reddit.com/r/Python/comments/11ggke/ide_with_a_editor_on_one_side_and_a_console_on/

http://code.google.com/p/pyscripter/ http://www.geany.org/ http://www.wingware.com/

http://packages.python.org/spyder/

https://github.com/cwsoft/cwsoft-WSPPDE#readme Scientific Python

http://www.pythonxy.com/

http://pytools.codeplex.com/ PythonTools for VisualStudio 2010

http://wiki.python.org/moin/IntegratedDevelopmentEnvironments

http://infoworld.com/d/developer-world/infoworld-review-nine-fine-python-development-tools-374

Python Shells

http://dreampie.sourceforge.net/

http://www.bpython-interpreter.org For windows: install curses from here before invoking "pip bpython" http://www.lfd.uci.edu/~gohlke/pythonlibs/#curses

http://ipython.scipy.org/moin/

http://packages.python.org/Brownie/

REPL in PYTHON

http://bogdanp.github.com/pyrepl.vim/

Tab completion (Linux):

import readline, rlcompleter; readline.parse_and_bind("tab: complete")

(executed at PYTHONSTARTUP) for tab completion.

Put into ~/.pythonrc.py file

Performance

http://wiki.python.org/moin/PythonSpeed/PerformanceTips

http://www.huyng.com/posts/python-performance-analysis/

http://www.leaseweblabs.com/2013/02/python-performance-optimization/

http://habrahabr.ru/post/151163/

Python Packaging

http://habrahabr.ru/post/166463/

http://blog.ionelmc.ro/2014/06/25/python-packaging-pitfalls/

http://vascop.github.com/blog/2012/09/08/simple-guide-to-packaging-python-applications.html

http://foobar.lu/wp/2012/05/13/a-comprehensive-step-through-python-packaging-a-k-a-setup-scripts/

http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/

http://www.ibm.com/developerworks/opensource/library/os-pythonpackaging/index.html

http://docs.python.org/tutorial/modules.html

http://mikegrouchy.com/blog/2012/05/be-pythonic-__init__py.html

http://mikegrouchy.com/blog/be-pythonic-initpy.html

easy_install and pip

add to Windows PATH: C:\Python27\Scripts

http://pypi.python.org/pypi/setuptools/

http://pypi.python.org/pypi/pip

http://www.reddit.com/r/Python/comments/nl4ip/just_a_reminder_to_use_pip_distribute_and/

File system structure of a Python project

http://vascop.github.com/blog/2012/09/08/simple-guide-to-packaging-python-applications.html

http://infinitemonkeycorps.net/docs/pph/

http://jcalderone.livejournal.com/39794.html

Do:

name the directory something related to your project. For example, if your project is named "Twisted", name the top-level directory for its source files Twisted. When you do releases, you should include a version number suffix: Twisted-2.5.
create a directory Twisted/bin and put your executables there, if you have any. Don't give them a .py extension, even if they are Python source files. Don't put any code in them except an import of and call to a main function defined somewhere else in your projects. (Slight wrinkle: since on Windows, the interpreter is selected by the file extension, your Windows users actually do want the .py extension. So, when you package for Windows, you may want to add it. Unfortunately there's no easy distutils trick that I know of to automate this process. Considering that on POSIX the .py extension is a only a wart, whereas on Windows the lack is an actual bug, if your userbase includes Windows users, you may want to opt to just have the .py extension everywhere.)
If your project is expressable as a single Python source file, then put it into the directory and name it something related to your project. For example, Twisted/twisted.py. If you need multiple source files, create a package instead (Twisted/twisted/, with an empty Twisted/twisted/__init__.py) and place your source files in it. For example, Twisted/twisted/internet.py.
put your unit tests in a sub-package of your package (note - this means that the single Python source file option above was a trick - you always need at least one other file for your unit tests). For example, Twisted/twisted/test/. Of course, make it a package with Twisted/twisted/test/__init__.py. Place tests in files like Twisted/twisted/test/test_internet.py.
add Twisted/README and Twisted/setup.py to explain and install your software, respectively, if you're feeling nice.

Don't:

put your source in a directory called src or lib. This makes it hard to run without installing.
put your tests outside of your Python package. This makes it hard to run the tests against an installed version.
create a package that only has a __init__.py and then put all your code into __init__.py. Just make a module instead of a package, it's simpler.
try to come up with magical hacks to make Python able to import your module or package without having the user add the directory containing it to their import path (either via PYTHONPATH or some other mechanism). You will not correctly handle all cases and users will get angry at you when your software doesn't work in their environment.

Get list of all the string methods

dir("foo")

['__add__', '__class__', '__contains__', (snipped a bunch), 'title','translate', 'upper', 'zfill']

information about a particular method you can call "help" on it.

help("foo".upper)

- type
- dir
- help
- sys

distribute fully replaces setuptools.

INSTALLED MODULES AND PACKAGES look under /lib/site-packages

http://pypix.com/python/create-python-library/

List of all modules: help ("modules")

List of installed packages: yolk or pip freeze

http://pypi.python.org/pypi/yolk/

http://marc-abramowitz.com/archives/2011/05/06/python-tool-of-the-day-yolk/

dir function is handy for 3rd party packages:

import sys

dir(sys)

#! /usr/bin/env python

import sys

try:

import pkg_resources

except ImportError:

sys.stderr.write("'pkg_resources' could not be imported: setuptools installation required\n")

sys.exit(1)

def list_package_modules(package_name):

"""

Returns list of module names for package `package_name`.

"""

try:

contents = pkg_resources.resource_listdir(package_name, "")

except ImportError:

return []

module_names = []

for entry in contents:

if pkg_resources.resource_isdir(package_name, entry):

module_names.extend(list_package_modules(package_name + "." + entry))

elif not entry.endswith('.pyc'):

if entry.endswith(".py"):

entry = entry[:-3]

module_names.append(package_name + "." + entry)

return module_names

if __name__ == "__main__":

m = list_package_modules("IPython")

print "\n".join(m)

List comprehension produces a result list, where the elements of the result list are the transformed elements of the iteration

*result* = [*transform* *iteration* *filter* ]

new_range = [i * i for i in range(5) if i % 2 == 0]

print new_range

[0, 4, 16]

Partition

x = "a@y.com"

x.partition('@')[-1]

'y.com'

File has 2 columns: name and value. Calculating % of 2nd column from total of 2nd col

import sys

name=sys.argv[1]

file = open(name)

total=0

while 1:

line = file.readline()

if not line.rstrip():

break

t = line.split()

total += int( t[1] )

file.close()

print "----------------"

print total

print "----------------"

file = open(name)

while 1:

line = file.readline()

if not line.rstrip():

break

t = line.split()

print t[0], t[1], int(t[1])*100/total

file.close()

Decorators

function is an object of the class function,a subclass of callable

http://agiliq.com/blog/2012/11/understanding-decorators-2/

http://www.brianholdefehr.com/decorators-and-functional-python

http://www.artima.com/weblogs/viewpost.jsp?thread=240808

http://www.saltycrane.com/blog/2010/03/simple-python-decorator-examples/

http://simeonfranklin.com/blog/2012/jul/1/python-decorators-in-12-steps/

http://pythonconquerstheuniverse.wordpress.com/2012/04/29/python-decorators/

http://www.thumbtack.com/engineering/a-primer-on-python-decorators/

http://habrahabr.ru/blogs/python/139866/

http://habrahabr.ru/post/141411/

http://blaag.haard.se/Python-Closures-and-Decorators--Pt--2/

http://www.artima.com/weblogs/viewpost.jsp?thread=240808

http://stackoverflow.com/questions/739654/understanding-python-decorators#answer-1594484

http://wiki.python.org/moin/PythonDecoratorLibrary

http://habrahabr.ru/post/157265/

Find out all strings which are matching at least one filter:

group by in python

http://code.activestate.com/recipes/304162-summary-reports-using-itertoolsgroupby/

issue with code below: file should be pre-sorted by key

## {{{ http://code.activestate.com/recipes/304162/ (r2)

from itertools import groupby

from itertools import imap

from operator import itemgetter

import csv

def summary(data, key=itemgetter(0), value=itemgetter(1)):

"""Summarise the supplied data.

Produce a summary of the data, grouped by the given key (default: the

first item), and giving totals of the given value (default: the second item).

The key and value arguments should be functions which, given a data

record, return the relevant value.

"""

for k, group in groupby(data, key):

# yield (k, sum(value(row) for row in group))

yield (k, sum(imap(value, group)))

if __name__ == "__main__":

# Example: given a set of sales data for city within region,

# produce a sales report by region

sales = csv.reader(open("file.txt"))

for region, total in summary(sales, key=itemgetter(0), value=lambda r: int(r[2])):

print "%10s: %d" % (region, total)

http://code.activestate.com/recipes/496726-summary-reports-using-itertoolsgroupby-extended-to/

following code does the same, but without itertools https://gist.github.com/839705

# Sales is a list of tuples

sales = [('Scotland', 'Edinburgh', 20000),

('Scotland', 'Glasgow', 12500),

('Wales', 'Cardiff', 29700),

('Wales', 'Bangor', 12800),

('England', 'London', 90000),

('England', 'Manchester', 45600),

('England', 'Liverpool', 29700)]

# Use a dict to store output

out_sales = {}

# Populate output dict

for region, city, total in sales:

# Set-or-increment that key

try:

out_sales[region] += total

except KeyError:

out_sales[region] = total

# Print the summary

for region in out_sales:

print "%s: %d" % (region, out_sales[region])

Most used words

http://www.reddit.com/r/Python/comments/jqvec/made_my_first_script_it_tells_you_the_most_used/

Parsing hadoop -lsr output to accumulate partitioner size:

import sys

import pprint

name=sys.argv[1]

file = open(name)

part_size={}

while 1:

line = file.readline()

if not line:

break

if line[0] == 'd':

pass

else:

t = line.split("/")

w = t[0].split();

part = t[-1].rstrip()

size = int(w[4])

try:

part_size[part] += size

except KeyError:

part_size[part] = size

file.close()

pprint.pprint(part_size)

#print sorted by size:

for part in sorted(part_size, key=part_size.get, reverse=True):

print part, part_size[part]

Functional Python

http://thecomputersarewinning.com/post/pythons-functional-training-wheels

http://www.electricmonk.nl/log/2011/05/20/closures-and-when-theyre-useful/

http://blog.dhananjaynene.com/2010/02/functional-programming-with-python-part-1/

Calling the executable from Python and consuming the output

import subprocess

p = subprocess.Popen(['some_executable', 'argument'], shell=True, stdout=subprocess.PIPE)

std_out, std_err = p.communicate()

print std_out

Count the number of occurrences each character in the string (ignore spaces, numbers)

import collections, string

import timeit

# fast version

def count1():

s = "thIs is a string for cOunTing"

result = collections.defaultdict(int)

for c in s.lower(): # only call lower() once

if c in 'abcdefghijklmnopqrstuvwxyz': # this removes one lookup

result[c] += 1

# another version

def count2():

s = "thIs is a string for cOunTing"

d = {}

for c in s.lower():

if c in string.lowercase:

d[c] = d.get(c,0) + 1

generators, coroutines and other advanced stuff

http://excess.org/article/2013/02/itergen1/

http://www.dabeaz.com/generators/

http://www.dabeaz.com/coroutines/index.html

http://stackoverflow.com/questions/101268/hidden-features-of-python#sort-top

http://norvig.com/sudoku.html

http://stackoverflow.com/questions/100003/what-is-a-metaclass-in-python/6581949#6581949

http://tech.blog.aknin.name/category/pythons-innards/

Profiling Python Performance

http://habrahabr.ru/company/mailru/blog/202832/

http://www.tracelytics.com/blog/profiling-python-performance-lineprof-statprof-cprofile/

https://julien.danjou.info/blog/2015/guide-to-python-profiling-cprofile-concrete-case-carbonara

https://nylas.com/blog/performance

https://blogs.dropbox.com/tech/2012/07/plop-low-overhead-profiling-for-python/

https://github.com/what-studio/profiling

STOCK FROM google.finance

#!/usr/bin/env python import json import pprint import urllib2 def get_stock_quote(ticker_symbol): url = 'http://finance.google.com/finance/info?q=%s' % ticker_symbol lines = urllib2.urlopen(url).read().splitlines() return json.loads(''.join([x for x in lines if x not in ('// [', ']')])) if __name__ == '__main__': quote = get_stock_quote('IBM') print 'ticker: %s' % quote['t'] print 'current price: %s' % quote['l_cur'] print 'last trade: %s' % quote['lt'] print 'full quote:' pprint.pprint(quote)

Foreign Function Interface (FFI)

https://news.ycombinator.com/item?id=7230820

http://joao.in/blog/post/multiprocessing-python-via-c/

https://news.ycombinator.com/item?id=10724396

http://eev.ee/blog/2013/09/13/cython-versus-cffi/

ctypes distutils cython

http://www.ibm.com/developerworks/ru/library/l-python_details_06/index.html

http://www.ibm.com/developerworks/ru/library/l-python_details_07/index.html

http://www.ibm.com/developerworks/ru/library/l-python_details_08/

https://github.com/thearn/simple-cython-example

http://maxburstein.com/blog/speeding-up-your-python-code/

http://www.asimihsan.com/presentations/going_faster_with_python/index.html#1

http://eli.thegreenplace.net/2013/03/09/python-ffi-with-ctypes-and-cffi/

http://jayrambhia.wordpress.com/2012/12/19/pythonc-api-making-a-type/

http://habrahabr.ru/post/168083/ Boost.Python

http://habrahabr.ru/post/157537/

http://www.linuxforu.com/2011/03/when-python-weds-c/

http://danfm.ca/posts/python-c-extensions/

http://ianozsvald.com/2012/03/18/high-performance-python-1-from-pycon-2012-slides-video-src/

http://jiaaro.com/python-performance-the-easyish-way

ctypes is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. ctypes function has an overhead that is 2 to 3 times to a similar C extension function; the calling overhead is severe, so if you'll be making a lot of calls into the library, and you're going to be writing the C code anyway (or at least compiling it), then go for cython or SWIG

C from Python

http://karuppuswamy.com/wordpress/2012/01/28/how-to-use-c-library-in-python-generating-python-wrappers-for-c-library/

int multiply(int num1, int num2){

return num1 * num2;

}

gcc -c -fPIC libtest.c

gcc -shared libtest.o -o libtest.so

From the directory containing 'libtest.so' try these commands:

>>> from ctypes import *

>>> import os

>>> libtest = cdll.LoadLibrary(os.getcwd() + '/libtest.so')

>>> print libtest.multiply(2, 7)

For Windows users:: create file dlltest.c

#include <windows.h>

BOOL APIENTRY DllMain(HANDLE hModule, DWORD dwReason, LPVOID lpReserved)

{

return TRUE;

}

__declspec(dllexport) int

multiply(int num1, int num2){

return num1 * num2;

}

compile:

cl -LD dlltest.c -Fetest.dll

>>> from ctypes import *

>>> libtest = cdll.LoadLibrary('test.dll')

>>> print libtest.multiply(2, 2)

More complicated example with pointers:

lib.c

/* sum of a list of doubles */

double sum(double *data, int n){

int i=0;

double sum=0.0;

for (i=0; i<n; i++) {

sum += data[i];

}

return sum;

}

/* mean of a list of doubles */

double mean(double *data, int n)

{

double s = sum(data, n);

return s/((double)n);

}

compile it:

$ gcc -shared -fPIC lib.c -o lib.so

In python we use ctypes:

from ctypes import *

so = CDLL("lib.so")

# Set up interfaces

so.mean.argtypes= [POINTER(c_double), c_int]

so.mean.restype = c_double

so.sum.argtypes = [POINTER(c_double), c_int]

so.sum.restype = c_double

# Call the functions.

def cmean(self, dlist):

doubleArray = c_double*len(dlist) # Data Type (double[n])

cx = doubleArray(*dlist) # Actual array

n = len(cx) # Length of the data

result = so.mean(cx, n)

return float(result)

def csum(self, dlist):

doubleArray = c_double*len(dlist)

cx = doubleArray(*dlist)

n = len(cx)

result = so.sum(cx, n)

return float(result)

# We can now use the functions as if they were pure python!

data = [1,2,3,4,5,6]

print cmean(data)

print csum(data)

For a more portable way of compiling the C source file (which will invoke gcc with the right parameters on Linux/OSX or msvc on Windows) we can use setuptools or distutils. In your setup.py, you would add:

from setuptools import setup, Extension

module1 = Extension('libtest',

sources = ['libtest.c'],

export_symbols = ['multiply'])

[...]

setup(

[...]

ext_modules = [module1],

)

http://www.europython.eu/

http://heather.cs.ucdavis.edu/~matloff/python.html

http://ru.wikiversity.org/wiki/Программирование_и_научные_вычисления_на_языке_Python

http://blip.tv/file/1947373/?bcsicoach AI in Python

http://www.devx.com/opensource/Article/21549/1954

http://tablib.org/tutorial.html

Page updated

Google Sites

Report abuse