[TBD]Useful Python programming info

Python IDEs

Anaconda PyCharm

Please refer https://www.anaconda.com/distribution/#macos

Spyder

Python editors

Jupyter Notebook

Ref https://jupyterlab.readthedocs.io/en/stable/getting_started/installation.html

pip install jupyterlab

Jupyter notebook installation

(neo4jenv) Deepaks-MacBook-Air:docker deepak$ pip3 install jupyterlab

(neo4jenv) Deepaks-MacBook-Air:docker deepak$ pip3 freeze | grep ju

jupyter-client==6.1.2

jupyter-core==4.6.3

jupyterlab==2.0.1

jupyterlab-server==1.0.7

(neo4jenv) Deepaks-MacBook-Air:docker deepak$

(neo4jenv) Deepaks-MacBook-Air:docker deepak$ jupyter notebook --version

6.0.3

(neo4jenv) Deepaks-MacBook-Air:docker deepak$

Start the Jupyter server in the host machine

Start Jupyter notebook and access from browser

(neo4jenv) Deepaks-MacBook-Air:docker deepak$ jupyter lab

[I 11:53:28.175 LabApp] Writing notebook server cookie secret to /Users/deepak/Library/Jupyter/runtime/notebook_cookie_secret

[I 11:53:30.082 LabApp] JupyterLab extension loaded from /Users/deepak/Documents/coding/python/neo4jenv/lib/python3.6/site-packages/jupyterlab

[I 11:53:30.083 LabApp] JupyterLab application directory is /Users/deepak/Documents/coding/python/neo4jenv/share/jupyter/lab

[I 11:53:30.089 LabApp] Serving notebooks from local directory: /Users/deepak/Documents/coding/python/twitter-neo4j/docker

[I 11:53:30.090 LabApp] The Jupyter Notebook is running at:

[I 11:53:30.090 LabApp] http://localhost:8888/?token=3af2738c9bb4327d25e93892f382b94954e02e681d34a0de

[I 11:53:30.090 LabApp] or http://127.0.0.1:8888/?token=3af2738c9bb4327d25e93892f382b94954e02e681d34a0de

[I 11:53:30.090 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

[C 11:53:30.100 LabApp]

To access the notebook, open this file in a browser:

file:///Users/deepak/Library/Jupyter/runtime/nbserver-42826-open.html

Or copy and paste one of these URLs:

http://localhost:8888/?token=3af2738c9bb4327d25e93892f382b94954e02e681d34a0de

or http://127.0.0.1:8888/?token=3af2738c9bb4327d25e93892f382b94954e02e681d34a0de

[I 11:53:34.162 LabApp] Build is up to date

Please refer https://jupyter-notebook.readthedocs.io/en/stable/public_server.html for setting password

Example run in browser

Below example shows run of print("hello world")

Sublime

Atom

Testing tools

• pylint

• pyflakes

• pep8

Modules to write unittest

• unittest

• doctest

Python multithreading vs multi-processing performance points

Python has lock to protect against wrong reference count for the object. For this, it uses GIL. Ref: https://www.youtube.com/watch?v=4zeHStBowEk&feature=youtu.be&t=11m40s

• I/O-intensive processes improved with multithreading. It It happens since CPU is released and so, can be taken up by other threads

  • • webscraping

    • reading and writing to files

    • network communications

• CPU-intensive processes improved with multiprocessing: It happens since each process can take up OS provided cycle without any dependency on voluntarily release

  • • computations

    • text formatting

    • image rescaling

    • data analysis

Parts of Python

• • Objects and data structures

    • Numbers

      • // for floor division (for example 7//4 output is 1)

    • Strings

      • 'It is a "big big" object '

      • " I'm happy "

      • s[1:] -> Grab everything past the first term all the way to the length of s which is len(s)

      • s[:3] -> Grab everything UP TO the 3rd index

      • s[-1] -> Last letter (one index behind 0 so it loops back around)

      • s[:-1] -> Grab everything but the last letter

      • s[::1] -> Grab everything, but go in steps size of 1

      • s[::2] -> Grab everything, but go in step sizes of 2

      • s[::-1] -> Print string backward for s[::1]

      • s.split() -> Split a string by blank space (this is the default)

      • s.split('W') -> Split a string by W as separator

    • Lis

      • List comprehension to generate list from expressions

      • lst.count(2) -> How many times 2 appear in the list

      • my_list * 2 -> It doubles the list

• • • Tuple

      • Tuple is faster than list (https://learnbatta.com/blog/why-tuple-is-faster-than-list-in-python-22/)

      • Its immutable

      • t.count('one') -> Use .count to count the number of times a value appears

    • Boolean,

    • Files

    • Python uses file objects to interact with external files on your computer

    • These file objects can be any sort of file you have on your computer, whether it be an audio file, a text file, emails, Excel documents, etc.

    • my_file.seek(0) -> Seek to the start of file (index 0)

    • my_file.readlines()

    • my_file.close()

    • %%writefile file1.py -> Creates a file from within python code

• • String Formatting

    • print('The {2} {1} {0}'.format('fox','brown','quick'))

      • The quick brown fox

  • print('First Object: {a}, Second Object: {b}, Third Object: {c}'.format(a=1,b='Two',c=12.3))

    • First Object: 1, Second Object: Two, Third Object: 12.3

  • print('A {p} saved is a {p} earned.'.format(p='penny'))

    • A penny saved is a penny earned.

  • Also please refer below example

• • Comparison operators

    • Basic

      • is matches type as well

      • == checks value only

        • 3.0 == 3 -> True

        • 3.0 is 3 -> False

  • Chained comparison operators

    • 1 < 3 > 2 -> True

  • Statements

    • Basic (If-elif-else, for, while , range)

      • list(range(0,11)) -> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

    • list(range(0,101,10)) -> [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100]

  • List comprehension

    • lst = [x**2 for x in range(0,11)] -> [0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

• Method and functions

  • Basic

    • Methods are treated as object

    • Methods can be deleted using del

    • Function can be defined within a function

    • Functions can be return value of a function

    • Function can be passed as argument

  • Lambda

• OOPs

  • Class

  • Inheritence

  • Special methods

    • __str__

• • • __len__

    • __del__

• Error and exception handling

  • try-except-finally

• Modules and Packages

  • Creating module

    • https://python4astronomers.github.io/installation/packages.html

    • https://docs.python.org/3/tutorial/modules.html#packages

  • Installing module

    • Ref: https://docs.python.org/3/tutorial/modules.html#packages

    • Packages are a way of structuring Python’s module namespace by using “dotted module names”.

    • When importing the package, Python searches through the directories on sys.path looking for the package subdirectory.

    • The __init__.py files are required to make Python treat directories containing the file as packages.

    • In the simplest case, __init__.py can just be an empty file, but it can also execute initialization code for the package or set the __all__ variable

      • Contains __all__ for supporting * import (from sound.effects import *)

• • Built-in modules

    • print(dir(math)) -> Displays method in math module

    • help(math.ceil) -> Help on ceil method in math module

• Built-in functions

  • all and any

    • all() and any() are built-in functions in Python that allow us to conveniently check for boolean matching in an iterable.

    • all() will return True if all elements in an iterable are True.

    • any() will return True if any of the elements in the iterable are True.

  • complex

    • Returns complex number

  • Map

    • The map function allows you to "map" a function to an iterable object.

    • That is to say you can quickly call the same function to every item in an iterable, such as a list.

    • map() can accept more than one iterable. The iterables should be the same length - in the event that they are not, map() will stop as soon as the shortest iterable is exhausted.

  • Reduce

    • Ref: https://www.learnpython.org/en/Map,_Filter,_Reduce

• Parameters

  • *args

    • When a function parameter starts with an asterisk, it allows for an arbitrary number of arguments, and the function takes them in as a tuple of values

  • **kwargs

    • Similarly, Python offers a way to handle arbitrary numbers of keyworded arguments

• Decorators

  • It is wrapper function which is called before the actual function

  • Decorator function has the responsibility to invoke actual function

    • If decorator doesn't invoke function, then only decorate code is executed

• Python generator and iterator

  • Generators allow us to generate as we go along, instead of holding everything in memory.

  • They are called in your code they don't actually return a value and then exit. Instead, generator functions will automatically suspend and resume their execution and state around the last point of value generation.

  • Generators are iterators

  • Generators do not store all the values in memory, they generate the values on the fly:

  • It has next method to get next yielded value [print(next(g))]

  • How generator works?

    • With regular functions, at some point the function returns a value, and the stack is "popped". The function's stack frame is discarded and execution resumes at the previous location.

    • When a function is a generator, it can return a value without the stack frame being discarded, using the yield statement. The values of local variables and the program counter within the function are preserved. This allows the generator to be resumed at a later time, with execution continuing from the yield statement, and it can execute more code and return another value.

    • yield maintains the state between function calls, and resumes from where it left off when we call the next() method again. Ref: https://stackabuse.com/python-generators/

    • The key advantage to generators is that the "state" of the function is preserved, unlike with regular functions where each time the stack frame is discarded, you lose all that "state". A secondary advantage is that some of the function call overhead (creating and deleting stack frames) is avoided, though this is a usually a minor advantage.

    • Similar to list comprehension, there is generator comprehension. generator comprehension yields value one by one (below example)

Useful link: https://docs.python.org/3/distutils/setupscript.html

https://stackoverflow.com/questions/4740473/setup-py-examples

https://docs.python.org/3/install/index.html

https://stackoverflow.com/questions/1471994/what-is-setup-py

Approach to start python module

Use -m option in python

The module-name should be a valid Python module name. Package names are also permitted. When a package name is supplied instead of a normal module, the interpreter will execute <pkg>.__main__ as the main module.

Useful link:https://docs.python.org/2/using/cmdline.html

Approach to list down installed modules

use python CLI and then fire help command

Approach to list modules via python console

bash-2.05b# python

Python 2.6.6 (r266:84292, Sep 1 2017, 06:48:46)

[GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd8

Type "help", "copyright", "credits" or "license" for more information.

>>> help("modules")

Please wait a moment while I gather a list of all available modules...

BaseHTTPServer clusdb493c5921fbd336f06073198afe5b17 lsng3646ac7dddcc30fdaa2743191d71256c sslp57c80f3026fc481b65cd267a14758d0f

Bastion clusdb7c55c4917023741c0ec3999677a9fe lsng52aeac3aa882529e09d883333ee0c4e5 sslp65fd0f76c641204398b03310fa818e1a

CGIHTTPServer clusddb9ce6eb277801189efd9599bbe228c lsng5fd091435a7d680c9d3054ea8fc65f6e sslp6af96d5eff7cbe791023f24beaeb7839

Canvas clusddc32ef4a81dc221d981a55c3d4730ac lsng64d2b4e9a0ceaa31a07937ba248deab7 sslp702545d9375016bf7fb69d74f4f29bc1

ConfigParser clusdfd92ee266f2cb64a3878977e9505762 lsng6610e243ce9bae5502b6a63ae044b9fc sslp738854c1c0a7cac04c9dc8c0cad13217

Cookie cluse051fa011e0fecada36f2cba96f9f263 lsng6d6504f1649487807fffeaae8107e0d6 sslp74a72a2c937a7577d2fa78ffed8898d9

Crypto cluse83a344dec855b4e1c92522164986de9 lsng7d8c50ac6f6e917f75abfeee02f4bfdb sslp76a1ada7dbdec410f2c2c979c43a8261

Dialog clusea88f8b23b6d6ab5e0399a9907c22f43 lsng81cb586c335c5206bee4646efb99be57 sslp8140cf7f442fce06598466556bda7ab0

DocXMLRPCServer clusef31a5dbf7ebe4375265838cc3ff03fa lsng98794091ec62e38bf83451f484fc6e72 sslp8522f22e7eb4bb911a39a1c3716310ca

FileDialog clusfd82a6aa1e1b84a0153717268897d191 lsng9955ede883f4119c0869dfadd35c613c sslp8548912b7d7bc49b3204f764730e605f

FixTk cmath lsnga180347062ab945c3e8b67541c74f28a sslp8824fac25ac120e62463403dc9733a82

HTMLParser cmd lsnga5959d44d493e6a527a96ae0ae3a730b sslp93c3256adea55b79db8ea97b48c4c4e5

IN cmpa49473be3fcc67379cbf88df8210d865d lsngbcc907bbb87ebb78f9f5d179fa1b293e sslpa569dce97d640557abd999af7a56efff

MimeWriter cmpa56d955d395b9a4508d5211fd568c66bd lsngbff8d6f8669821c1e3b824f120131f16 sslpa75b94d2cad34a0aac3abeefd213d941

QueryProcessor cmpa83425d9c47fede8f04152df6b5875d43 lsngc0e851d47339fc84a03057c0547bc309 sslpad5208903ed16db415882b027e25a77f

Queue cmpa8899212ecf644bbe92c9dfbc079c3ca5 lsngc2024d3d46f252ac6bd8458c545afa93 sslpad549a485931bc26a8de773e550b41ac

....

....

Useful link: http://blog.revathskumar.com/2011/10/python-list-all-packages-installed.html

Approach to locate python module

Use sys.dir and then search manually

python -v >>> import sys >>> sys.path ['', '/usr/local/lib/python25.zip', '/usr/local/lib/python2.5', ... ]

Useful link: https://stackoverflow.com/questions/269795/how-do-i-find-the-location-of-python-module-sources

Code to connect ssh server

Please refer https://github.com/krdpk17/python/blob/master/myssh.py

Useful link: http://linux.byexamples.com/archives/346/python-how-to-access-ssh-with-pexpect/

How to perform SNMP query

Please refer https://github.com/krdpk17/python/blob/master/testsnmp.py

Useful link: https://stackoverflow.com/questions/8601324/how-to-get-data-from-snmp-with-python

Approach to pass variable length argument/parameter

Use *args and **kwargs

Variable number of arguments and parameters

root@6b774e439f98:/# cat temp2.py

def test_var_args(f_arg, *argv):

print "first normal arg:", f_arg

for arg in argv:

print "another arg through *argv :", arg

test_var_args('yasoob','python','eggs','test')

root@6b774e439f98:/# python temp2.py

first normal arg: yasoob

another arg through *argv : python

another arg through *argv : eggs

another arg through *argv : test

Useful link: https://pythontips.com/2013/08/04/args-and-kwargs-in-python-explained/

Approach for embedding python code in bash script

Use python - option

Embed python code in bash script

root@cc472a7a3f34:/# cat test.sh

python - <<END

import socket

with open("/etc/hosts", 'r') as hosts:

with open("/etc/hosts_new", "w") as hosts_new:

for line in hosts:

addr=line.split()[0:]

try:

socket.inet_aton(addr[0])

hosts_new.write(line)

except socket.error:

print 'ignoring ' +str(addr)

continue

END

$(mv /etc/hosts_new /etc/hosts_1)

root@cc472a7a3f34:/# cat /etc/hosts_1

127.0.0.1 localhost

172.17.0.2 cc472a7a3f34

root@cc472a7a3f34:/#

Useful link: https://stackoverflow.com/questions/2189098/embedding-short-python-scripts-inside-a-bash-script

Approach to read a file data

Refer http://www.pythonforbeginners.com/files/reading-and-writing-files-in-python

Sample program

Sample program for creating web-server

code

root@ubuntu:~# cat web-server.py

import tornado.ioloop

import tornado.web

import socket

class MainHandler(tornado.web.RequestHandler):

def get(self):

self.write("Hostname: " + socket.gethostname())

def make_app():

return tornado.web.Application([

(r"/", MainHandler),

])

if __name__ == "__main__":

app = make_app()

app.listen(8888)

tornado.ioloop.IO

sample output

google180697_student@k8s-workshop-module-1-lab:/kickstart$ curl -k http://localhost:8888Hostname: k8s-workshop-module-1-labgoogle180697_student@k8s-workshop-module-1-lab:/kickstart$

Approach to make your module executable by python

Use __name__ check

Reference: https://stackoverflow.com/questions/419163/what-does-if-name-main-do

Troubleshooting

Method to display backtrace

Below example code will print the traceback (Ref: below example output).

Useful link: https://stackoverflow.com/questions/1278705/python-when-i-catch-an-exception-how-do-i-get-the-type-file-and-line-number

Difference between is and == operator

Refer https://dbader.org/blog/difference-between-is-and-equals-in-python

Useful link: https://dbader.org/blog/difference-between-is-and-equals-in-python

Reference

https://github.com/Pierian-Data/Complete-Python-3-Bootcamp

https://docs.python.org/3/library/random.html