LINKS

• https://www.programiz.com/python-programming/tutorial

Importing and Exporting

Opening, Writing, Closing File

• f = open("#C:/test.txt", r #read or w #write or a #append) #open file

• with open("#test.txt", 'w' or 'a' #write or append, encoding = '#utf-8') as #f: #automatically closes the file
  • #f.read(#number of data e.g. 4) #read the first 4 data
  • #f.readlines(#number of characters to be read e.g. 4) #read each line
    • for #line in #f:
      • print(#line)
  • #f.write("#my first file\n") #write data
    • for #line in #f:
      • #f.write(#line)
• #f.name #return name of file
• #f.mode #return mode of file (e.g. read, write)

• with open("#file.txt", 'r') as #r:
  • with open("#new.txt", 'w') as #w:
    • for #line in #r:
      • #w.write(#line)

• f.close() #close file

• import os
  • os.getcwd() #get current working directory
  • os.chdir('#D:\\Hello') #change current directory
  • os.listdir() #list all sub-directories and files in that path
  • os.mkdir('#test') #making new directory
  • os.rename('#test1','#test2') #rename directory
  • os.remove('#old.txt') #delete file

Importing function from other modules

• import #function as #shortened function
  • e.g. import #pandas as #pd
    • #data= #pd.read_csv or read_excel or read_sql(#path or url, header = #e.g. None)
    • #data.head(#e.g. 5) #show first e.g. 5 rows, can also be used to extract top e.g. 5 rows
    • #data.tail(#e.g. 5) #show last 5 rows
    • #data.columns = [#header1, #header2] #add column headings
    • #dataframe = pd.DataFrame(#data) #import value into dataframe
    • #dataframe.dropna(subset=["#col"], axis=#e.g. 0 for row or 1 for col, inplace = #e.g. True to rewrite data back to dataframe) #remove NA
  • import #numpy as #np
    • numpy.array([#elt1, #elt2])
    • type(#array) #returns type of array e.g. numpy.ndarray
    • #array.ndim #returns type of dimension, i.e. number of nested list (e.g. 2D for [ [], [] ])
    • #array.dtypes #returns type of elements in array
    • #array.shape #returns size of array e.g. (e.g. (3, 2) for 3 rows (i.e. axis 0, i.e. first [ ) and 2 i.e. columns (i.e. axis 1, i.e. subsequent nested [ ) )
    • #array.size #returns total number of elements in array (e.g. 6 for 3 rows and 2 columns)
    • #array.T #transpose array
• from #e.g. math import #e.g. pi #import variables from other modules

Packages

• Installing packages
  • get-pip.py
  • python3 get-pip.py
  • pip3 install #package
  • import #package as #abbreviation
  • from #package.subpackage import #function as #abbreviation
• Scientific Computing Libraries
  • Pandas #data frame and tools for fast indexing access to data, data manipulation and analysis
  • NumPy #fast processing for arrays and matrices
    • np.array(#list)
    • #array.shape #display row and col
    • np.random.normal(#min, #max, #numbers)
    • np.column_stack((#col1, #col2))
  • SciPy #math problems (differential equations, optimization) and data visualization
  • math #math operations
• Data Visualisation Libraries
  • Matplotlib #plot graphs
    • print ('Matplotlib version: ', mpl.__version__)
  • Seaborn #heat map, time series and violin plots
• Algorithmic Machine Learning Libraries
  • Scikit-learn #regression, classification, Clustering
  • Statsmodels #statistical models and tests

Exporting data frame

• pandas: #dataframe.to_csv('#filename.csv') #export data frame to csv

Variables

• Index starts form 0 in Python, 1 in R

• True, False, None are in proper case, other keywords are in lowercase (e.g. if, for)

• global #var #define global var in function (Note: if function does not have the variable defined, Python will check global variable)

Data Types

• int() #integer [Note: int64 in Pandas]
• float() #number with decimal [Note: float64in Pandas]
• str() #string [Note: object in Pandas]
• constant #constants usually written in a separate module (e.g. constant.py) in CAPITAL_LETTERS (e.g. PI), then call for constant.PI

• #dataframe.isnull() #check for null values and return True or False
• #dataframe.notnull() #check for non-null values and return True or False

• #dataframe["#col"].value_counts().idxmax() #count categorical data e.g. number of missing data and identify max value
  • for #col in #dataframe.columns.values.tolist():
    • print (#dataframe[#col].value_counts().to_frame()) #count number of values

• #dataframe["#col"] = #dataframe["#col"].astype("#data type") #convert type

• a = 0b1010 #Binary Literals
• b = 100 #Decimal Literal
• c = 0o310 #Octal Literal
• d = 0x12c #Hexadecimal Literal
• float_2 = 1.5e2 = 150
• x = 3.14j (x.imag = 3.14, x.real = 0)

• multi-line code: \ or inside parentheses ( ), brackets [ ] and braces { }
• multi-line comments for documentation: ' ' ' or " " " #help(#function) will return the documentation of the function
• print documentation string (Comments): print(x.__doc__)

• type() #find out type [Note: dtypes in Pandas]
• isinstance(e.g. 1+2j, complex) #check type
• #dataframe.describe(include="all") #display count, [unique, top, freq], mean, std, min, 25%, 50%, 75%, max in Pandas
• #dataframe.info() #display top 30 and bottom 30 rows of data frame in Pandas

• str.capitalize('#value') #Capitalize first letter
• str.upper('#value') #Uppercase
• str.replace('#old', '#new) #Replace value (Not for list)

• #list.index('#value') #retrieve value
• #list.count('#value') #retrieve count
• #list.append('#value') #append value
• #list.remove('#value') #remove value
• #list.reverse() #reverse values

• #dataframe[['#col1', '#col2']] #retrieve column
• #dataframe.columns.values #retrieve columns
• #dataframe.ix[#row', '#'first_col':#last_col'] #retrieve value
• #dataframe.iloc[#row', '#'first_col_no':#last_col_no'] #retrieve value based on index
• #dataframe.loc[#row', '#'col_name'] #retrieve value based on label
• data.loc[#df.#col.isin(['#value1', 'value1'])] #retrieve value based on label and filter
• #dataframe['#col'].unique() #retrieve unique values in column
• #dataframe[#dataframe['#col'] > '#criteria'] #retrieve values
• list(map(str, range(#col, #col))) #convert numbers to string

Objects

• Object is an instance of a type, with data attributes with set of procedures interacting with it (i.e. methods / functions)
• Class has data attributes and methods
  • class #name of class (#parent of class):
    • def _init_(self, #attribute1, #attribute2)
      • self.#parameter = #parameter;
• dir(#object) #return data attributes and methods

String Manipulation

String manipulation

• Last element (i.e. starting from the back as -1, not -0) can be represented using #str_name[-#n]
• #str_name[#start:#end] #select characters from start to end-1
• #str_name[::#n] #select characters in every other n
• #str_name[#start:#end:#n] #select characters from start (starting as 0) and every other n from start (including) to end-1

• len(#str_name) #length of string

• #str1 + #str2 #concatenate string

• #n * "#str" #repeat string for n times

• \n #new line
• \t #tab
• \\ #tab or place r in front of "str"

• #str.upper() #convert string to uppercase
• #str.replace('#old', '#new') #replace string
• #dataframe["#col"].replace('#numpy.nan', #dataframe["#col"].mean()) #replace NaN string
• #dataframe.set_index('#col', inplace=True) #set index for column
• #dataframe.reset_index(drop=True, inplace=True) #reset index for rows
• #dataframe.rename(columns={'#old col name' : '#new col name'}, inplace = True) #rename column
• #str.find('#str') #find substring - return actual index value of first match, else -1 if not found

• #new_list = sorted(#list) #sort string and return new list
• #list.sort() #sort string in original list
• #list.reverse() #sort string in reverse order

• #fruits = ["#apple", "#mango", "#orange"] #list: ordered sequence of (various types of) items which can be modified
  • #list.append([#var1, #var2]) #add new elements as a 1 element to list
  • #list.extend([#var1, #var2]) #add new elements to list
  • del(#list[#n]) #delete items in list
  • #list.split("#delimiter") #split string and add to list
  • #B:#A[:] #clone list
• #numbers = (#1, #2, #3) #tuple: ordered sequence of (various types of) items which cannot be modified
• #tuple = (#1, (#2, #three), #4) #nested tuple e.g. #tuple[#1][#1] returns #three
• #alphabets = {'#a':'#apple', '#b':'#ball', '#c':'[#cat, #carrot]'} #dictionary
  • #dictionary["#key"] will return value
  • '#key' in #dictionary will return True if it is found
  • #dictionary.keys() returns all keys
  • #dictionary.values() returns all values
• #vowels = {'#a', '#e', '#i' , '#o', '#u'} #set: unordered collection of unique items, meaning no element position and have unique elements
  • #set = set(#list) #convert list to set
  • #set.add("#value") #add value
  • #set.remove("#value") #remove value

• #set1 & #set 2 #return intersection
• #set1.difference(#set2) #return difference between sets

• #set1.union(#set2) #return union
• #set2.issubset(#set1) #return if set 2 is subset of set 1
• #set2.issuperset(#set1) #return if set 2 is superset of set 1

• pandas.get_dummies(#dataframe["#col"]) #one-hot encoding to convert categorical data to numeric data in respective columns
• #dataframe = pandas.concat([#dataframe1, #dataframe2], axis=1) #merge data frame
• #dataframe[['#col1', '#col2', '#col3']].groupby(['#col1', '#col2'], as_index = True).#function e.g. mean() #group by
• #dataframe.pivot(index='#col1', columns='#col2') #Pivot Table

Input and Output

• num = input('Enter a number: ') #prompt users to key in

• print("The value is", x) #prints The value is x (space automatically inserted)
• print("a[0:2] = ", a[0:2]) #returns 1st and 2nd value (excluding last value)
• print('The value of x is {} and y is {}'.format(x,y)) #print variables into statements in the order of x, y
• print('The value of x is {x} and y is {y}'.format(x="x",y="y")) #print variables into statements in the order of x, y

Functions

if...elif...else

num = 3

if num > 0:

print(num, "is a positive number.")

elif: #else if

print(num, "is a negative number.")

else:

print("This is always printed.") #return line has no indentation

for

# iterate over the list

numbers = [6, 5, 3, 8, 4, 2]

sum = 0

for #i in numbers:

sum = sum + i

print("The sum is", sum) #output: The sum is 28

for #i in range(#start, #end):

sum = sum + i

print("The sum is", sum) #output: The sum is 28

for #i , #list in enumerate(#list):

sum = sum + i

while

while i <= n:

sum = sum + i

i = i+1 # update counter

print("The sum is", sum)

continue

• continue statement is used to skip the rest of the code inside a loop for the current iteration only. Loop does not terminate but continues on with the next iteration.

pass

• pass statement is used for empty function or loop #return None

function

def print_lines():

print("#line1")

print("#line2")

return #x

#function call

print_lines()

map

numbers = (1, 2, 3, 4)

result = map(lambda #function e.g. x: x*x, #variables e.g numbers) #map is like sapply in R (apply function to each element); lambda is a temporary unnamed function

print(result)

# converting map object to set

numbersSquare = set(result)

print(numbersSquare)

Model Validation

• import pandas as pd
• from sklearn.metrics import mean_absolute_error
• from sklearn.model_selection import train_test_split
• from sklearn.tree import DecisionTreeRegressor

• # Path of the file to read
• iowa_file_path = '../input/home-data-for-ml-course/train.csv'

• home_data = pd.read_csv(iowa_file_path)
• # Create target object and call it y
• y = home_data.SalePrice
• # Create X
• features = ['LotArea', 'YearBuilt', '1stFlrSF', '2ndFlrSF', 'FullBath', 'BedroomAbvGr', 'TotRmsAbvGrd']
• X = home_data[features]

• # Split into validation and training data
• train_X, val_X, train_y, val_y = train_test_split(X, y, random_state=1)

• # Specify Model
• iowa_model = DecisionTreeRegressor(random_state=1)
• # Fit Model
• iowa_model.fit(train_X, train_y)

• # Make validation predictions and calculate mean absolute error
• val_predictions = iowa_model.predict(val_X)
• val_mae = mean_absolute_error(val_predictions, val_y)
• print("Validation MAE when not specifying max_leaf_nodes: {:,.0f}".format(val_mae))

• # Using best value for max_leaf_nodes
• iowa_model = DecisionTreeRegressor(max_leaf_nodes=100, random_state=1)
• iowa_model.fit(train_X, train_y)
• val_predictions = iowa_model.predict(val_X)
• val_mae = mean_absolute_error(val_predictions, val_y)
• print("Validation MAE for best value of max_leaf_nodes: {:,.0f}".format(val_mae))

• # Set up code checking
• from learntools.core import binder
• binder.bind(globals())
• from learntools.machine_learning.ex6 import *
• print("\nSetup complete")

• from sklearn.ensemble import RandomForestRegressor
• # Define the model. Set random_state to 1
• rf_model = RandomForestRegressor (random_state = 1)

• # fit your model
• rf_model.fit(train_X, train_y)
• rf_preds = rf_model.predict(val_X)

• # Calculate the mean absolute error of your Random Forest model on the validation data
• rf_val_mae = mean_absolute_error(rf_preds, val_y)
• print("Validation MAE for Random Forest Model: {}".format(rf_val_mae))

Map

• import geopandas as gpd
• #full_data = gpd.read_file(".shp")
• #data = #full_data.loc[:, ["#CLASS", "#COUNTY", "#geometry"]].copy() #Subset
• #data.#CLASS.value_counts() #Count of col

• #ax = #counties.plot(figsize = (#10, #10), color = '#none', edgecolor = '#gainsboro', zorder = #3)
• #data1.plot(color='#lightgreen', ax = #ax)
• #data2.plot(color='#maroon', markersize = #2, ax = #ax)

• folium: #map = folium.Map(location=[#lat, #long], zoom_start = #e.g. 4, tiles = '#e.g. Mapbox Bright or Stamen Toner or Stamen Terrain')

#Obtain geojson

• !wget --quiet #url.json -O #name.json
• geojson: {"type" : "FeatureCollection", "features" : [{"type" : "Feature", "properties" : {"name" : "#country"}, "geometry" : {"type" : "Polygon", "coordinates" : [ [#lat, #long], [#lat, #long] ] } "id" : "#3 letter country name"}

#Create point and label

• #point = folium.map.FeatureGroup() #create point showing feature group
  • for lat, long, in zip(#dataframe.Y, #dataframe.X, , #dataframe.Category):
    • #point.add_child(folium.features.CircleMarker([#lat, #long] radius = #e.g. 5, color = '#e.g. red', fill = True, fill_color = '#e.g. red', fill_opacity = #e.g. 0.6))

• #latitudes = list(#dataframe.Y)
• #longitudes = list(#dataframe.X)
• #labels = list(#dataframe.Category)
• for #lat, #long, #label in zip(#latitudes, #longitudes, #labels):
  • folium.Marker([#lat, #long], popup = #label).add_to(#map)
• #map.add_child(#point)

#Add clusters

• from folium import plugins
  • #point = plugins.MarkerCluster().add_to(#map)
  • for lat, long, label, in zip(#dataframe.Y, #dataframe.X, #dataframe.Category):
    • folium.Marker(location = [lat, long], icon = None, popup = label).add_to(#point)

#Create choropleth map

• #map = folium.Map(zoom_start = #e.g. 4, tiles = '#e.g. Mapbox Bright or Stamen Toner or Stamen Terrain')
  • #country_geojson = r'#mapname.json'
  • #map.choropleth(geo_path = #country_geojson, data = #dataframe, columns = ['#country', '#col'], key_on='#key in GeoJSON file that contains country name e.g. features.properties.name', fill_color = "#e.g. YlOrRd', legend_name = "#legend name", threshold_scale = #rescaling e.g. threshold_scale)
    • threshold_scale = np.linspace(#dataframe['#col'].min(), dataframe['#col'].max(), #number of bins e.g. 6, dtype = #int)
    • #threshold_scale = threshold_scale.tolist() #change numpy array to a list
    • #threshold_scale[-1] = #threshold_scale[-1] + 1 #make last value of list is greater than max
  • #map