Module 3 Software Training - Data Retrieval using web APIs

Data Retrieval using web APIs and automating the Capital Asset Pricing Model

GitHub hosts numerous open-source Python projects, offering a wealth of reusable code and libraries. Many Jupyter notebooks and tutorials covering diverse topics, from automation to machine learning. Below we make use of Aldo Dector's GitHub to automate the CAPM estimation - heavily relied upon by Accountants and Financial Practitioners when assessing investment appraisal.

Data Manipulation with pandas and numpy

Data Loading and Manipulation using pandas DataFrames

In the realm of automation and rpa, the ability to seamlessly load, manipulate, and transform data is the cornerstone of insightful decision-making. Enter pandas DataFrames, the versatile and dynamic toolset that facilitates navigating complex datasets with finesse. This exploration delves into how to load and manipulate using data using pandas DataFrames, unraveling the essence of this fundamental process. 

The Foundation: Understanding pandas DataFrames

At the heart of pandas, a popular open-source data analysis and manipulation library, like DataFrames. Think of them as two-dimensional data structures akin to tables in a relational database or spreadsheets. Each column within a DataFrame can store data of different types, making it a flexible container for heterogeneous datasets.

Data Loading: The Gateway to Analysis

The journey begins with data loading, a process that entails bringing external data into the pandas environment. One of pandas' most celebrated features is its ability to ingest data from diverse sources, ranging from CSV and Excel files to databases and APIs. Automating Data extraction is hugely beneficial, and drives efficiency. It also can ensure compatibility that the analytical journey isn't constrained by data formats.

Loading data into a pandas DataFrame is remarkably intuitive. A simple one-liner, such as pd.read_csv('data.csv'), can import a CSV file into a DataFrame, ready for analysis. The library's support for various data formats eradicates the need for manual data conversion, streamlining the initial stages of analysis.

Data Manipulation: Sculpting Insights

Data manipulation is where pandas truly shines. It's akin to molding raw data into the shape required for insightful analysis. This involves actions like rearranging columns, filtering rows, merging datasets, and deriving new variables. The beauty of pandas lies in its extensive arsenal of functions that facilitate these operations with elegance.

Columns can be easily selected, dropped, or renamed, making data pruning a breeze. Functions like df['column'].mean() calculate statistics on the fly, while conditional selection extracts subsets that satisfy specific criteria. This flexibility enhances data exploration by allowing analysts to focus on relevant portions.

Data reshaping is another pivotal aspect. The ability to pivot, melt, or stack DataFrames empowers analysts to transform data from one shape to another. This is invaluable for tasks like creating summary tables, reshaping time series data, or preparing data for visualization.

Python facilitates the querying of online databases programmatically. An important database for Economists and Financial Analysts  is FRED Federal Reserve Economic Data  — that has in excess of 800,000 economic time series. They cover banking, business/fiscal, consumer price indexes, employment and population, exchange rates, and many economic time series. These are compiled by the Federal Reserve and many are collected from US government agencies such as the U.S. Census and the Bureau of Labor Statistics.  

0:00 Introduction to QuantEcon Lectures and quantecon-notebook-python

1:13 Load directly the python notebook into Google Colab using github url

3:51 !pip install -- upgrade pandas_datareader version 0.10

5:00 import pandas as pd

5:11 pd.series create a pandas series

8:41 create an index for the pandas series

11:05 create a pandas dataframe

12:58 pd.read_csv and url link

14:25 select particular rows using standard Python array slicing notation

15:45 pandas iloc

17:08 pandas loc

20:18 change population column to scientific notation format using pandas

20:50 create a new column in pandas dataframe

21:26 visualize pandas series and dataframes using matplotlib

24:43 query online databases (e.g. FRED St. Louis Federal Reserve) programmatically using pandas reader

26:00 FRED unemployment data import directly into Google Colab using pandas

Quantecon Github on Pandas

QuantEcon develops  open source software for economic/financial modeling critical to building open, collaborative platforms for sharing. Development is largely based on open source scientific computing environments such as Python, R and Julia. Here we show how to open a  QuantEcon Github Python notebook directly in Google Colab and extract data from external sources.