Introductions
Discussion of Research Careers
Title IX Training
Discussion of Research Procedures
Discussion of Effective Reading/Writing of Research Papers
Begun RCR Training
Begun Fundamentals Training
Completed RCR Training
Continued Fundamentals Training
Was introduced and added to this Research Project
Discussed initial project and literature review goals
Continued Fundamentals Training
Began Literary Research(9 Papers Found)
Attended first Advisor meeting to set workflow and discuss literary review strategies
Continued Fundamentals Training
Consolidated current Literary Review information into a streamlined document
Created the initial version of this website
Completed Fundamentals Training
Created initial rough draft of abstract
Held project progress presentation with Jake
Continued Literary Review (4 papers selected for further review on ACM)
Advisor Meeting
Discussed potential disruptions due to schedule changes
Discussed resources for neural networks, initial target project goals, and starting points for data acquisition and model creation
Created Presentation, Team, and Blog pages for the website
Added my information from 5/20-5/28 to this blog
Transferred current deliverables to REU supplement drive.
Reviewed literature collected from ACM.
Removed 1 paper for lack of relevance
Collected 2 more papers for later review
Daily Meeting with Research Mentor
Discussed the potential of building a transformer model for the project, whether or not it would get advisor approval
Discussed getting original presentation slides from the advisor for overall project goal reference
Discussed what's novel about our project (Multi-resolution Data, Trading Strategy, Architecture)
Noted that Yahoo Finance API was already looked into, changed data API focus onto Alphatrade & Alphavantage
Discussed what futures were with Jake
Completed review of papers sourced from ACM (5 papers)
Began looking into Alphatrade python package, discovered that the relevant trading platform was only accessible to citizens of India
Discovered that Alphavantage's terms of service might consider our project as a commercial usage of their API
Alphavantage also separates Crude Oil into two API calls and does not provide access to data on Gold
Began reading "Understanding Deep Learning" for reference when working on the transformer model with Jake
Daily Meeting with Mentor
Discussed current Lit. Review process
Discussed multi-head attention mechanisms
Discussed Alphavantage TOS, recommended to test API to gauge viability
Recommended to look into Tradingview site for data collection.
Discussed how tradingview handles data with Jake. Determined that free trial would give us 42-180 days of 1min intraday data
Sent email to Alphavantage on clarification of TOS, was told that academic research is permissible under non-commercial usage of their API
Learned that Gold is shared across multiple trusts, need to find a forex for Gold
Learned that Alphavantage paywalls intraday info on Forex, no support for Gold
Learned that Tradingview has a dedicated futures page for Gold and Crude Oil
Learned that accessing Crude Oil from Alphavantage would place FRED api TOS on us. Recieved data would also be coarse
Learned that Alphavantage does not currently support accessing data on indices
Formulated plan for obtaining data from Tradingview by chunking csv's and merging them together
Refined Literature Review on recently found papers for Advisor meeting
Advisor Meeting
Discussed Jake's sourcing of 23 years of Euro/USD data at 1 minute time-steps, along with the transformer model he built
Introduced to Mostafa, another contributor to our project
Clarified that we are predicting stock prices rather than predicting stock trends
Discussed how our work should have a "story" (Recognizing previous attempts, noting what's new about our attempts)
Noted how you are allowed to send emails to all contributors of a paper, but only Corresponding Author is expected to respond
Discussed strategy for supplying paper figures in literature review document
Discussed plan to acquire data on remaining indices through Tradingview
Discussed categorizing papers from literature review for purposes of manuscript
Discussed objectives for our next presentation
Discussed where to train our transformer model and how to configure it
Recommended to look further into "Hierarchical Multi-Scale Gaussian Transformer for Stock Movement Prediction", a paper by Qianggang Ding et. al which proposes unique optimizations to Transformer models for interfacing with stock data
Began preparations for data collection from Tradingview
Learned that Tradingview uses an alternative exchange for data
Daily Meeting with Mentor
Discussed how to interpret OHLCV data in small timesteps
Discussed desired inputs for the transformer model
Discussed how/why gaps appear in market data, and how smaller timesteps are more, but not exactly, continuous
Discussed how you can view market charts in either linear or logarithmic view depending on scaling
Discussed ways to change parameters of the transformer model to train it faster
Discussed plans to learn more about transformer architecture
Learned that Tradingview chart is restrictive, could only access 42 days worth of 1 minute data
Used the Bar Replay feature of Tradingview to double the amount of data, collected 82 days worth of 1-minute data on remaining indices
Devised a plan to continually update data sources from Tradingview with current-day data.
Wondered if predicting stock prices was similar to predicting the positions of electrons
Read "Understanding Deep Learning" to understand the architecture of a Transformer head
Discussed how to process stock data into transformer model with Jake
Prepared for 6/3 Presentation
Looked at review papers for data sources, found Investing.com
Began process to register for MID-Sure
Discussed seasonal patterns in a trading year, and how stock data is a discrete function
Discussed how we should prioritize processing requirements for our model
DIscussed how to present data to the transformer and balancing between training speed, quality and data size
Discussed how we can normalize the data set
Collected 143 days worth of 1-minute data from Investing.com[12/1/23-4/21/24]
Discussed if Investing.com should be the main source of index data until better source is found with more historical data
Received links from mentor's collaborator for more data sources.
Learned about alternative names for indices[Dow Jones -> US 30, Nasdaq -> US 100]
Collected 2 years of 1-minute data for Dow Jones from Dukascopy [1/1/22 - 1/1/24]
Collected 14 years of 1-minute data for Gold/USD from Histdata [2009 - 2023]
Learned that there are multiple Crude Oil Exchanges in the US (Brent, Light, WTI)
Setup registration for Mid-SURE 2024
Setup dependencies to begin collaborating on the Transformer model with Jake
Daily Meeting with Mentor
Discussed embeddings and stock prices going outside of embedding ranges
Discussed current data situation
Discussed the current state of the Transformer model along with collecting comparison models
Discussed the current state of Mid-SURE 2024 Registration
Researched the embedding layer to see how it can better handle integer input
Learned that problem space is adjacent to Transformers forecasting Time Series data
Researched Transformers and Time Series Forecasting, discovered models such as the inverted Transformer
Attempted to get Transformer code persistent access to Training Data while hosted on Google Colab
Worked on and implemented Gaussian Prior Bias in Transformer Model (Based on implementation in "Hierarchical Multi-Scale Gaussian Transformer for Stock Movement Prediction")
Discussed investigating existing time series models
Discussed questions for advisor meeting
Created a Time Splitter Mask for the transformer model
Discussed ways to make sense of evaluation, such as plotting prediction output on a graph
Trained Model with Gaussian Prior, got results similar to baseline
Trained Model with Gaussian Prior and Time Splitter Mask, got results worse than baseline
Discussed how to identify overfitting
Advisor Meeting
Discussed how to share the current baseline version of the Transformer model with the entire team
Discussed the current state of the Transformer model
Discussed how to calculate the output of the Transformer model, such as plotting loss
Discussed using Mean Square Error rather than Cross Entropy Loss to better compare the model to existing methods
Discussed ways to use a trained model
Discussed the current state of Time Series Transformers
Discussed the question of how different NLP and Time Series Forecasting is
Discussed how Arxiv is the "bleeding edge" for academic papers since they are often uploaded before being published in a journal
Discussed what the inputs & outputs of the Transformer model should be, and how long-term our predictions should be
Settled on making the Transformer model output 2 hour-long predictions
Discussed training on sequential or randomized chunks of the dataset
Reiterated the need to look at both high-resolution and low-resolution data along with multiple indices
Discussed professional trading component, how it will likely be a classical system, and how it will heavily rely on the reliability of the Transformer's prediction.
Daily Meeting
Discussed brainstorming the finer details of the abstract
Discussed what influences the inputs/outputs of the Transformer model
Discussed the differences between accuracy and loss
Discussed comparison models & availability on GitHub
Discussed potential for extra meeting today
Broke down the current abstract and title into distinct points for revision
Researched multivariate time series forecasting
Discussed how T4 GPUs on Google Colab are similar to GTX 1070s with extra ram
Prepared 6/10 presentation
Hypothesized "Sliding Trust" for Transformer predictions in trading strategy algorithm and a GUI application for demoing project.
Found an example notebook for training flow-forecast models
Prepared for the weekly presentation
Learned about paperswithcode.com from Jake
Looked into how to plot model predictions on a graph
Discussed what to present for midterm, and whether or not it should be a GUI application
Daily Meeting with Mentor
Discussed outstanding issues with the project
Went over new model improvements
Was recommended to provide more figures in future presentations
Discussed how to benchmark against other models
Checked maximums and minimum bid/ask spreads in data sets.
Began scaffolding GUI application for showing Transformer predictions in dearpygui
Added functionality for loading and plotting training loss
Added functionality for loading and plotting stock data
Daily Meeting with Mentor
Discussed how our current training loss is higher than our validation loss during some runs
Discussed new abstract, steps to training, single-resolution training vs. multi-resolution training, and single variate vs. multivariate training
Discussed ways to plot predictions of the model
Discussed testing methodology with Jake
Reviewed papers for Informer and Autoformer to understand their architecture
Discussed the need for local computing power due to limitations imposed by Google Colab
Discussed holding off on demo application to focus on evaluating model results
Discussed how the Transformer could correlate stock trends to the time of day
Created data preprocessor script to unify the timestamp date in datasets
Began creation of trading strategy module, scaffolding its required elements
Setup remote access to a home computer with Chrome Remote Desktop found that performance was poor
Daily Meeting with Mentor
Discussed current progress
Discussed hyperparameters and testing various scalers on the product
Discussed training multiple models and using AnyDesk to access home computer
Got Informer to train on EUR/USD sample in Jupyter notebook, but received strange results
Setup training for AutoInformer on EUR/USD sample through PyPOTS library
Modified data preprocessing script to allow converting timestamp data into other formats besides Unix timestamps
Discussed oddities of Unix timestamps in training data, later realized that oddities were due to floating-point imprecision
Moved the Informer notebook to the PyPOTS library
Created a revised Draft of the Abstract
Discussed how all data in a tensor must share the same data type
Setup remote access with good performance to my home computer
Ran Transformer model on local RTX 3070
Looked into automating testing of notebooks, chose Papermill for automating executing notebooks with injected parameters
Scaffolded example notebook using Papermill to automate notebook execution
Created a document for noting what parameters are worth modifying to test model efficiency
Created notebook for testing Inverted Transformer on EUR/USD sample
Tested out various target sequence lengths to see how loss over time is affected
Discussed whether to include days with incomplete ranges
Created a larger 2-year sample of EUR/USD data for training
Created model weight file of Transformer model with 50 inputs and 25 outputs
Advisor Meeting
Discussed desired outputs of models regarding plots
Discussed how to convey the meaning of plots
Discussed inverting scaler preprocessing to get predictions on the original data scale
Discussed showing when a data set is in time
Hypothesized why the base Transformer is currently making very conservative guesses on predictions
Discussed how it would be more accurate to receive longer input sequences and predict shorter output sequences
Discussed differences in loss output between Colab and local RTX 3070
Discussed how seasonal patterns are unlikely to occur at the intraday level
Discussed adding seed numbers to models to improve reproducibility
Discussed how we aren't going to/from a domain, influencing the use of the decoder/encoder blocks
Discussed when the model should be causal
Discussed potential of shifting focus to include classical models instead of more deep learning models
Daily Meeting with Mentor
Discussed the plan to update comparison models to output similar graphs as the base Transformer
Discussed balancing parameter testing and time taken
Discussed Colab Pro
Prepared for the next presentation
Discussed why RTX 3070 may not be fully utilized during training
Began reworking AutoFormer to predict values rather than imputing them
Learned how scaling needs to have preservation of tensor shape to allow inverse transforms
Discussed workflow for testing models and sending batches to train
Discussed how I would prioritize studying classical models in case predictions were not accurate with the comparison Transformers
Began execution of 23 tests, each taking approximately 2 hours
Discussed oncoming heat wave, and how cooling for RTX 3070 would be inadequate for the high temperatures
Devised plan to limit RTX 3070 usage to the night when it was cooler, using the GTX 1070 during the day since it resides on campus
Finished 36 hours of testing, completing 15 tests
Discussed results, and how Google Drive truncated file names during upload
Prepared for the weekly presentation
Worked on loading models into the demo application
Studied optimal methods of saving models & using them in other projects
Discussed how to split up training time across available computing sources
Modified early stopping to decrease the amount of model weight files saved
Realized that PyPOTS configures most models for imputation only, instead of forecasting
Restarted work on comparison models with the "Time-Series-Library" library
After Presentation Meeting
Discussed problems with current output
Discussed plans for the week, consisting of testing the base model with embeddings and standardizing output across all models
Discussed plan for today and Wednesday
Created a fork of the Time-Series-Library repository to fit the needs of the project
Worked to get Autoformer running on data, was able to progress to the training loop
Daily Meeting with Mentor
Discussed current progress
Discussed current problems with the embedding in the base transformer, along with quantization
Discussed setting up 1-year dataset
Created 1-year dataset for EUR/USD data
Configured the output of the Autoformer
Discussed how to standardize plot outputs
Discussed the possibility of caching pip installs for Google Colab environments, saving time when running notebooks on new instances.
Tweaked graph output to show the real values of EUR/USD, rather than scaled ones
Ran tests on Autoformer, Informer, and Inverted Transformer
Hypothesized with another group on why current comparison model predictions are poor
Created ability for comparison models to plot loss over training epochs
Updated tests to include loss visualization
Advisor Meeting
Discussed outputs of comparison models and base models
Discussed presenting outputs for the upcoming Midterm presentation
Decided on what to present as results: Predictions for 50 inputs/10 outputs and 50 inputs/outputs, and models ranked by MSE
Discussed the potential of using pre-trained models from Huggingface in the future
Discussed how research can rush results, losing opportunities for further insights into reasons for results
Discussed how to explain the results during the Midterm presentation
Discussed how to come up with conclusions
Daily Meeting
Discussed outline of Midterm presentation
Discussed whether or not to force models to train for full epoch length
Discussed how/if to complete the rest of the test suite
Discussed improving training time on PyTorch base transformer
Discussed how the PyTorch base transformer embeds input
Discussed tensor squeezing & unsqueezing., and how linear layers extrapolate values
Discussed why it would be unnecessary to do squeezing and unsqueezing on some data inputs
Began working on the midterm presentation
Discussed current state of models with another research group
Attempted to add loss into day graph output in comparison models, but ran into an issue with time embedding
Prepared for Midterm presentation
After Presentation Meeting
Discussed fixing comparison model graph legend
Discussed working on a fixed embedding for comparison models
Discussed how to start trading strategy
Discussed modifying embedding blocks to denote minutes in a day rather than arbitrary position
Discussed how to test trading strategy on predictions
Discussed how to continue meeting after advisor leaves the country
Discussed future of project after REU program
Discussed acquiring Google Colab Pro
Updated comparison model graph output with three separate lines for Input Sequence, Target Sequence, and Prediction
Discussed how fixed-time embeddings work on comparison models
Discussed initial trading algorithm
Discussed loss functions and how one could make them account for volatility
Setup comparison models for working on other datasets
Discussed saving the scalar for future use after training
Discussed why comparison model loss was so low despite their prediction behavior
Ran tests to check loss calculations on comparison models
Began working on trader evaluation algorithm
Daily Meeting with Mentor
Discussed current status of work
Discussed possibility of using a different loss function that considers volatility
Discussed current efforts to improve base Transformer models
Continued work on trading algorithm
Researched volatility indicators
Ran into issue with CUDA asserts while extending NASDAQ dataset
Daily Meeting with Mentor
Discussed the current state of progress
Discussed what to present at the advisor meeting
Discussed possible pathways for the project depending on whether we can find a model with acceptable predictions
Discussed attention masking in the base transformers
Ran comparison models on NASDAQ, Dow Jones, and Gold data
Checked to see if CUDA asserts happen on Google Colab, and learned that they assert only on the fixed embeddings
Advisor Meeting
Discussed advisor going out of the country, when to decide on future of research
Met with Doctor Fu and Professor Hannah, and discussed a new location to continue research at
Discussed the need to preprocess data further
Discussed using linear operators for multi-resolution data
Discussed using RLN/GLN or recursion for trading strategy
Completed tests on NASDAQ, Dow Jones, and Gold data
Worked on further data preprocessing to slim data range to the actual day trading period, along with blocking holidays
Met with Professor Hannah at their lab to discuss the new workspace
Triaged CUDA assert to Fixed Embedding creation
Trained comparison models on new preprocessed NASDAQ data
Prepared for Monday's presentation
Confirmed that the models' prediction behavior is dataset agnostic
Realized that the Inverted Transformer will not work well when training on single data feature
Discussed results of tests with Jake
Determined that we need to refocus our efforts for the remaining time in this program
Prepared for presentation
After Presentation Meeting
Discussed content of presentation
Discussed dropping the original Transformer model the project started with
Discussed how to better incorporate the Inverted Transformer into our workflow by supplying it with more features
Discussed where our priorities should lie in current work
Discussed how the base Transformer is somehow using masking on sequences rather than individual values
Discussed how to setup multivariate prediction on comparison models
Discussed trading algorithm,how much they should trade per action
Discussed using Deep Learning for trading algorithm
Created a script to make multivariate data for the inverted transformer
Preprocessed Dow Jones, Gold, Oil, EUR/USD data
Began to train Autoformer and Informer on preprocessed data
Discussed current results of experiments with Jake, and was recommended to look into learning rate scheduling
Began creation of a script to create a multivariate dataset for the Inverted Transformer
Outlined new method of presenting work of the week during advisor meetings
Discussed adding learning rate scheduling to comparison models, but decided it wasn't worth it due to what was already implemented
Finished creation of multivariate dataset for the Inverted Transformer
Created loss functions that are influenced by the volatility of predictions
Discussed how one would be able to profit off of trading on EUR/USD
Learned that EUR/USD data is in BID prices and that we'd need to find the equivalent ASK data to determine the current commission rate.
Daily Meeting with Mentor
Discussed current progress
Discussed root mean squared error and mean squared error
Discussed where to estimate market intricacies in trading algorithm
Discussed if we should account for inflation in trading algorithm
Found BID/ASK data for EUR/USD and Gold on Dukascopy, decided to unify all data to be from Dukascopy
Reviewed current testing procedure on comparison algorithms
Found that multivariate data did not improve prediction behavior for Inverted Transformer
Tested Autoformer with MSE and VRMSE
Went to Open Review to analyze discussion around Inverted Transformer, found that the Inverted Transformer became a stackable Linear Forecaster under univariate data scenarios
Realized that model loss targetting was bugged in previous Inverted Transformer experiment, but a fix did not produce correct results
Discussed how VRMSE improved prediction behavior on Autoformer
Tested VRMSE on Inverted Transformer and used VRMSE for other datasets on Autoformer and Informer
Discussed how learning rate adjustments could impact Inverted Transformer
Created static learning rate feature for comparison models, ran experiments with Inverted Transformer and found that it also did not improve prediction behavior
Tested Inverted Transformer with increased encoder layers, found that prediction behavior did not improve
Re-ran Gold with VRMSE loss experiment due to errors that occurred
Discussed what a manuscript would entail
Prepared for discussion of project with Dr. Hannah
Hypothesized more variations of VRMSE
Prepared for Monday presentation
Discussed loss, RMSE and volatility
Reviewed Mid-SURE requirements
Meeting with Mentor and Dr. Hannah
Discussed goal of project
Discussed if you can even buy stock in NASDAQ
Was suggested to experiment with input/prediction sequence length
Discussed what the model exactly is being trained on
Recommended to look into trading volume as a metric
Discussed a range of trading decisions
Discussed how error rates are problematic as a metric
Discussed need to prove stability due to arbitrary parameters
Discussed how this project has two main aspects: the predictions and the trading algorithm
Discussed how you can separate output sequence into training & inference
Discussed how the derivative of the loss function is important
Prepared for presentation
Discussed current objectives
Discussed present time constraints
Looked into Mid-SURE poster requirements
Set deadline for poster creation to 7/17
Discussed hierarchical view of sequences to plot loss on models with other research group
After Presentations Meeting with Mentor
Discussed calculated covariance results
Discussed hardcoding the BID/ASK spread, due to the fact that models could predict impossibilities
Discussed results of loss function experiments
Discussed dividing trading actions into 50 minute chunks
Discussed design notes for poster
Made note of required deliverables for program
Setup skeleton of poster and research report
Discussed using the base Transformer code from Time-Series-Library to replace the independent PyTorch based Transformer
Added the ability to save the scaler and optimizer parameters for a model
Daily Meeting with Mentor
Discussed data correlationresults
Discussed the research report
Discussed plans for the week
Updated notebook output to print on the fly
Drafted introduction section of research report
Analyzed CUDA asserts for fixed embeddings
Discussed oddities with the label length parameter for our models
Discussed issues with hardcoding in the library
Fixed CUDA asserts by fixing order of embedding array insertion
Uploaded test results for posterity
Continued drafting the introduction section of the research report
Began drafting the literature review section of the research report
Finalized Abstract & Title
Setup model weights for trading algorithm
Discussed problem timestamps were having with daylight savings time
Discussed problems with timezones in processed data
Ran first inference test with Autoformer
Began drafting the methodology section of the research report
Daily Meeting with Mentor
Discussed what to put in the abstract, focusing on comparisons besides competitive results
Discussed whether or not we go in-person to Mid-SURE, kept original plan to go in-person
Discussed current progress
Discussed required deliverables for the program
Updated abstract and title with Jake on our Mid-SURE application
Updated VRMSE versions of models for experiments
Continued drafting methodology & literature review sections fo research report
Prepared for Monday presentation of poster draft
Daily Meeting with Mentor
Discussed BID/ASK spread & ways to estimate it
Discussed whats important to put on our poster
Prepared draft of Poster
Completed setup of inference models
Discussed plans for testing
Discussed plans for today
Continued work on the literature review section of the research report
Discussed discrepancies in the data timestamps & our intended trading range
Daily Meeting with Mentor
Discussed oddities with our data
Discussed state of trading strategy
Discussed diagram design
Discussed needing advisor approval before sending finalized poster(our advisor was out of the country at this point)
Discussed the status of experiments
Discussed how to obtain advisor approval for poster design when advisor is out of the country
Discussed how to effectively compare prediction sequences of different lengths
Got first results from experiments
Daily Meeting with Mentor
Discussed results
Discussed testing the trading strategy with perfect predictions
Discussed how to deal with our (slightly disappointing) results
Discussed comparing trading strategy results with primitive forecasts
Discussed creating a money pool
Generated inference results for all test cases
Discussed state of result methodology
Discussed how to generate results
Saw that for NASDAQ: Transformer MSE beat Informer MSE beat Transformer VRMSE beat Informer VRMSE beat Autoformer VRMSE beat Autoformer MSE
Created graphs for all datasets
Discovered that some results produced no profit trades, eventually learned that this was a bug
Discussed setting more accurate estimate of BID/ASK spread per dataset
Determined commission rates for data
Learned that trading CFD's is banned in the US, leading us to drop Crude Oil from experiments
Wrestled with how to best standardize across datasets for poster results.
Completed final draft of poster for review
Got poster design approved by the mentor and submitted
Worked on the methodology section of the research report
Discussed plans for handling the research report next week
Created a tool for documenting monthly profit from models for the research report
Created data tables for research report
Worked on drafting the results section for the research report
Discussed what arguments to make based on the provided data
Worked on discussion & results section of the research report
Discussed how to incorporate prediction behavior into discussion section
Discussed complications with conclusions
Daily Meeting with Mentor
Discussed what conclusions can be made
Discussed results of additional side-testing
Discussed justifying our choices for parameters
Discussed using formal algorithm notation for describing trading algorithm
Ran experiments to test runtime of input sequences longer than 50
Attended Mid-SURE
Deliverables Meeting
Discussed deadlines for deliverables
Discussed actions needed to be taken after completion of program
Worked on remaining deliverables
Finalized Research Report
Finalization and submission of deliverables