Introductions + icebreaker activity
Research career presentation
Title IX training presentation
Research paper collection/writing presentation
Research procedures presentation
Began RCR Training
Began fundamentals training
Project presentations + selection
Discussion with Professor Siadat regarding next steps
Continued RCR training
Continued fundamentals training
Completed RCR Training
Began literature review + collection
Attended first weekly meeting with Professor Siadat regarding high level objectives
Completed fundamentals training
Continued literature review (14 papers total)
Holiday
Turned in RCR Training Certificate
Investigated + submitted Oakland's ILLiad resource loan request forms
Conducted project presentation with Ian
Attended meeting with Professor Siadat and Mahdi regarding our modeling approach, data acquisition, and continued literature review objectives
Began watching 3blue1brown's Deep Learning series (4 videos)
Updated project blog website to current date
Attended daily meeting with Mahdi regarding continued research efforts, tasks completed, and today's goals
Received + reviewed + requested documents from Oakland's ILLiad loaning service
Continued literature review - 2 new relevant papers regarding transformer models
Researched futures in the context of the Shanghai crude oil market + INE data
Daily meeting with Mahdi - talked about further data acquisition and challenges
Obtained very granular data for past 23 years regarding EURUSD Forex market
Began looking into Pytorch - got a basic transformer model running
Attend weekly meeting with professor Siadat - met Mostafa, shared progress and future plans
Daily meeting with Mahdi - talked about transformer code and data
Updated Google Drive folder with newly arrived loaned papers and code
Collaborated with Ian, preparing for Monday's Week 3 project presentation
Review two papers for the presentation
Email authors of a paper that Ian found requesting their source code
Made scripts to process Forex Tester EUR/USD data to shorten it for testing and understand its gaps in time
Coded multiple versions of our transformer, incrementally improving and attempting possible solutions.
Attempting different data formats -> sliding window with random numbers now functions
Got data processing it work + functional transformer that uses EUR/USD data with a sliding window
Created two additional versions of the transformer (versions 0.0.4 and 0.0.5)
Corresponded with Mahdi to discretize the EUR/USD values, preserving the embedding layer functionality
Prepared for and presented project presentation
Had daily meeting with Mahdi - conversed about data processing, data acquisition, and the transformer model code
Had a meeting with Professor Siadat after presentations regarding high level transformer implementation and data processing
Investigated the high level functionality of the transformer class
Collaborated with Ian to download necessary datasets, including the USA 100 Technical Index from Jan 1, 2022 to Jan 1 2024
Discussed the output of the model's dimensionality with Ian
Discussed the submission of Mid-SURE application with Ian and next steps
Investigated the TensorDataset and DataLoader pytorch objects, and implemented adjustable prediction window sizes
Began to implement batching
Attended daily meeting with Mahdi
Prepared a document for weekly meeting with Professor Siadat
Investigated limitations of the usage of embeddings with continuous data
Optimized the model for the GPU to gauge its potential performance bottlenecks
Completed implementing batching and splitting the dataset ingto train, validation, and test sets
Added GradScaler and Autocast to cast operations to float16 when appropriate, which will improve performance
Began to benchmark the current model
Continued to prepare for weekly meeting with Professor Siadat
Topics include: transformer code, features, performance, problems, solutions
Investigated the informer model, finding some pytorch code we could potentially use
Had problems with long upload times to google colab
Mounting google drive vs downloading from github vs uploading manually
Made code to save/load the model
Attended meeting with Professor Siadat, Mahdi, Ian, and Mostafa. Takeaways were:
Plot loss function using MSE loss
Fix issues with precision values
Adjust input and target sequences and the output shape
Investigated the vanilla transformer model implementation further
Fixed the output shape to just get a value amount
Further investigated the usage of embeddings and the potential of not needing them
Discussed this with Ian, and ended up removing them
Reduced the number of required parameters that specified the desired input/target sequence data from three to one
Implemented MSE loss function, and adjusted the model to allow its usage
Completed the presentation for upcoming the Monday
Discussed with Ian about the current state of the project
Its current implementation, the problems, and the goals
Found a series forecasting leaderboard on paperswithcode.com
Met with Mahdi - I caught him up with the changes in the current implementation
Gave our presentation and met with Professor Siadat afterwards
Discussed desired inputs, outputs, and time it takes to train
I made a version of the model to run on the TPU
Wrote code to plot the training and validation loss and to save/load the data in a CSV file
Planned on what the next plots I make should include
Attended weekly meeting. Takeaways were:
Make graphs to compare into-day' values
Ensure we have enough time to look into additional models
Investigate comparing our model(s) to other existing models
Investigate optimizing the parameters and standardizing their testing
Gathered all metrics to log and how the data should be represented
Completed getting the graphs working
Meeting with Mahdi
I shared my graphs I had made
Try using other scaling methods other than MinMax like the standard scaler
Talked about plans of fixing some graphing code and new potential graphs
Debugged input sequence timestamp data precision issues
Attempted to change shape of data, but it did not solve the errors I was recieving
Followed bug back to the sequence generation function where the precision was being casted incorrectly
Further adjusted input sequence timestamp data so that it could be batched
Experimented with changing different precision values to minimize memory usage
Standardized variable names to create a distinction between sequences and batches
Fixed graphs and csv files that were affected
Attended weekly meeting
Try to increase the input size and see if that affects the loss
Look into adding back embeddings and adding an encoder
Have an illustration of the model
Meeting with Mahdi
Talked about modelling approach as discussed in last meeting
Discussed automating the running of notebooks with different parameters
Made a new transformer version aimed at being automated with papermill to test the affects of different hyperparameters
Adjustable parameters
Standardized output names
Outputs the time it takes to train and validate per epoch and overall time taken
Discussed and prepared the upcoming presentation on Monday
Implemented early stopping mechanism
Added more hyperparameters to the parameter code block
Debugged and completed the papermill loop
Includes 23 separate paramter sets to test
Sent to Ian to run over the weekend
Meeting with Mahdi
Talked about looking at obtained test results
Investigate using the encoder
Analyze the available first 15 test results that were able to be completed
Main takeaway - reducing the output size reduced the loss the most
I walked Ian through the early stopping mechanism
Gave our weekly presentation and had a meeting with Professor Siadat and Mahdi afterwards. Takeaways:
Analyze step by step how the data is being transformed
Consider using a decaying exponential function for loss to prioritize immediate predicitons
Inverse transform the values before displaying them on the graph
Investigate including temporal data along with value data
Further research the usage of embeddings in regards to combining continuous and categorical data
Implications of including information like the current month/season with the equity value data
Conferred with Ian regarding our current day objectives
Focus on just getting the embeddings from our old transformer version back into our current transformer
Ensure the output is correct and predicting the correct values
Successfully implemented the embeddings
The old version includes an encoder as well. It was added back into this new version
Trained on twenty-five epochs with very poor results due to the data being quantized and having an average value of about ten thousand
Set to train overnight for one thousand epochs
Observed results from the overnight training - loss was significantly better but still poor
Meeting with Mahdi
Mahdi suggested adjusting the learning rate
We conversed about the potential lack of dependence in the sequences' output
Caught the blog up to current
Prepare and attend weekly meeting
Mention investigation into 50 input sequences and reducing output predictions to only 10
Investigate Hugging Face pretrained models - can they predict well for our use case?
Assess new transformer results
Still very poor
Attempt to figure out issue with total embedding/vocab size from 20,000 to under 300
Assess results - poor results. It seems as if the model is overtraining
Meeting with Mahdi
We talked about the PyTorch base transformer model and the current errors regarding the embedding/vocab size and overtraining
Attempted to outline some potential conclusions for the report
Could we talk about the impact of output length of the transformers regarding the performance of predictions?
Correspond with Mahdi about the nature of what a conclusion should look like
Investigate how multi-head attention works so I can describe it for the presentation
Work on upcoming presentation for Monday
Include our currently selected models
Get specifics about Transformer architecture
Encoder - decoder layers and how they operate
Create decoder-only Transformer model based on currently developed Transformer model
Implement the standard scaler for the decoder-only model
Meeting with Mahdi and Professor Siadat
Converse about the general structure of the trading strategy
Investigate lower resolution data and evaluate the prediction performance
Recognize data leak issue where model predictions are impossibly close to actual data and scaling issue
Begin to get model weights and timing metrics for the decoder-only, encoder-decoder, and PyTorch base Transformer models
Give presentation to REU members and professors
Finalize getting the model weights and timing metrics for the decoder-only, encoder-decoder, and PyTorch base Transformer models
Upload papermill tests to the drive folder
Resolve issue with filenames being too long due to their descriptive nature
Correspond with Ian about issue regarding data validation across all models
We need to split the data based on chronology, keeping training, validation, and testing data strictly split
Implement scaler saving and loading functionality
Test Google Colab's performance and usage of the Google Colab Pro Plus subscription
The T4 GPU still seems to perform the best for our use case while not consuming too many resources
Implement learning rate scheduling
Daily meeting with Mahdi
Discussed Google Colab, scaler, time based volatility, loss functions, and coarser data
Begin meeting document for our Thursday weekly meeting
Add current findings and Google Colab results
Added graphs to visually and quickly determine state of our Transformer models
Look into the model's target masking implementation and ensure it is working as intended
Create a script that will generate coarser data based on higher resolution data
Run 4 tests on the transformer with the target masking implementation enabled and disabled with 50 and 500 input and prediction lengths
Read the PyTorch nn.Transformer class's source code to see if it does target masking by default
It does not do target masking by default. Manual implementation is required for the model
Discuss with Ian about bugs regarding fixed embeddings in his library code
Add results of target masking to meeting notes
Analyze why validation loss is higher than test loss, which is an abnormal phenomenon
Configure the papermill script to load and save files from the drive folder directly
Weekly meeting
Dr. Fu and Dr. Hanna attended briefly to discuss Dr. Siadat's departure to Iran
We corresponded with Dr. Hanna to gain access to work in his lab
Present meeting document
Talk about implementing a recursive trading strategy that buys low and sells high
Get papermill script tested and running
Start the recursive trading strategy
Work on presentation for Monday
Talk about basic trading strategy approaches
Mention model target masking complications and further development approaches
Adjust the Transformer notebooks to output the files to different paths and revert to best model weights before evaluation
Go to Dr. Hanna's lab and meet the graduate students that work there
Added model loading functionality to Transformer notebook
Continue working on the trading strategy
Meeting with Mahdi
Look into Transformer model implementation and understand why training time results are strangely short despite long input sequences in testing
Analyze target masking usage and identify bugs with the linear layers that prepare the input data
Ensure the feature amount is correct
Have one feature per minute data, since we are only looking at open value of that particular minute
Fix the target masking that is affected by the feature size adjustments
Run the new version of the transformer with fixed feature sizes twice
The first test used a custom implemented learning rate scheduler based on the Attention Is All You Need paper, and the second uses a simple step learning rate scheduler
Debugged the step learning rate as it was erroring when the notebook was lasting too long
Continued working on the trading strategy
Alter the way data gets processed from a sliding window to a single pass approach
Include timestamps into the predicted output values
Look closely at errors with the step learning rate scheduler
Meeting with Mahdi
Explain the current errors we are having in regards to the learning rate scheduler
Update Mahdi on the trading algorithm status
Discuss the current results of what can be understood from the learning rate scheduling test results
Devise method to predict ask price based on bid price, as it would not be feasible to predict both bid and ask price in the scope of this project
Download bid and ask data of EURUSD that Ian found on Dukascopy
Continue working on the trading strategy
Modify approach to work sequentially rather than recursively
July 4th - No work
Finish the basic functionality of the trading algorithm
Investigate the spread value between the bid and ask values
Clear up a misconception about how the spread data affects profit and switch bid and ask price when buying and selling
Added a 'units per trade' functionality that I can set the units desired to make each trade
Add counters of how many trades exist after each step of the trading algorithm
Analyze trading algorithm results based on different parameters specified
Create some candidates for novel loss functions that improve on VRMSE to have horizontal asymptotes
Work on presentation for Monday
Include visualizations of trading strategy
Give presentation to REU members and professors
Investigate reinforcement learning in regards to a more advanced trading strategy
Imitation learning is a good candidate for the reinforcement learning algorithm architecture
Take covariance of all of the datasets
Meeting with Mahdi
Discus covariance results
We did not know how to interpret them well
Look into finding and analyzing the correlation results, which will show the relation between the datasets
Begin to adjust the trading code to accept data from Dukascopy
Meeting with Mahdi
Discuss correlation between all of the datasets
The datasets are not correlated enough to want to train the models to predict all datasets at once
Investigate the Time-Series Library's scaling methodology, ensuring it scales the train, validation, and test data correctly
Add timestamp data to the batches in the Time-Series Library, preparing to implement model inferencing
Many errors debugged. Timestamp data did not want to load properly
Make changes to the inference script to solve issues of it running with CPU only
Download 1 day's worth of Nasdaq data for fast inference testing
Investigate issue with data loss
It turns out to be the last batch getting dropped since the data does not nicely fit into a batch
Issues with prediction loss in true values
Tried several different ways of debugging
Graphed out step by step process of how the true values are generated
Had to implement an entirely new system that grabbed the true values from the original csv to fix this problem
Fix issue where inferencing worked incorrectly across shift to and from daylight savings time
Make two attempts at estimating the spread between the bid and ask data
The first attempt totaled the bid and ask values, and divided them by the number of respective values present, and then getting the difference between the two averages
This did not account for the differences in total number of bid and ask values though
The second attempt calculated the average spread between bid and ask values that existed in both datasets
This method provided a more accurate estimated spread
Meeting with Mahdi
Talk about our abstract and some changes we could make
Work on an algorithm that matches real bid values to predicted values
Ask values are optionally found if they are available, since they may not exist
Debug the algorithm, as it takes far too long to match the values
Work on methodology section of the poster for Mid-SURE
Work with Ian about how the diagram should look like
Meeting with Mahdi
Discussed setting the ask values to a fixed number offset from the bid values to save time
Talk about what content we will and will not need for our poster
Refine our methodology graph
Ian and I presented our current draft of the poster for Mid-SURE
We did not have a 'results' section yet, as our results have not been generated yet
We got a lot of useful feedback for the poster's layout and content
Devised a method of representing profits per day over time
Discuss implementing retraining processes to improve trading results as time gets further into the future if time permits
Refactor methodology diagram with Ian
Implement a flat spread between bid and ask data, based on the calculated average spread of the 2 years of the data from Dukascopy
Include an optimization for data matching of results and true values
I was slicing the array each iteration in hopes it would speed up the matching process, but it was in fact making it much slower. This was removed
Transfer over code from original trading strategy to a new notebook in order to refine it
Ensure new buy and selling rules for the trading strategy to closer reflect our use case of short-term trading
Specify open and closing times of the market for the trading strategy to operate within
Step through the entire trading strategy, fixing bugs that occur at edge cases
Generate sample data that crosses the open and closing market times
Test that the trading strategy operates properly when crossing the opening and closing boundaries
Separate the profit pairs generated by the trading strategy by day for daily evaluation
Run trading strategy as per Mahdi's request on the actual data to ensure that it works as intended
It does in fact generate profit if it knows the actual data's price at a given time as expected
Cooperate with Ian regarding the graphing for the poster for Mid-SURE
Work on papermill script to run the trading algorithm across all datasets
Fix issue of trader where spread offsets were applied incorrectly
We adjusted units per trade to equal 1 in order to not give preferential treatment to any of the datasets since we do not have a confidence metric
Work late with Ian to get the graph generation completed and included into the poster for Mid-SURE
Finalize the poster for Mid-SURE and turn it in to Dr. Fu for printing
Talk with Ian about last minute adaptations to the poster for Mid-SURE as the hard deadline is today
Correspond with Dr. Fu about these adjustments
Work on the research report with Ian
Focus on the trading algorithm configuration section
Create subsections that describe all of the parts included in the trading algorithm
Work on deliverables
Correspond with Dr. Fu about applying for a travel grant to receive funding for driving to Mid-SURE
Revise trading algorithm configuration section
Add information to the dataset and preprocessing section
Include information about data postprocessing
Continue writing and editing the report
Make many semantical specifications
Begin to work on potential conclusions we can make
Section out different areas of conclusions regarding performance of the models, loss functions, and volatility
Describe how VRMSE affects each of the models
Figure out how to justify the input and prediction lengths
Create pseudocode for the trading algorithm for the report
Attend the Mid-SURE poster presentation
Ian and I were evaluated by two evaluators
Both evaluators gave us very good evaluations
Ian and I presented our poster to several interested people that were attending the presentation
Debrief with other REU members about the experience and our performance
Attend several other students' poster presentations
Attend deliverable requirements meeting with Dr. Fu
Finish writing the report with Ian and revise what we have already written
Figure out how to answer why the Transformer's profit per trade better than the other models
Finish updating the blog post website
Check out deliverables with Mahdi to ensure they are completed
Finalize the format of the deliverables
Meet with Dr. Fu to ensure that our deliverables are completed
Reflect about the REU program with Ian