Coding and Engineering Experiments
Last update: February 22, 2015
I am a junior researcher in computer vision. Probably, not everything here is correct...we learn by time :)
General
General
The Big picture
The Big picture
- Never forget the pig picture of your research: technical and non-technical
- If you build some disciplines (e.g. every day will have 30 min doing X) follow them.
- When experiments gives bad performance we put all the time and effort in the code
- One bad thing about that, you forget the non-technical goals and objectives
- You typically narrow your mind to little coding sub-components and the issue may be in another area. Even error may be a trivial missing line somewhere.
- As juniors need more time in engineering, when you implement your idea and it gives bad results, we have 2 cases
- The idea is wrong. It will never produce good work
- It is nice. Just some bugs + tuning some parameters
- Ask experienced guys/advisor for little help.
- Try forums. Many variations exist on Quora, Linkedin, Researchgate.
Approach Implementation
Approach Implementation
- Try to start with a simplified version of approach. Code, test and evaluate.
- Try simple learning models. E.g. Linear SVM before NonLinear SVM
- Think about memory caching points and data storage points.
- Sometimes we do the same processing in each trial.
- Save as possible as you can if doable in time/memory/disk space.
- If your code is a pipeline of multiple stages, try to start from the tail if someone provides some head processing for first pipeline phases
- The point: mistaking in a phase highly affect the whole performance and it is hard to detect problems.
- Say you implement BOW. You need a dictionary and use it to quantize images. Finally, learn a classifier
- If someone provided good quantized images, write the code that learns classification first.
- If someone provided dictionary, start from quantizing and so on...shorter code per
- This lets you have correct base and build over it
Coding
Coding
- Don’t reinvent the wheel.
- Think ahead of code components and interfaces.
- Separation of concerns. Divide code as much as possible. A shorter code per class is highly preferred
- Sub-folder the code
- It is fine to have code chaos at some points. In the nearest time (few
- days), back everything organized. Chaos is fine too during prototyping.
- When the code becomes larger, functional is not a good idea...Think OOP
- Write modular code. Methods have shortcode.
- Upload the code to the web, allowing licensees to let all the people share it
- E.g. pull requests on GitHub
- Do as much as logical asserts, even if you think it is not needed
- We change the code all the times, bugs are introduced
- Assert save your debugging time.
- E.g. Object is null...we forgot to initialize it in new changes!
- Code Refactoring
- Setup a weekly check on the code and how to make it better
- How to remove redundancy
- How to make it more modular
- How to minimize function length and reasonable methods names
- When ever finished coding some functions/classes:
- Do code review
- Once finished function class, think about cases and trace by eyes
- Trace method lines by debugger
- Next day morning review the code again
- Always keep a copy of each day code, then every day you have last day version and the comparison reveals 1 day changes
- May be use some SVN to do this
- Do unit testing if possible
- If possible, print / visualize outputs and make they them make sense
- Nothing is worse than investigating the whole code for a trivial bug! The more careful you are, the less consumed time
- Do code review
- Always have mini-dataset to apply to make sure overall is fine. Don’t discover problems due to crash after X hours of running!
- Have a nice size to use later as a guide for some estimations. E.g. size = 10, 100
- Documentation...Documentation...Documentation...Documentation...Documentation…
- Write initially head notes about function target. Which variables are output
- Later, use standard format to describe each function
- Code body preferred to have some comments. Variables names expressive.
Logging
Logging
- Always log for every important function
- File logging is good to avoid machine restarts/crash
- In C++, it is easy to orient all output toward file using freopen
- Print gentle logging in program head for the selected configuration
- E.g. what pipeline to run
- E.g. What are the important parameters values?
- You may force a confirm (input) to proceed with data.
- Avoid logging too much..make it smart
- E.g. if we process X images, each 10% give status
- Let some algorithms log once, then stop logging
- Visualizing in Videos
- Videos has many frames, it is good idea to write per frame the important info
Floating Point Precision
Floating Point Precision
- Whenever saving precision numbers to file, write this with as many digits as possible (e.g. 3.5024614207824543e-05)
- When the application is case sensitive to the tiny fractional part, it will affect performance
- As example, Say you trained linear SVM, then compressed all support vectors in 1 vector to be used in O(k) to predict. All values will be so tiny. Be careful.
Time Complexity
Time Complexity
- Don’t run an experiment and you don’t know the overall time for it!
- Let every main block do an estimate for the expected time to finish
- E.g. sum first 5 times per step (e.g. for loop extract sift for image), use average to estimate. Simple class/struct will be nice.
- Print average per step and overall expectation
- Use a mini-dataset to calculate manually whole pipeline of processing times
Memory Complexity
Memory Complexity
- Never write a code that you don’t estimate its memory
- Don’t wait program to crash due to memory concern
- Do balance between using hard disk and available memory
- Matlab is the best when comes to saving many matrices
- Avoid text data. Save binary
- When saving data that could be post-processed in several ways, save it unprocessed, so that you don’t lose data
- E.g. you have training data and could be normalized/scaled in several ways.
- If possible in language/platform, write function that estimate memory per important functions
- Check memory used by the process in the memory..process that keeps increasing in unexpected way is ALARM for you
Using new staff
Using new staff
- Whenever using new staff, specially learning components (train in SVM, k-means, GMM..):
- Read Documentation carefully
- Make sure to test them well independently
- Try them with different parameters
- Visualize them
- Internal data
- sometimes you want to use internal data not just direct interface
- E.g. internal support vectors not just predict method
- Be highly careful. Typically docs don’t go in such details
- Don’t make assumptions
- E.g. OpenCV multiply -1 in his internal data on opposite to libsvm (reverse for -1 and 1 class)
- sometimes you want to use internal data not just direct interface
Generated Artifacts
Generated Artifacts
- Always check your generated data. Sometimes they are weird and something wrong happened.
- Visualization is good key to find bugs/improvements
- Have a separate folder for each artifact.
- If take a copy from some outputs, write notepad what are them..How generated
Parameters tuning and Automation
Parameters tuning and Automation
- Parameters document
- Whenever you think something might be a parameter that affects performance:
- List all these variables and possible values ranges. Later, we tune them: E.g.
- -ve examples generation: boxes, olis, hard negatives. Ratio: 1 => 5
- Dict size = 500, 1000, 2000, 3000, 4000, 500.
- Points to cluster: 50,000 -> 200,000
- Image keypoints
- SURF + dense (step: 6, 10, 20...) + randpoints (1000 -> 10000) + point scale (1, 6...)
- Svm C / Gamma: 1e-4, 1e-3.....1 10 100 1000
- Classifier: LinearSVM, RBF, ANN
- Estimate initial good values
- This is very challengeable.
- Any initial bad combination could result in severe low performance
- When read papers, keep your eyes on their parameter values
- VOC dict size: 400, 1000, 3000, 4096
- 1000 appear a lot
- In some problems: C = 1, Gamma: 1 / FeaturesCount
- E.g. paper filtered near boxes with 70
- E.g. paper removed overlapped -ve box with 20 IoU
- VOC dict size: 400, 1000, 3000, 4096
- Consistency
- If 2 blocks need same processing, better give same parameters
- E.g. generating dictionary and quantizing image in BOVW extract keypoints from images
- If selected rand points for dictionary and sift points for quantizing, this may behave bad
- If 2 blocks need same processing, better give same parameters
- Don’t change params in code to try. Have all needed parameters as variables.
- Manually changing algorithm parameters is a big waste of time. Tune by code as possible.
- Once determined your application set of parameters, write some greedy algorithm to automate them
- This way you don’t lose time
- Have better idea about algorithm performance
- we can’t search for every possible combination….think greedily
- E.g. for C, search in range 1e-5, 1e-3…...100
- Then, start for the best one to search deeper (e.g. 2e-3, 3e-3..)
- Google grid search. See
- Tuning and accuracy variance
- sometimes u r testing recall. u have 2 choices. both gave same recall.
- However, on full set, they will give recall and one with good precision too and other note. E.g.
- -ve generation method. one method could tell classifier better about -ve classes
- How to handle? better to not have only ur dataset for recall images only. Have some other data for -ve data (e.g. same as +ve data)
- Randomization...be careful
- Sometimes your data is based on randomization (e.g. generate -ve training examples)
- Given fixed tuned parameters, results range from good-bad with this randomization.
- Force any function that depends on randomization to generate SAME results under any sequence of calls to your pipeline
- Best to use RNG rng(value) from Boost
- Let each function has its RNG initialized based on sent parameters.
- If it calls another function, allow setting, so that you force call to generate the same sequence
- Watch out from algorithms that has randomization behavior (many approximation algorithms)
- OpenCV - FlannBasedMatcher
- STL - random_shuffle
Efficient Experimenting
Efficient Experimenting
Minor Datasets
Minor Datasets
- It is not good to run experiments on whole data
- Build 1 or minor dataset of your work. Typically, use randomization to extract some data. E.g. 10% of the dataset.
- Use always this dataset to inform you about the performance of your system
Recall before Precision
Recall before Precision
- Don’t run all ur code on whole dataset to judge together the 2 measures
- Typically, recall happens on subset of data where items exist
- E.g. in object detection. Say validation set is 2500 image. Class aeroplane exist in only 200 images
- Recall care much with these 200 images only. Precision care with all.
- If you can’t get good recall on 200 images initially, you don’t have algorithm
- Work on 200 till you have good performance
- Use Recall images for initial tuning for the parameters
- Now, move to precision...use F-Score as a united measure. ROC curves.
- Don’t again run on the whole. Pick 2 classes only and test on them.
Accuracy of the whole training before cross validation
Accuracy of the whole training before cross validation
- We might have a model, and we want to apply CV to get best parameters based on average accuracy
- However, this doesn’t make sense if training, then prediction on whole training data results in low performance.
Experiments Results
Experiments Results
- Be organized as possible.
- When running an experiment, document its results. Point to locations of its generated artifacts.
- Copy program arguments with the program to indicate your choices.