Last update: February 22, 2015

I am a junior researcher in computer vision. Probably, not everything here is correct...we learn by time :)

General

The Big picture

• Never forget the pig picture of your research: technical and non-technical
• If you build some disciplines (e.g. every day will have 30 min doing X) follow them.
• When experiments gives bad performance we put all the time and effort in the code
  • One bad thing about that, you forget the non-technical goals and objectives
  • You typically narrow your mind to little coding sub-components and the issue may be in another area. Even error may be a trivial missing line somewhere.
  • As juniors need more time in engineering, when you implement your idea and it gives bad results, we have 2 cases
    • The idea is wrong. It will never produce good work
    • It is nice. Just some bugs + tuning some parameters
      • Ask experienced guys/advisor for little help.
      • Try forums. Many variations exist on Quora, Linkedin, Researchgate.

Approach Implementation

• Try to start with a simplified version of approach. Code, test and evaluate.
• Try simple learning models. E.g. Linear SVM before NonLinear SVM
• Think about memory caching points and data storage points.
  • Sometimes we do the same processing in each trial.
  • Save as possible as you can if doable in time/memory/disk space.
• If your code is a pipeline of multiple stages, try to start from the tail if someone provides some head processing for first pipeline phases
  • The point: mistaking in a phase highly affect the whole performance and it is hard to detect problems.
  • Say you implement BOW. You need a dictionary and use it to quantize images. Finally, learn a classifier
  • If someone provided good quantized images, write the code that learns classification first.
  • If someone provided dictionary, start from quantizing and so on...shorter code per
  • This lets you have correct base and build over it

Coding

• Don’t reinvent the wheel.
• Think ahead of code components and interfaces.
• Separation of concerns. Divide code as much as possible. A shorter code per class is highly preferred
• Sub-folder the code
• It is fine to have code chaos at some points. In the nearest time (few
• days), back everything organized. Chaos is fine too during prototyping.
• When the code becomes larger, functional is not a good idea...Think OOP
  • Write modular code. Methods have shortcode.
  • Upload the code to the web, allowing licensees to let all the people share it
    • E.g. pull requests on GitHub
• Do as much as logical asserts, even if you think it is not needed
  • We change the code all the times, bugs are introduced
  • Assert save your debugging time.
    • E.g. Object is null...we forgot to initialize it in new changes!
• Code Refactoring
  • Setup a weekly check on the code and how to make it better
  • How to remove redundancy
  • How to make it more modular
  • How to minimize function length and reasonable methods names
• When ever finished coding some functions/classes:
  • Do code review
    • Once finished function class, think about cases and trace by eyes
    • Trace method lines by debugger
    • Next day morning review the code again
      • Always keep a copy of each day code, then every day you have last day version and the comparison reveals 1 day changes
      • May be use some SVN to do this
    • Do unit testing if possible
    • If possible, print / visualize outputs and make they them make sense
  • Nothing is worse than investigating the whole code for a trivial bug! The more careful you are, the less consumed time
• Always have mini-dataset to apply to make sure overall is fine. Don’t discover problems due to crash after X hours of running!
  • Have a nice size to use later as a guide for some estimations. E.g. size = 10, 100
• Documentation...Documentation...Documentation...Documentation...Documentation…
  • Write initially head notes about function target. Which variables are output
  • Later, use standard format to describe each function
  • Code body preferred to have some comments. Variables names expressive.

Logging

• Always log for every important function
• File logging is good to avoid machine restarts/crash
  • In C++, it is easy to orient all output toward file using freopen
• Print gentle logging in program head for the selected configuration
  • E.g. what pipeline to run
  • E.g. What are the important parameters values?
  • You may force a confirm (input) to proceed with data.
• Avoid logging too much..make it smart
  • E.g. if we process X images, each 10% give status
  • Let some algorithms log once, then stop logging
• Visualizing in Videos
  • Videos has many frames, it is good idea to write per frame the important info

Floating Point Precision

• Whenever saving precision numbers to file, write this with as many digits as possible (e.g. 3.5024614207824543e-05)
  • When the application is case sensitive to the tiny fractional part, it will affect performance
  • As example, Say you trained linear SVM, then compressed all support vectors in 1 vector to be used in O(k) to predict. All values will be so tiny. Be careful.

Time Complexity

• Don’t run an experiment and you don’t know the overall time for it!
• Let every main block do an estimate for the expected time to finish
  • E.g. sum first 5 times per step (e.g. for loop extract sift for image), use average to estimate. Simple class/struct will be nice.
  • Print average per step and overall expectation
• Use a mini-dataset to calculate manually whole pipeline of processing times

Memory Complexity

• Never write a code that you don’t estimate its memory
• Don’t wait program to crash due to memory concern
• Do balance between using hard disk and available memory
  • Matlab is the best when comes to saving many matrices
  • Avoid text data. Save binary
  • When saving data that could be post-processed in several ways, save it unprocessed, so that you don’t lose data
    • E.g. you have training data and could be normalized/scaled in several ways.
• If possible in language/platform, write function that estimate memory per important functions
• Check memory used by the process in the memory..process that keeps increasing in unexpected way is ALARM for you

Using new staff

• Whenever using new staff, specially learning components (train in SVM, k-means, GMM..):
  • Read Documentation carefully
  • Make sure to test them well independently
  • Try them with different parameters
  • Visualize them
  • Internal data
    • sometimes you want to use internal data not just direct interface
      • E.g. internal support vectors not just predict method
    • Be highly careful. Typically docs don’t go in such details
    • Don’t make assumptions
      • E.g. OpenCV multiply -1 in his internal data on opposite to libsvm (reverse for -1 and 1 class)

Generated Artifacts

• Always check your generated data. Sometimes they are weird and something wrong happened.
• Visualization is good key to find bugs/improvements
• Have a separate folder for each artifact.
• If take a copy from some outputs, write notepad what are them..How generated

Parameters tuning and Automation

• Parameters document
  • Whenever you think something might be a parameter that affects performance:
  • List all these variables and possible values ranges. Later, we tune them: E.g.
    • -ve examples generation: boxes, olis, hard negatives. Ratio: 1 => 5
    • Dict size = 500, 1000, 2000, 3000, 4000, 500.
      • Points to cluster: 50,000 -> 200,000
    • Image keypoints
      • SURF + dense (step: 6, 10, 20...) + randpoints (1000 -> 10000) + point scale (1, 6...)
    • Svm C / Gamma: 1e-4, 1e-3.....1 10 100 1000
    • Classifier: LinearSVM, RBF, ANN
  • Estimate initial good values
    • This is very challengeable.
    • Any initial bad combination could result in severe low performance
    • When read papers, keep your eyes on their parameter values
      • VOC dict size: 400, 1000, 3000, 4096
        • 1000 appear a lot
      • In some problems: C = 1, Gamma: 1 / FeaturesCount
      • E.g. paper filtered near boxes with 70
      • E.g. paper removed overlapped -ve box with 20 IoU
  • Consistency
    • If 2 blocks need same processing, better give same parameters
      • E.g. generating dictionary and quantizing image in BOVW extract keypoints from images
      • If selected rand points for dictionary and sift points for quantizing, this may behave bad
• Don’t change params in code to try. Have all needed parameters as variables.
• Manually changing algorithm parameters is a big waste of time. Tune by code as possible.
• Once determined your application set of parameters, write some greedy algorithm to automate them
  • This way you don’t lose time
  • Have better idea about algorithm performance
• we can’t search for every possible combination….think greedily
  • E.g. for C, search in range 1e-5, 1e-3…...100
  • Then, start for the best one to search deeper (e.g. 2e-3, 3e-3..)
  • Google grid search. See
• Tuning and accuracy variance
  • sometimes u r testing recall. u have 2 choices. both gave same recall.
  • However, on full set, they will give recall and one with good precision too and other note. E.g.
  • -ve generation method. one method could tell classifier better about -ve classes
  • How to handle? better to not have only ur dataset for recall images only. Have some other data for -ve data (e.g. same as +ve data)
• Randomization...be careful
  • Sometimes your data is based on randomization (e.g. generate -ve training examples)
  • Given fixed tuned parameters, results range from good-bad with this randomization.
  • Force any function that depends on randomization to generate SAME results under any sequence of calls to your pipeline
    • Best to use RNG rng(value) from Boost
    • Let each function has its RNG initialized based on sent parameters.
    • If it calls another function, allow setting, so that you force call to generate the same sequence
  • Watch out from algorithms that has randomization behavior (many approximation algorithms)
    • OpenCV - FlannBasedMatcher
    • STL - random_shuffle

Efficient Experimenting

Minor Datasets

• It is not good to run experiments on whole data
• Build 1 or minor dataset of your work. Typically, use randomization to extract some data. E.g. 10% of the dataset.
• Use always this dataset to inform you about the performance of your system

Recall before Precision

• Don’t run all ur code on whole dataset to judge together the 2 measures
• Typically, recall happens on subset of data where items exist
  • E.g. in object detection. Say validation set is 2500 image. Class aeroplane exist in only 200 images
  • Recall care much with these 200 images only. Precision care with all.
  • If you can’t get good recall on 200 images initially, you don’t have algorithm
  • Work on 200 till you have good performance
  • Use Recall images for initial tuning for the parameters
  • Now, move to precision...use F-Score as a united measure. ROC curves.
    • Don’t again run on the whole. Pick 2 classes only and test on them.

Accuracy of the whole training before cross validation

• We might have a model, and we want to apply CV to get best parameters based on average accuracy
• However, this doesn’t make sense if training, then prediction on whole training data results in low performance.

Experiments Results

• Be organized as possible.
• When running an experiment, document its results. Point to locations of its generated artifacts.
  • Copy program arguments with the program to indicate your choices.