Address the Little Stuff

Address the Little Stuff

Everything is intertwined in an analysis problem like this. It is very common that one field that appears to be in error is not in fact wrong, but some other error is making this one look wrong. Thus it is important to always have both the big and little picture in mind at the same time, if possible.

Clearing up some of the little problems early may help solve other problems by:

• Removing other error messages that are only there because something else was out of place.

• Reducing the number of error messages to only the ones about the "big stuff" that you are trying to solve.

Here are several things you can do; the order they are done in will vary depending on your data, and you will probably jump around from one to another, or revisit some of them several times.

• Look at the error messages for other fields (besides ps/sn/se).

• Start with the ones with the most occurrences. Is there something obvious going on? Can you fix something easily to make one big block or messages go away? [TO DO: add examples]

• Start with the ones with the fewest occurrences. Sometimes these are easy to address.

• Pick one error message, and then step through a lot of the entries in that filter, and see if there is a general pattern for why those are failing. If there is one thing you can do that will fix several of them, do that first. Then come back and address the more diverse problems.

• Keep working until the only error messages left are the ones you put off working on "until the rest of the data is cleaner".