One of the things about my analysis that I often think about is the difference in value between my perception of the trenches and the wider NFL community at large. The approach that I use for valuation currently is derived from PFF and their Wins Above Replacement model. While theirs is definitely more robust than mine (and sometimes agrees more with the consensus than my home brew version), I generally agree with my numbers.
My model is using the team grading in different facets and comparing that to the teams pythagorean wins in a linear regression model. Then I take each player's grading an scale them accordingly using these coefficients as weights for each facet of grading measured in the units of wins. I do a log based scaling to account for the nature of the PFF grading system not being linear with 60 being the average or replacement level player and additionally scale based on the number of snaps that a player has. The end product is a WAR metric that is defined by how far above or below 60 a player gets on the PFF grading scale in the most valuable elements of the game that also accounts for how often you were playing at that level. It is much easier to grade above 90 on 15 snaps than it is on 500 snaps, so the WAR metric should scale with snaps played.
The issues I have with my current analysis is that the overall team dynamics are not being factored in (a 90 grade across 500 snaps is equivalent regardless of situation), there are some conversion issues between the team grading and the player grading, and finally that my model values the passing game above all else. While I agree with the value of the passing game, I think that there are edge cases (Aaron Donald, Myles Garrett, etc.) that are valuable enough that I would actually rather have those players than a second tier WR who technically has a higher WAR because of the position that they play.
This is why I tried creating Scarcity Adjusted WAR. Ideally, I wanted a way to adjust my grading to give me a true "Value" of a player based on how impressive it was for them to grade how they are grading. Sure, currently it is constructed so that it is measured in the base units of wins and already accounts for their value relative to the average or replacement level player, so in theory you should just take the highest graded players as better. However, for the aforementioned edge cases, it is much harder to find a Myles Garrett than a Gabe Davis so I know who I would rather have on my team WAR be damned. Additionally, I wanted to account for the fact that not every position is as stable in terms of grading. I added in a feature for scaling based on the yearly stability of your position with regards to grading.
Scarcity Adjusted WAR takes the top 5 players at your position and uses their average WAR to create a benchmark for how good the top of the position is (I played around with numbers other than 5 as well, but this usually shows the top end of the position). Given that top end benchmark, I computed the player's WAR percentile and did zscores to see how impressive they were compared to the rest of their position. I found that while it did do most of what I set out to do (and it closed some of the gap between my numbers and the consensus valuations) I still wasn't that happy with the results. I now had a way of seeing which players were the furthest away from the mean of their position and that this improved the feeling of which players were special and more scarce, but I wasn't sure the best way to adjust my numbers off of it. Additionally, you ended up having players who were good at historically poorly graded positions show up really highly. For example, not a lot of DI or TE grade super highly, so from a zscore perspective are going to show up almost too highly just because they were so far above their positional average. Just because you are the best of a weak position group, doesn't mean you are actually that good globally. I feel like scarcity is something that should come in when you are dealing with team building strategy, not in valuation. Additionally, I feel like there would just be better approaches (such as my future project for yards based value, or expected contracts) for defining value that wouldn't require adjusted my home brew version of PFF WAR this much to better match the consensus. Maybe I will come back to this idea in the future.