I have both a bachelor’s degree and a master’s degree in statistics, so I know what I’m about to say will not be a popular opinion among my academic peers. But 17+ years of experience in the real world has led me to the not-so-simple conclusion that…
FORECASTING ISN’T JUST ABOUT THE NUMBERS.
As statisticians, we graduate from college wanting to produce the most accurate forecasts possible. There are rules and guidelines that are drilled into our brains for four (or more) years. I’m not knocking this training or the rigor that should be applied to forecasting. Plus, someone who lands a job in a highly technical research lab or as an actuary probably needs to adhere to almost all of those rules and regulations. But what I quickly learned in my job is that the real world isn’t black and white. I’m guessing a lot of other college graduates have also found this out after leaving the structure of their classrooms.
Simply put - the numbers don’t always tell the full story. I have found this to be especially true when it comes to forecasting caseload and personnel for courts, which I spend a lot of time doing each year. The historical data may say one thing, but there are numerous other factors at play, such as policies, demographics, and location-specific nuances.
So if we can’t rely on numbers alone, what’s the answer? In my experience, the best forecasts include a blend of both quantitative and qualitative input. That’s why I firmly believe that forecasting can’t be solely a statistical process. Because there is no such thing as an “accurate” forecast, we are trying to come up with the most reliable and valid forecasts possible. What better way to do this than to factor in all relevant data and assumptions, even those that can’t so easily be plugged into a model?
Quantitative Meets Qualitative – A Marriage Made in Heaven
I have produced tens of thousands of caseload and personnel forecasts each year for the past 17+ years. I start with the statistical side, and then a team of reviewers comes in to provide qualitative input. To produce reasonable, defensible forecasts, it is important to start with the numbers as the solid foundation, then layer on some qualitative review. Read on for a summary of how this works.
Start with Statistical Forecasts
The process always begins with a statistical approach. Actually, the process always begins with multiple statistical approaches. Let the numbers lead the way in the beginning. Take the historical data and use whichever forecasting methods fit the data to come up with a set of initial forecasts. These initial forecasts are based purely on the data, rules, and mathematical black and white knowledge you can glean from the numbers. But, don’t stop there!
Add Qualitative Knowledge
Despite having many years of experience with my company, some of my colleagues have even more experience and work directly with the courts while I’m working with the data. These colleagues travel around the country to hear directly from court representatives on the trends going on in their district and region. Then they analyze these trends from multiple angles. Why would I not tap into this amazing source of knowledge when producing forecasts?
You may wonder how subject matter expertise feeds into the statistical forecasting process. Well, after I produce the initial statistical forecasts, this team of seasoned reviewers looks at the statistical options and provides feedback. If there is an initial forecast among the many options that seems reasonable, then great! If not, the reviewers provide me with other data, trends, or observations I can use to help adjust the initial forecasts. For instance, knowledge of local demographics, policies, or budgets might support treating various data points as outliers, while the numbers alone might not have led me to that conclusion.
The reviewers may also have information about an upcoming legislative initiative that is likely to stall or boost caseload. This qualitative input often helps us choose what we consider the best forecast among multiple forecast options, or helps us tweak the parameters within the model (e.g., adjust which locations to use in cluster analysis, which years of historical data to use, etc.).
I realize that not everyone has highly experienced colleagues at their fingertips, but there are many reputable sources available. Seek out subject matter experts and information that can help inform your forecasts.
Bottom Line: Numbers Don’t Tell the Whole Story
People often say that statisticians can make the numbers say anything. To a certain extent, I’d say that’s true. You can slice and dice the numbers to your liking, and politicians, corporations, etc., are certainly known to do so. But when trying to plan for long-range space needs, we aren’t looking to fiddle with numbers in order to paint a certain picture. We are trying to provide the best prediction possible about what may happen in the future. Important decisions rest on this. Do we want numbers alone to tell the whole story? I know I don’t.
No one is ever going to predict the future with 100% certainty. But here’s my prediction: if you make a concerted effort to blend quantitative methods with the best subject matter expertise you can find, you’ll get a whole lot closer to a realistic vision of the future. And while not perfect, I’d say that’s a marriage made in heaven.
