The Efficient Analyst (part four): The over-processing epidemic

When we come across companies that need help with complex data workflows, the most prominent problem we see is over-processing. What do we mean by over-processing? Data goes through superfluous steps whose value is not seen by the end user.

Over processing can take many different forms:

• Intermediate files or sheets are created to "stage" data. Maybe an output from one model is rearranged in a unique spreadsheet before it is loaded into another model.

• Multiple human beings need to touch or analyze the data, and each person has their own structure or way of doing things.

• Legacy spreadsheets that are complicated and intertwined, with a deep rooted fear of "rocking the boat".

Why is over-processing a problem? By increasing the number of nodes in a work process, things take longer, results are harder to audit and the risk of error increases dramatically.

Like many workflow problems, over-processing occurs most frequently during periods of tight deadlines or heavy work requirements. "Band-Aid" solutions are much more common, and reaching the deadline becomes more important than the manner in which it is reached.  Then, those processes and steps become ingrained in the process; and part of the future iterations. 

Streamline your workflow

After these heavy work periods are over, there is enormous value in evaluating workflows that are likely to be used again. By streamlining them into efficient, easy to understand processes, the value of the work can be raised hugely in the long term. Here are a few steps to follow to try and reduce instances of over-processing.

1. Evaluate the process. Take time to chart out each node in a particular process so that it's clear how it works in the current state. Share this workflow document with anyone who is responsible for touching the data on its way to the end-user. Ensure that each step is laid out. Here is a breakdown of a hypothetical workflow:

2. List out the demands of the end-user or customer. What is the purpose of your process? In order to fully assess those steps that are unnecessary, your team must have a full breakdown of what each step does to add value to the final product.

3. Determine which steps can be consolidated, automated or eliminated.  Revise your workflow so that it looks something like the chart below. It's okay to dream big. Indicate processes that you want to automate before you know how - then seek a solution that can work.

4. Implement and iterate. Test your changes. Revise as needed.