Jul 7, 2014

The Efficient Analyst (part four): The over-processing epidemic

When we come across companies that need help with complex data workflows, the most prominent problem we see is over-processing. What do we mean by over-processing? Data goes through superfluous steps whose value is not seen by the end user.

Over processing can take many different forms:
  • Intermediate files or sheets are created to "stage" data. Maybe an output from one model is rearranged in a unique spreadsheet before it is loaded into another model.
  • Multiple human beings need to touch or analyze the data, and each person has their own structure or way of doing things.
  • Legacy spreadsheets that are complicated and intertwined, with a deep rooted fear of "rocking the boat".
Why is over-processing a problem? By increasing the number of nodes in a work process, things take longer, results are harder to audit and the risk of error increases dramatically.

Like many workflow problems, over-processing occurs most frequently during periods of tight deadlines or heavy work requirements. "Band-Aid" solutions are much more common, and reaching the deadline becomes more important than the manner in which it is reached.  Then, those processes and steps become ingrained in the process; and part of the future iterations. 

Streamline your workflow

After these heavy work periods are over, there is enormous value in evaluating workflows that are likely to be used again. By streamlining them into efficient, easy to understand processes, the value of the work can be raised hugely in the long term. Here are a few steps to follow to try and reduce instances of over-processing.

1. Evaluate the process. Take time to chart out each node in a particular process so that it's clear how it works in the current state. Share this workflow document with anyone who is responsible for touching the data on its way to the end-user. Ensure that each step is laid out. Here is a breakdown of a hypothetical workflow:

2. List out the demands of the end-user or customer. What is the purpose of your process? In order to fully assess those steps that are unnecessary, your team must have a full breakdown of what each step does to add value to the final product.

3. Determine which steps can be consolidated, automated or eliminated.  Revise your workflow so that it looks something like the chart below. It's okay to dream big. Indicate processes that you want to automate before you know how - then seek a solution that can work.

4. Implement and iterate. Test your changes. Revise as needed.


  1. It is perfect time to make some plans for the future and it is time to be happy. I’ve read this post and if I could I desire to suggest you few interesting things or tips. Perhaps you could write next articles referring to this article. I want to read more things about it!

    PMP Course in Malaysia

  2. I think about it is most required for making more on this get engaged
    data science certification

  3. Cool you write, the information is very good and interesting, I'll give you a link to my site.
    data science course

  4. You actually make it look so easy with your performance but I find this matter to be actually something which I think I would never comprehend. It seems too complicated and extremely broad for me. I'm looking forward for your next post, I’ll try to get the hang of it!

    PMP Certification

    PMP Course
    PMP Course in Malaysia
    PMP Training in Malaysia
    PMP Training

  5. I am overwhelmed by your post with such a nice topic. Usually I visit your blogs and get updated through the information you include but today’s blog would be the most appreciable. Well done!data science course

  6. Share great information about your blog , Blog really helpful for us . We read your blog , share most useful information in blog . Thanks for share your blog here . Miami Private Investigation

  7. Kardinal Stick Siam - relx a great promotion. Express delivery in 3 hours.

    ufa football betting, casino, slots, lottery, direct website 1688, stable financial, 100% UFABET168.

    Online Baccarat FOXZ24 Easy to apply, fast, บาคาร่า deposit-withdraw 10 seconds with the system.

    Watch movies online sa-movie.com, watch new movies, series Netflix HD 4K ดูหนังออนไลน์, watch free movies on your mobile phone, Tablet, watch movies on the web.

    SEE4K Watch movies, watch movies, free series, load without interruption, sharp images in HD FullHD 4k, ดูหนังใหม่ all matters, all tastes, see anywhere, anytime, on mobile phones, tablets, computers.

    GangManga read manga, read manga, read manga online for free, fast loading, clear images in HD quality, อ่านการ์ตูน all titles, anywhere, anytime, on mobile, tablet, computer.

    Watch live football ผลบอลสด, watch football online, link to watch live football, watch football for free.

  8. Fascinating blog! Is your theme custom made or did you download it from somewhere?A theme like yours with a few simple tweeks would really make my blog jump out. Please let me know where you got your design. With thanks
    vé máy bay từ mỹ về việt nam hãng eva

    gia ve may bay vietjet tu han quoc ve viet nam

    Cách săn vé máy bay giá rẻ tu Nhat Ban ve Viet Nam

    giá vé máy bay từ singapore về hà nội

    Ve may bay Vietjet tu Dai Loan ve Viet Nam

    chuyến bay giải cứu Canada 2021

  9. I like this blog so much this is very nice site blog its too good.
    sony vegas torrent