Jul 7, 2014

The Efficient Analyst (part four): The over-processing epidemic


When we come across companies that need help with complex data workflows, the most prominent problem we see is over-processing. What do we mean by over-processing? Data goes through superfluous steps whose value is not seen by the end user.

Over processing can take many different forms:
  • Intermediate files or sheets are created to "stage" data. Maybe an output from one model is rearranged in a unique spreadsheet before it is loaded into another model.
  • Multiple human beings need to touch or analyze the data, and each person has their own structure or way of doing things.
  • Legacy spreadsheets that are complicated and intertwined, with a deep rooted fear of "rocking the boat".
Why is over-processing a problem? By increasing the number of nodes in a work process, things take longer, results are harder to audit and the risk of error increases dramatically.

Like many workflow problems, over-processing occurs most frequently during periods of tight deadlines or heavy work requirements. "Band-Aid" solutions are much more common, and reaching the deadline becomes more important than the manner in which it is reached.  Then, those processes and steps become ingrained in the process; and part of the future iterations. 

Streamline your workflow

After these heavy work periods are over, there is enormous value in evaluating workflows that are likely to be used again. By streamlining them into efficient, easy to understand processes, the value of the work can be raised hugely in the long term. Here are a few steps to follow to try and reduce instances of over-processing.

1. Evaluate the process. Take time to chart out each node in a particular process so that it's clear how it works in the current state. Share this workflow document with anyone who is responsible for touching the data on its way to the end-user. Ensure that each step is laid out. Here is a breakdown of a hypothetical workflow:




2. List out the demands of the end-user or customer. What is the purpose of your process? In order to fully assess those steps that are unnecessary, your team must have a full breakdown of what each step does to add value to the final product.

3. Determine which steps can be consolidated, automated or eliminated.  Revise your workflow so that it looks something like the chart below. It's okay to dream big. Indicate processes that you want to automate before you know how - then seek a solution that can work.


4. Implement and iterate. Test your changes. Revise as needed.

20 comments :

  1. Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. IEEE Projects for CSE in Big Data But it’s not the amount of data that’s important. Final Year Project Centers in Chennai It’s what organizations do with the data that matters. Big data can be analyzed for insights that lead to better decisions and strategic business moves.

    Spring Framework has already made serious inroads as an integrated technology stack for building user-facing applications. Corporate TRaining Spring Framework the authors explore the idea of using Java in Big Data platforms.
    Specifically, Spring Framework provides various tasks are geared around preparing data for further analysis and visualization. Spring Training in Chennai

    ReplyDelete

  2. It is perfect time to make some plans for the future and it is time to be happy. I’ve read this post and if I could I desire to suggest you few interesting things or tips. Perhaps you could write next articles referring to this article. I want to read more things about it!

    PMP Course in Malaysia

    ReplyDelete
  3. I am impressed by the information that you have on this blog. It shows how well you understand this subject.
    data analytics course
    data science course
    big data course
    big data course
    360DigiTMG

    ReplyDelete
  4. I think about it is most required for making more on this get engaged
    data science certification

    ReplyDelete
  5. It is perfect time to make some plans for the future and it is time to be happy. I’ve read this post and if I could I desire to suggest you few interesting things or tips. Perhaps you could write next articles referring to this article. I want to read more things about it!
    360DigiTMG PMP Certification
    360DigiTMG PMP Certification in Malaysia
    360DigiTMG PMP Course
    360DigiTMG PMP Course in Malaysia

    ReplyDelete
  6. I am impressed by the information that you have on this blog. It shows how well you understand this subject.
    data analytics course
    big data analytics malaysia
    big data course

    ReplyDelete
  7. Great Article it its really informative and innovative keep us posted with new updates. its was really valuable. thanks a lot.
    data science course

    ReplyDelete
  8. This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea.
    360DigiTMG PMP Certification
    360DigiTMG PMP Certification in Malaysia
    360DigiTMG PMP Course in Malaysia
    360DigiTMG PMP Course
    360DigiTMG PMP Training in Malaysia
    360DigiTMG PMP Training

    ReplyDelete
  9. I am impressed by the information that you have on this blog. It shows how well you understand this subject.
    data analytics course
    big data analytics malaysia
    big data course

    ReplyDelete
  10. Cool you write, the information is very good and interesting, I'll give you a link to my site.
    data science course

    ReplyDelete
  11. You actually make it look so easy with your performance but I find this matter to be actually something which I think I would never comprehend. It seems too complicated and extremely broad for me. I'm looking forward for your next post, I’ll try to get the hang of it!

    PMP Certification

    PMP Course
    PMP Course in Malaysia
    PMP Training in Malaysia
    PMP Training

    ReplyDelete
  12. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!
    data analytics course
    big data analytics malaysia
    big data course

    ReplyDelete
  13. I am overwhelmed by your post with such a nice topic. Usually I visit your blogs and get updated through the information you include but today’s blog would be the most appreciable. Well done!data science course

    ReplyDelete
  14. Here at this site really the fastidious material collection so that everybody can enjoy a lot.

    Data Science Course

    ReplyDelete
  15. Your work is very good and I appreciate you and hopping for some more informative posts.

    Data Science Training

    ReplyDelete
  16. Share great information about your blog , Blog really helpful for us . We read your blog , share most useful information in blog . Thanks for share your blog here . Miami Private Investigation

    ReplyDelete
  17. Terrific post thoroughly enjoyed reading the blog and more over found to be the tremendous one. In fact, educating the participants with it's amazing content. Hope you share the similar content consecutively.

    artificial intelligence training in bhilai

    ReplyDelete
  18. Impressive blog to be honest definitely this post will inspire many more upcoming aspirants. Eventually, this makes the participants to experience and innovate themselves through knowledge wise by visiting this kind of a blog. Once again excellent job keep inspiring with your cool stuff.

    Data Science certification in Bhilai

    ReplyDelete
  19. Extraordinary blog went amazed with the content that they have developed in a very descriptive manner. This type of content surely ensures the participants to explore themselves. Hope you deliver the same near the future as well. Gratitude to the blogger for the efforts.

    Digital Marketing training in Bhilai

    ReplyDelete