May 28, 2014

The Efficient Analyst (part one): Stop moving your data around!

Many organizations collaborate by moving data and information between people, files or applications.  While movement may provide analysts with the illusion of adding value, data movement is not the same thing as data sharing. While the latter is a good thing to be nurtured, the former causes version control problems, wasted time, and analyst frustration.

In our original post on why data analysts are so inefficient, we covered the six core causes of data waste. In today's post we'll tackle the first cause: conveyance.  Below are five ways to reduce data movement and put the focus back on sharing.

1. Stop emailing (IMing, Skyping) files back and forth. Email and chat programs are some of the most popular ways to move data around. While we are all guilty of doing this every now and then, it creates huge access and version control problems for your team. Stop doing it. The most effective alternative is to utilize your company's sharing environment to store and update files. Don't have a sharing environment? Go to #4.

2. Eliminate "staging" files or sheets. A good general rule is: do not put data into a spreadsheet, database or file that does not add specific value. Many use staging files when restructuring or reformatting data before being used by someone or something else. Eliminating these types of files not only eliminates waste and confusion, but also ensures faster assembly of process documentation.

3. Be careful with "movement automation". This category can include FTP syncs, VBA macros or any other kind of automated process that pulls data from one location into another. FTP syncs often create timing bottlenecks (to be discussed in a future post) or create confusion among users. VBA macro code in Excel can cause chaos when there are structural changes in either the source or destination spreadsheets. In general, any kind of data automation must be handled with a high level of care and used only when the risk outweighs the amount of effort it takes to manually execute the task.

4. Create a sharing environment. There are many products for sharing data across organizations. Some upload files to share drives or use Sharepoint or Dropbox, while others will build custom SQL databases to store raw data. Some use Shooju. Whatever option you choose, ensuring that it is a complete solution is critical to creating any form of internal data harmony. Because of the confusion they can cause without full adoption, partial solutions can be worse than no solution at all.

5. Google it. Some organizations have no love lost for the cloud, while others are rapidly making the transition. Wherever your company is positioned, Google Drive can be an excellent solution to specific sharing problems. While seasoned data analysts may scoff at this idea, there are great ways to track changes and store raw data sets that streamline shared data flow.

A word about sharing across applications

The suggestions in this post are most effective when moving data between individuals using the same application - i.e. moving an Excel file to another Excel user. Sharing data across applications is much more difficult. In many organizations, analysts that use more advanced analytical tools like SAS often have to export results into Excel spreadsheets (creating a "staging problem") and email them (gasp!) to colleagues who are non-SAS users. Additionally, keeping this staging files updated can be a big pain.

Readers: what tips do you have to solve data sharing across applications? Leave them in the comments below.


  1. I am grateful for touching data breach topic. As the staff member of one of the best online data room providers I want to emphasize the problem of the private account invasion, and also I underline that it should be given more attention to that issue by the law makers.

    1. Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. IEEE Projects for CSE in Big Data But it’s not the amount of data that’s important. Final Year Project Centers in Chennai It’s what organizations do with the data that matters. Big data can be analyzed for insights that lead to better decisions and strategic business moves.

      Spring Framework has already made serious inroads as an integrated technology stack for building user-facing applications. Corporate TRaining Spring Framework the authors explore the idea of using Java in Big Data platforms.
      Specifically, Spring Framework provides various tasks are geared around preparing data for further analysis and visualization. Spring Training in Chennai

  2. When the day dawns and you have to move your belongings from one place to another the first thing you have to do is ask yourself just how to get your stuff from your present location to your destination. Secondly, you have to find a good moving company. Movers Dallas

  3. Overseeing of the huge information isn't less demanding then you think it takes a few methods and traps for the overseeing of huge type of unthinkable information, so in the event that you don't have the foggiest idea about these systems and trap then you ought to need to visit this website the original source Active Wizards for getting the best information researcher for you who will oversee you enormous information for you.

  4. This blog is a punchy bit of composing, as it has a solid impact.
    Removals service

  5. Pretty amazing work on the author's part and This blog is a five star work.
    man with van services

  6. Imagine yesterday went to some online casino practicable online casino gambling sites and decided to ponastalgirovat. so many pleasant memories when you again see this slot machine mmm cool

  7. Glad to be a part of your blog I have find some informative info here. Last month I Moved my home to another city and (Safe Relocation) helped me. They are the best, Highly recommended!!

  8. I have read all the comments and suggestions posted by the visitors for this article are very fine,We will wait for your next article so only.Thanks! cheap moving companies denver

  9. Within this webpage, you'll see the page, you need to understand this data. Moving companies near me

  10. Extraordinary interest by female and transsexual customers and all the more as of late, because of society's mentalities, the quantity of male customers is expanding.

  11. In this time your only wish is to settle things as quickly as possible because right now you and your family are surrounded by a lot of mess.

    Removals Rugby