10 steps to speed the journey from data prep to insight
Connecting state and local government leaders
Organizations can streamline the data analytics process with data preparation tools and processes, including self-service data prep.
Preparing data for analysis is a slow, difficult, tedious process that can cause major delays and inefficiencies when it comes time for analysis. So improving the methods and technology to process data can greatly improve and speed up the time to insight.
Half of the respondents to a recent TDWI Research survey spent more time cleaning the data than using it. Most (73 percent) spent at least 41 percent of their total time preparing the data for analysis. Little more than a third spent between 61 and 80 percent of their time preparing data, and 8 percent spent nearly all of their time on data preparation.
Nearly all respondents thought the process of preparing data for business intelligence and analytics activities could be improved, especially when it comes to the accuracy, quality and validity of their data and its consistency across datasets.
Below are 10 recommendations from TWDI on ways organizations can streamline the data analytics process, improve productivity and help IT staff better serve users:
1. Cut the time needed for data preparation by evaluating current data preparation procedures to eliminate unnecessary routines. Increase automation and standardized processes for incorporating and integrating new data.
2. Use data catalogs, glossaries and metadata repositories as part of the data cleansing process and enable reuse of data by others.
3. Automation makes it easier to find relevant data, while a centralized data catalog facilitates data discovery and collaboration.
4. Use technologies and methods to increase repeatability so that scripts, workflows and other elements can be reused. Develop a collaborative framework to encourage sharing of repeatable elements.
5. Consider self-service data preparation solutions for existing business intelligence and data analytics as well as data integration, transformation, data quality and metadata management. Test these self-service capabilities on smaller datasets before trying them on bigger, more complex sources.
6. Integrate self-service data preparation with self-service business intelligence and visual analytics. Evaluate new technologies enabling nontechnical users to prepare data themselves. Encourage experienced users to share their expertise with nontechnical users.
7. Create a flexible structure for analytics. Data transformation rules and associated technology should allow for tailoring data preparation processes according to the data source.
8. Ensure self-service data preparation, follows guidelines and regulations to avoid data chaos.
9. Include data preparation in data governance objectives and educate staff that following the guidelines leads to better data and data analytics.
10. Create centers of excellence to improve IT staff responsiveness to users’ needs. A team comprised of both business and IT leadership will ensure data projects align with the organization’s overall strategy.
NEXT STORY: NOAA, NASA support flood response with data