Forward-thinking companies are overcoming the data conundrum and re-calibrating their data for growth and innovation.
Can there be too much of a good thing? In the case of data, yes.
We live in a data-driven world where companies are creating a stream of data that doesn’t end. Companies are under competitive pressure to make better-informed decisions using data. It’s more important than ever to develop fast and cost-effective processes to access historical business intelligence (BI) and the predictive power of business analytics (BA).
Data is the Foundation for Growth and Innovation
To outperform your peers, you need to successfully generate business value from your data. Yet, the volume of data is quickly getting out of hand. Research shows the average company is managing over 30 data sources and is seeing data volume grow by over 50% a year.
The volume and complexity of data can be a challenge, but it doesn’t have to derail your success. While you can now gather data insights using big data, machine learning (ML) and artificial intelligence (AI), you can’t just tell any data scientist you hired yesterday to dump the data from your enterprise resource planning (ERP) system such as SAP, Oracle E-Business Suite, and JD Edwards into a big data algorithm and call it a day.
It’s not that simple.
Although ERP systems have enormous amounts of data, they only represent one data source. You may have data in your customer relationship management (CRM) and human resource (HR) systems, or coming from Internet of Things (IoT) devices.
No matter the size of your business or where you are on your business analytics journey, you can benefit from solving today’s data conundrum. Whether you sit in the C-suite level or lead a line of business, you can gain insights from your data.
So, how do you do that? Let’s dive in.
How to Overcome the Hands-Off Approach to Data
Data lives in different ponds throughout an organization. Some are murkier than others to be able to find and access what you need. Over time, common approaches to storing and managing data have surfaced.
Many organizations take a hands-off approach to data. For example, much of the data generated over doing business is collected, stored, and archived. That’s it. It’s just sitting there.
Beyond feeding standard in-system reporting, this untouched data represents a lost opportunity to drive profitability, operational efficiency, and change. Remarkably, there are companies out there that don’t know who their top 10 customers are. Don’t be one of those businesses. Let’s discuss your options.
Export to Spreadsheets
Exporting data into Microsoft Excel spreadsheets helps companies move toward business analytics. However, it takes tons of manual effort to do and what you can actually do with the data is limited. Perhaps one of the greatest concerns of exporting data into spreadsheets is data integrity. When the data is wrong, insights can’t be trusted.
On-Premises Data Warehouse
Data warehousing was introduced in the late 1980s and early 1990s to provide a central repository of integrated data from one or more disparate sources. Fast forward to today. One of the biggest limitations of a data warehouse is the need to define up front what you want to get out of the data on the backend.
Establishing a data warehouse, defining your metrics, and mapping everything can involve significant upfront capital expenditure, time, and effort. For example:
- SAP customers: HANA is a super high-performing in-memory database with an analytics add-on. The SAP database is very rigid. You may run into licensing issues.
- JD Edwards and Oracle EBS customers: These tools have challenges with data warehousing. The architecture does not let you blend data from other data sources external to the ERP.
Data Lake Fundamentals
The time is now to leverage advances in cloud computing. It makes a lot more sense today to send both structured and unstructured data from multiple data systems to a common cloud repository, known as a data lake. Within a data lake, you can apply security, governance, and access to appropriate persons and personas in your organization.
Leading-edge companies are already seeing business gains from adopting a data lake approach. Aberdeen found that data lake leaders are outperforming followers by 9% in organic revenue growth. Efficient data capture, improved data access, and the ability to apply information to business decisions in a timelier manner are key components of their success with data lakes.
What’s the Difference Between a Data Warehouse and a Data Lake?
Here’s a quick overview of key differences between a data warehouse and a data lake.
| Traditional Characteristics | Data Warehouse | Data Lake |
| Data | Relational from transactional systems, operational databases, and line of business applications | Non-relational and relational from IoT devices, web sites, mobile apps, social media, and corporate applications |
| Schema | Designed prior to the DW implementation (schema-on-write) | Written at the time of analysis (schema-on-read) |
| Price/Performance | Fastest query results using higher cost storage | Query results getting faster using low-cost storage |
| Data Quality | Highly curated data that serves as the central version of the truth | Any data that may or may not be curated (i.e. raw data) |
| Users | Business analysts | Data scientists, Data developers, and Business analysts (using curated data) |
| Analytics | Batch reporting, BI and visualizations | Machine Learning, Predictive analytics, data discovery and profiling |
Testing the Waters
A traditional data warehouse project can take months before you can perform meaningful data analysis. Not so with a data lake. Think weeks, not months.
Working with a trusted cloud and analytics partner, you begin with a planning session or two, sharing the details of your particular data environment in order to map out a successful data lake solution. Data pulls from your ERP, legacy systems, and cloud application program interfaces (APIs) can then proceed quickly, so you’re working with the data in weeks to gain value and insight.
The serverless simplicity of data lake technology makes for a powerful solution. With a typical data warehouse solution, you would have to set up servers, install the software, make sure it’s compliant with the operating system, and deal with upgrades regularly.
so
When you use an Amazon Web Services (AWS) Lake Formation as the underlying technology to build, secure, and manage your data lake, you can rapidly increase the speed to deployment. At regular intervals, you can determine the “crawlers” that connect to your various data sources, extract the schema and metadata of all the objects, and catalog the data so it’s searchable in the lake. Users will also have secure self-service access to the data through their choice of analytics services.
Expediency and Access to All
The value of a data lake is simple. It lies in the expedient ability to access and analyze the wealth of data you have to inform business decisions. No big capital outlay. No waiting for months of development. Companies of any size and maturity can benefit with infinite scalability as you grow.
Using a data lake does not mean that you have to abandon your existing data warehouse. The data warehouse can still have a place for specific use cases, such as supporting specific reports or recurring dashboards that are consumed at a high frequency.
A data lake can help make legacy systems more efficient by offloading capacity to this newer, more flexible infrastructure. Data can be moved to the data lake and stored as flat files. You can still use the engine of the data warehouse to query for fast results and leverage high-power query capabilities of the old with the cost-effective data storage of the new.
Bringing It All Together
Regardless of the talent you engage to drive business analytics, finding a competitive advantage remains a futile fishing expedition without a viable way to access and manage your data.
To learn more about how we can help you support your business analytics needs, watch our on-demand webinar about data lakes and data warehouses, download our Business Analytics whitepaper, and visit our resources page. You can also contact us today so we can help you get more insights from your data with our Business Analytics Services.