Over the past few months, my team and I have launched a new Data Warehouse project in production. The opportunity to start from scratch is always a valuable chance to gain new insights and expertise. I would like to share the experiences from this success story in the hope that they will be as beneficial to others as they have been to us.

Understand Business Requirements

The first step in starting any project, not only a Data Warehouse, is to fully understand the business requirements. This is the difference between success and failure, not just a formality. If you skip this step, I can tell you with certainty that your project will be a waste of time, energy, and resources.

To really understand what business wants to see and what your team needs to do, it’s essential to spend time talking to the people who will be using the Data Warehouse. What do they hope to accomplish? How will it help them do their jobs better? How do they plan to use the data? Getting a clear picture of their goals is essential to making sure your project is on the right track.

However, this is where things often get complicated. People usually do not understand each other, especially people in different departments who have different perspectives, priorities, and terminologies. Sometimes people do not even understand what they are saying. Business guys are the ones who are easily attracted to marketing buzzwords on the Internet believing that these terms are the solutions to their problems. I have to say that the marketing departments of data companies do a really good job of re-inventing new names for the similar term. During this project, there were dozens of times the guys told me let’s use this tool, why not use this technology, money is not a problem (until they actually got the bill).

In one of my previous projects, a stakeholder told me that he wanted a visually stunning real-time dashboard that would make the numbers dance instantly whenever users did something in the web application. And I had to explain to him:

• Visually stunning: Yes, the data analysts team can always help you with that.
• Real-time: There is no real time. If the sun disappears, we can know it only after 8 minutes. So does the data.
• We do not really need it. Business is not going to sit still and watch the numbers dance every second.

Patience is the key. They do not understand those technical buzzwords. Yes. But isn’t that why you are here as a technical specialist? Your responsibility is to listen to them, understand them, empathize with them, and tell them what you will do to help them. Your job is to translate their requirements into a workable solution.

Remember that the business stakeholders are not only the end users, but also the investors. Without their buy-in, the project can’t even get off the ground. They are funding the project, and they deserve the best service.

By starting with a clear understanding of business requirements, you set the stage for a Data Warehouse project that is aligned with the organization’s goals, ensuring that the final product delivers real value.

Understand System

A Data Warehouse is not an isolated island. It is more like a bustling city that relies on a network of interconnected systems. It receives supplies from surrounding farms and industrial areas. Since Data Warehouse pulls data from other systems, you can not build a successful Data Warehouse without understanding how the other systems work.

Imagine stakeholders telling you they want the sales figure. Then you need to know exactly which systems hold the sales number. How is that number populated in each system? It may be manually entered by users, it may be automatically calculated, it may be synchronized from other sources, it may be read-only or editable… You need to know all the surrounding information to decide the source of truth for the number we desire. You may argue that all you need to do is copy the source database over and the business will know what to do with the data. Believe me, they don’t. In fact, they have never seen the database a day in their lives. And you are the one who will tell them what they can do with your Data Warehouse.

And not knowing how the system works also risks your project design. You certainly don’t want to discover a surprise when you’re almost done with the implementation, such as a scheduled job that archives data from the database daily. If you had known that from the beginning, your design would have been very different.

Understanding the entire system in detail can be time-consuming. You should have a good sense of how the interconnected systems work together, but don’t expect to understand them in detail at the beginning of your project. Instead, I would suggest building strong relationships with the teams responsible for maintaining these systems. Meet with them, tell them what you are doing, and ask for their advice and insights. They are a goldmine of information. You can also experiment with sandbox environments and databases to uncover hidden patterns and processes.

Design a reliable Data Warehouse

Reliability is the backbone of any Data Warehouse. If your business can’t rely on the data coming out of your Data Warehouse, your project is completely a failure.

Having a solid testing strategy will greatly help. Testing is not just about finding bugs, it’s about building confidence. When you start designing the data warehouse, think less about the time when the system is running happily, there is nothing for us to do if the system keeps running as it should. Think more about the time when the system is not working and what we are going to do in that time.

And even if you do your best, bugs and issues will still happen. Don’t expect your system to be bug-free; instead, build processes to handle issues as soon as they arise. And most importantly, be transparent. If the business comes to you and asks about an issue they found, tell them what happened and what you are doing to help. Transparency is the key to trust. If you tell a lie, you are part of the problem; if you are transparent, you are part of the solution. A reliable Data Warehouse isn’t just about technology. It’s about building trust.

Choose the right tool for the right job

To build a Data Warehouse, you need a toolbox filled with different pieces to complete the picture: tools for copying data, transforming it, orchestrating jobs, and more. It is technically possible to create the tools yourself, especially if you are in a big corporation and want to control every aspect of the technology. However, in most of the cases, it is impractical. You do not have enough resources to own the technology. Thus, developing a Data Warehouse solution usually means picking the available tools and services and making them work together.

The real challenge is choosing the right tools. Beware of your enemies, the shiny marketing promises. The person who writes those buzzwords may not be the one who writes the code. Sometimes I don’t understand what they wrote, and I think they don’t understand what they wrote either. These tools are very expensive. It is important to avoid overkill. Focus on what your business really needs, not just what sounds cool. We are not going to use the most popular or the most expensive tools; we are going to find the right fit for our specific needs.

Start small, Grow big

Your investors do not have infinite patience. They want to see progress and value. Building something small but functional is far better than promising a grand project that never finish. By starting small, you can quickly deliver value and gather feedback from users.

With limited resources, we can not get everything done at once. It is important to prioritize. What matters most to your business? What will deliver the biggest impact to your customers? Concentrate on delivering those core features first. You can break the project into phases, which is a good practice. Each phase focuses on specific business requirements, data sources, or user groups. And you can gradually expand the capabilities of the Data Warehouse.

Engage users

A Data Warehouse is not just a technical marvel. It is a tool for your business. To ensure it delivers maximum value, you need to involve your users from the very beginning.

Imagine building a house without consulting the people who will live in it. People can still live in it, but they never feel it is their home. By involving them early and often, you will gain valuable insight into their needs, expectations, and challenges.

How can you engage your users?

• Involve them in the planning phase: Understand their data needs, pain points, and desired outcomes.
• Provide regular updates: Keep them informed about project progress and involve them in decision-making.
• Offer training and support: Equip users with the skills to effectively use the Data Warehouse.
• Gather feedback: Encourage users to share their thoughts and suggestions for improvement.

Remember that if you can not engage your users, any slightly higher number in their reports will quickly become your problem. If you can engage them and make them feel like they are part of the project, then any issue will become everyone’s problem.

Conclusion

Building a successful Data Warehouse is a challenging journey that requires careful planning, execution, and continuous improvement. It all starts with a deep understanding of the business requirements to ensure that every decision is aligned with the organization’s goals. Start small, iterate often, and always keep the user at the center of your efforts. A successful Data Warehouse is a collaboration between the engineering team and the business. By working together, you can create a solution that truly delivers value.

OLAP

In my previous post, we discussed the differences between Online Analytical Processing (OLAP) and Online Transaction Processing (OLTP). As a reminder, OLAP which is the access pattern of analytical queries typically:

• Consume a large number of records
• Focus on only a specific subset of columns from each record
• Aggregate data to calculate statistics (e.g., averages, sums)

Row-Oriented Storage

Row-oriented storage, a type of storage engine optimized for OLTP, stores all values belonging to a single row near each other. The entire row is essentially stored as a sequence of bytes and is usually indexed for quick retrieval. When you provide a key, the database efficiently locates the physical location of the row on disk. It then goes to that address, loads the sequence of bytes into memory, and parses it to extract the specific values you need. Let’s think of it like a csv file. A row is stored as a string of characters. If you want to access the 10th row, you have to scan the file for the 10th line break, read all the characters until you reach the next line break. Now you have to parse the result by separating it by commas to get the information you want.

While row-oriented storage is great for reading and writing individual records, it quickly becomes less suitable when faced with the demands of OLAP:

• Index, the data structure behind the ability of most row-oriented data storage to quickly locate the data, doesn’t work with analytical queries. Because analytical queries are OLAP, they don’t access data using a specific key or ID. Instead, they often use multiple conditions, such as date created within a year or product category is of some specific types. Any column in the table can be used in the where clause, and we can’t just create a separate row-based index for each column.
• Reading a single row in row-oriented storage requires loading the entire sequence of bytes from disk into memory. Thus, reading a huge number of rows with hundreds of columns (which is typical in OLAP) quickly becomes inefficient.

Column-Oriented Storage

Column-oriented storage is based on a simple idea: instead of storing all the values from one row together, just store all the values from each column together. Because the data is organized by column, a query only needs to access and process the columns that are relevant to its needs. This significantly reduces the amount of data that needs to be transferred and parsed, resulting in dramatic performance gains.

Let’s look at the example below. A sales table stored in a row-oriented format looks like this.

A column-oriented storage serializes all values in a column and store them together (as a sequence of bytes). For our example table, the data would be stored in this way:

Advantages of Column-Oriented Storage

Reading data from column-oriented storage provides several key advantages over traditional row-oriented storage, especially for analytical workloads:

• Column compression: Due to the denormalization nature in modern data warehouse, values in a column tend to be repeated. Many popular compression algorithms, such as LZW or run-length encoding, make use of the similarity of adjacent data to optimize data size. Look at the column PAYMENT_METHOD in our example. What if, instead of storing a full 4-byte string, we only needed 1 bit for it: 0 for card and 1 for bank? Now the whole column becomes one long bitmap where each row consumes only 1 bit on disk.
• Access time: Disk access is a real bottleneck. When working with data on disk, we always need to use a different set of data structures and algorithms to minimize access time (B-tree for example). By accessing only the data needed to process the query and using data compression strategy, we can scan more rows in a single read. This means fewer reads to scan an entire table with trillions of rows, and therefore less disk access time.
• Throughput: Fetching only the necessary columns and better data compression also lead to better throughput, or the amount of data processed in a given time. Throughput is extremely important when compute and storage are not in the same place and data must be transferred over the network.

Conclusion

Each database implementation can vary in its specific optimization. However, the fundamental principle - storing and processing data by column rather than by row - remains the same, leading to significant performance gains for analytical queries. Understanding how your database works behind the scenes is beneficial for you as an engineer. Knowing what your tool does also means knowing what you do.

OLTP

OLTP, or Online Transaction Processing, is a pattern by which we access and manipulate data in the database transaction-by-transaction. A transaction refers to a single unit of work, such as a money transfer, a book, a blog post, and so on. Typically, users often only interact with one or a few transactions at a time. Therefore, most of the time, applications look up a small number of records in databases by some keys. Application databases implement special indexing techniques such as B-tree or LSM-tree to handle OLTP efficiently. They can quickly access a particular transaction given its indexed key.

OLAP

As businesses grow and accumulate data, they need to analyze it to gain valuable insights about their market and customers. Then they can make informed decisions and gain competitive advantage. When it comes to analytics, access patterns will be very different. Typically, analytic queries consume a large number of records, look for only a few specific columns of each record, and often aggregate data to calculate statistics (min, max, sum, average,…). This pattern of accessing data in the database is called Online Analytic Processing (OLAP).

Difference between OLTP and OLAP

From the above definitions, we can somehow distinguish OLTP vs OLAP. The following table shows the typical differences:

* Differences between OLTP and OLAP

Problems of OLTP Databases with OLAP queries

When your business is still young, it is easy to run analysis directly on application databases. However, as the volume of data and the need for analysis grows along with the business, problems arise. Databases that were optimized for OLTP using indexing techniques such as LSM tree or B-tree now struggle to execute OLAP queries efficiently. As a result, running OLAP queries becomes costly and negatively impacts application performance, which is critical to business success.

As the business continues to grow, different business units tend to operate independently with their own goals, priorities, concerns, and IT budgets. Each business unit will maintain its own applications running on separate databases. Performing analysis when data is scattered in different locations is difficult. And analysts often end up exporting data from different places, putting it into a single Excel file, and VLOOKUP.

Data Warehouse

In response to the challenges of running OLAP queries on traditional business databases, the concept of a data warehouse is emerging as a solution.

• Data Warehouse functions as a dedicated space for analytical purpose. It allows business to store massive amounts of historical and current data without impacting operational databases.
• Data Warehouses are designed with a focus on analytical processing. Their storage engines use specialized techniques to speed up OLAP queries. We may explore these techniques in other posts.
• Data warehouses serve as a centralized repository for data from various sources. Analysis is now easier because all of the necessary data is in a single place.

Conclusion

In this post, we’ve gone over the definitions and differences between OLTP and OLAP. We also looked into the role of the data warehouse in conducting business analysis. Understanding them should give you the confidence when you say to your boss, “We need a data warehouse.”