What tool should I learn in depth?

I understand why people keep having such questions. The tech world moves extremely fast. Every few months, there is a new framework, a new orchestration tool, a shiny feature in a cloud, or some articles filled with buzzwords that make you feel like you are already behind. The pressure to keep up is real. But here is something I have learned over the years, and I want to say it clearly: focus on learning the basics in depth. Tools will change, 0 and 1 will not.

Years ago, everyone was talking about HDFS, MapReduce, Pig, Hive. Fast-forward a few years, Spark took over. Then cloud-native pipelines. Now we have got real-time streaming, feature stores, vector databases, and AI-generated pipelines. If I had spent all my time chasing tools, I would be exhausted. Always playing catch-up - and probably still behind.

Instead, I focused on understanding how data actually works; how it moves; how it is stored and transformed; how to model it for clarity, flexibility, and performance; how SQL engines work under the hood; how replication and partitioning affects performance, how to make a data schema clean and extensible. Those things have not changed. These lessons apply whether you are working with BigQuery or Snowflake, self-hosted DBT or Azure Data Factory on the cloud, or something that does not even exist yet.

Data Engineering is developing in a direction where engineers write less and less code. Unlike several years ago, you now have a lot of tools publicly available. For a new project, you are provided with a comprehensive toolbox. Your primary task then becomes picking the right tools and making them work seamlessly together. By writing less code, it makes the engineers harder to understand the technology behind the scene. I began my technology journey by writing a lot of code in Pascal - my favorite programming language. I still write some nowadays in my free time. My coding experience helps me a lot. When I work with a tool today, I can often imagine the actual code running on the machine.

Yes, you are not required to be able to write the tools from scratch. But, you are required to understand the core technology behind the tools. Because even if you are just picking items from your toolbox, you still need to pick the right tools. And picking the right tools for the right jobs is still a highly advanced skill in our industry today.

Here is another undeniable truth we can not ignore: AI is becoming incredibly good at repeating what it has learned, and it is getting more and more involved in our daily work. Honestly, I use it a lot to help me develop. This fact actually highlights the advantages of understanding the basics. If AI is not good enough, people need you because you understand the underlying concepts and can operate the AI reliably. If the AI is already good enough, why do they still need you? Because you are better than AI at understanding the things it does not know - the true “why” behind the “what”.

When you understand the basics, tools become just syntax. You can pick up new ones quickly. You can even build your own if you want to. Investors care about one thing. It is the value you created, not the tools you used. They want to know: Did you help the business make better decisions? Did you save money? Did you unlock insights faster? Those results do not come from stacking the flashiest tools, but from engineers who know what matters and make the right decisions. That is what turns you from someone who follows instructions into someone who builds solutions.

And in this field, that is the difference between being useful and being indispensable.

Over the past few months, my team and I have launched a new Data Warehouse project in production. The opportunity to start from scratch is always a valuable chance to gain new insights and expertise. I would like to share the experiences from this success story in the hope that they will be as beneficial to others as they have been to us.

Understand Business Requirements

The first step in starting any project, not only a Data Warehouse, is to fully understand the business requirements. This is the difference between success and failure, not just a formality. If you skip this step, I can tell you with certainty that your project will be a waste of time, energy, and resources.

To really understand what business wants to see and what your team needs to do, it’s essential to spend time talking to the people who will be using the Data Warehouse. What do they hope to accomplish? How will it help them do their jobs better? How do they plan to use the data? Getting a clear picture of their goals is essential to making sure your project is on the right track.

However, this is where things often get complicated. People usually do not understand each other, especially people in different departments who have different perspectives, priorities, and terminologies. Sometimes people do not even understand what they are saying. Business guys are the ones who are easily attracted to marketing buzzwords on the Internet believing that these terms are the solutions to their problems. I have to say that the marketing departments of data companies do a really good job of re-inventing new names for the similar term. During this project, there were dozens of times the guys told me let’s use this tool, why not use this technology, money is not a problem (until they actually got the bill).

In one of my previous projects, a stakeholder told me that he wanted a visually stunning real-time dashboard that would make the numbers dance instantly whenever users did something in the web application. And I had to explain to him:

• Visually stunning: Yes, the data analysts team can always help you with that.
• Real-time: There is no real time. If the sun disappears, we can know it only after 8 minutes. So does the data.
• We do not really need it. Business is not going to sit still and watch the numbers dance every second.

Patience is the key. They do not understand those technical buzzwords. Yes. But isn’t that why you are here as a technical specialist? Your responsibility is to listen to them, understand them, empathize with them, and tell them what you will do to help them. Your job is to translate their requirements into a workable solution.

Remember that the business stakeholders are not only the end users, but also the investors. Without their buy-in, the project can’t even get off the ground. They are funding the project, and they deserve the best service.

By starting with a clear understanding of business requirements, you set the stage for a Data Warehouse project that is aligned with the organization’s goals, ensuring that the final product delivers real value.

Understand System

A Data Warehouse is not an isolated island. It is more like a bustling city that relies on a network of interconnected systems. It receives supplies from surrounding farms and industrial areas. Since Data Warehouse pulls data from other systems, you can not build a successful Data Warehouse without understanding how the other systems work.

Imagine stakeholders telling you they want the sales figure. Then you need to know exactly which systems hold the sales number. How is that number populated in each system? It may be manually entered by users, it may be automatically calculated, it may be synchronized from other sources, it may be read-only or editable… You need to know all the surrounding information to decide the source of truth for the number we desire. You may argue that all you need to do is copy the source database over and the business will know what to do with the data. Believe me, they don’t. In fact, they have never seen the database a day in their lives. And you are the one who will tell them what they can do with your Data Warehouse.

And not knowing how the system works also risks your project design. You certainly don’t want to discover a surprise when you’re almost done with the implementation, such as a scheduled job that archives data from the database daily. If you had known that from the beginning, your design would have been very different.

Understanding the entire system in detail can be time-consuming. You should have a good sense of how the interconnected systems work together, but don’t expect to understand them in detail at the beginning of your project. Instead, I would suggest building strong relationships with the teams responsible for maintaining these systems. Meet with them, tell them what you are doing, and ask for their advice and insights. They are a goldmine of information. You can also experiment with sandbox environments and databases to uncover hidden patterns and processes.

Design a reliable Data Warehouse

Reliability is the backbone of any Data Warehouse. If your business can’t rely on the data coming out of your Data Warehouse, your project is completely a failure.

Having a solid testing strategy will greatly help. Testing is not just about finding bugs, it’s about building confidence. When you start designing the data warehouse, think less about the time when the system is running happily, there is nothing for us to do if the system keeps running as it should. Think more about the time when the system is not working and what we are going to do in that time.

And even if you do your best, bugs and issues will still happen. Don’t expect your system to be bug-free; instead, build processes to handle issues as soon as they arise. And most importantly, be transparent. If the business comes to you and asks about an issue they found, tell them what happened and what you are doing to help. Transparency is the key to trust. If you tell a lie, you are part of the problem; if you are transparent, you are part of the solution. A reliable Data Warehouse isn’t just about technology. It’s about building trust.

Choose the right tool for the right job

To build a Data Warehouse, you need a toolbox filled with different pieces to complete the picture: tools for copying data, transforming it, orchestrating jobs, and more. It is technically possible to create the tools yourself, especially if you are in a big corporation and want to control every aspect of the technology. However, in most of the cases, it is impractical. You do not have enough resources to own the technology. Thus, developing a Data Warehouse solution usually means picking the available tools and services and making them work together.

The real challenge is choosing the right tools. Beware of your enemies, the shiny marketing promises. The person who writes those buzzwords may not be the one who writes the code. Sometimes I don’t understand what they wrote, and I think they don’t understand what they wrote either. These tools are very expensive. It is important to avoid overkill. Focus on what your business really needs, not just what sounds cool. We are not going to use the most popular or the most expensive tools; we are going to find the right fit for our specific needs.

Start small, Grow big

Your investors do not have infinite patience. They want to see progress and value. Building something small but functional is far better than promising a grand project that never finish. By starting small, you can quickly deliver value and gather feedback from users.

With limited resources, we can not get everything done at once. It is important to prioritize. What matters most to your business? What will deliver the biggest impact to your customers? Concentrate on delivering those core features first. You can break the project into phases, which is a good practice. Each phase focuses on specific business requirements, data sources, or user groups. And you can gradually expand the capabilities of the Data Warehouse.

Engage users

A Data Warehouse is not just a technical marvel. It is a tool for your business. To ensure it delivers maximum value, you need to involve your users from the very beginning.

Imagine building a house without consulting the people who will live in it. People can still live in it, but they never feel it is their home. By involving them early and often, you will gain valuable insight into their needs, expectations, and challenges.

How can you engage your users?

• Involve them in the planning phase: Understand their data needs, pain points, and desired outcomes.
• Provide regular updates: Keep them informed about project progress and involve them in decision-making.
• Offer training and support: Equip users with the skills to effectively use the Data Warehouse.
• Gather feedback: Encourage users to share their thoughts and suggestions for improvement.

Remember that if you can not engage your users, any slightly higher number in their reports will quickly become your problem. If you can engage them and make them feel like they are part of the project, then any issue will become everyone’s problem.

Conclusion

Building a successful Data Warehouse is a challenging journey that requires careful planning, execution, and continuous improvement. It all starts with a deep understanding of the business requirements to ensure that every decision is aligned with the organization’s goals. Start small, iterate often, and always keep the user at the center of your efforts. A successful Data Warehouse is a collaboration between the engineering team and the business. By working together, you can create a solution that truly delivers value.

What is Reliability?

Everyone has their own intuition about what is reliable:

• A piggy bank is reliable because it consistently holds your money and accurately reflects what you’ve deposited. You trust that when you put a coin in, it will be there later, and the total will reflect your savings. And when you want to make a withdrawal, you can get your money immediately.
• A calculator is reliable because it consistently produces accurate results based on your input. You trust that regardless of who uses it, 2 + 2 will always equal 4. And the result should appear instantly on the screen.

Different systems have different reliability requirements. In general, we can define reliability as follow:

Reliability refers to the ability to always do the expected things in the expected way.

For software, reliability means consistently performing the designed function at the expected level of performance. Consider a calculator: we expect it to immediately display 4 after typing in 2+2. If it shows me 5, I will give it 1 star and never use it again. If it takes me 5 minutes to do such a simple arithmetic addition, I will send an email to the United Nations to report it as crypto-mining malware. (actually I won’t)

Wait a minute! There is one more important word in my definition above: “always”. What do I mean by “always”? A piggy bank wouldn’t be very reliable if it held my money and suddenly became inaccessible for a week. Of course, there is no perfect “always” in real world. There may be unforeseen situations that cause systems to stop working. But systems should be designed in such a way that the disruption doesn’t hurt business operations. Reliability focuses on minimizing the occurrence of system failures and their impact on functionality.

Reliable Data system

Just like you trust your piggy bank to hold your coins securely, you need to trust your data systems to hold your information reliably. Your piggy bank wouldn’t be very reliable if the coins sometimes disappeared, a data system wouldn’t be reliable if the information kept changing or disappearing. Reliability means you can trust the information it holds. This means the data is always available, accurate, and delivers consistent results when you need it. Common expectations for a data system:

• Integrity: This ensures the data is accurate, complete, and consistent. Imagine your piggy bank if someone took coins without putting them back, or if different amounts appeared out of nowhere. It wouldn’t be reliable! Similarly, data integrity prevents missing, incorrect, or inconsistent information, thereby ensuring its reliability.
• Availability: You wouldn’t find your piggy bank locked when you need it most. Likewise, reliable data systems must be accessible when you need them. This means the data is readily available for authorized users, minimizing downtime and ensuring critical information is always at hand.
• Performance: A sluggish piggy bank wouldn’t be very useful. Similar to how you expect quick access to your coins, data systems should deliver reasonable performance. This translates to fast retrieval times, smooth operation, and responsiveness to your needs, enabling efficient decision-making.
• Timeliness: Data freshness is crucial. Old coins are worth the same, but old data is not. In data systems, timeliness ensures that information is current and up to date. This reduces reliance on outdated data, resulting in more accurate insights and informed actions.
• Safety: Just like keeping your piggy bank safe from theft, protecting your data is critical. Data safety ensures that information is protected from unauthorized access. If someone you don’t trust knows where you keep your piggy bank, you won’t put any coins in it.

How important is Reliability

Reliability is not limited to life-or-death situations such as nuclear power plants. It is fundamental to all software applications, large and small. Sure, bugs in a note taking app may not have catastrophic consequences, but they do cause frustration and erode user trust. Let’s shift our focus from “avoiding disaster” to “delivering value”. Every software application has a purpose, whether it’s to simplify tasks, improve communication, or entertain users. When an application crashes, malfunctions, or produces incorrect results, it fails to fulfill its purpose. Every software application has a responsibility to its users. Frustrated users abandon unreliable applications, businesses lose productivity, and trust erodes. Investing in reliability is about more than avoiding the negative consequences of failure. It’s about building trust, delivering value, and ensuring that your software does what it’s supposed to do.